A method of upgrading an ebullated bed hydroprocessing system that includes one or more ebullated bed reactors with recycling vacuum bottoms and without recycle buildup of asphaltenes, comprising: operating an ebullated bed reactor using a heterogeneous catalyst to hydroprocess heavy oil at baseline conditions, optionally with recycling of vacuum bottoms and recycle buildup of asphaltenes, the baseline conditions including severity, which is related to operating temperature, throughput, and conversion; adding or forming in situ dispersed metal sulfide catalyst particles in the ebullated bed reactor to yield an upgraded ebullated bed reactor with a dual catalyst system comprised of heterogeneous catalyst and dispersed metal sulfide catalyst particles; operating the upgraded ebullated bed reactor using the dual catalyst system at same or higher severity than when operating the ebullated bed reactor at the baseline conditions to hydroprocess heavy oil to produce hydrocarbon products; subjecting the hydrocarbon products to one or more separation processes including vacuum distillation and separating distillates from vacuum bottoms containing residual metal sulfide catalyst particles; and recycling at least a portion of the vacuum bottoms containing residual metal sulfide catalyst particles into the upgraded ebullated bed reactor without recycle buildup of asphaltenes in the ebullated bed hydroprocessing system, the vacuum bottoms being recycled at a recycle ratio of from 1% to 50% based on the flow rate of recycled vacuum bottoms as a volume percent of the flow rate of fresh heavy oil added to the upgraded ebullated bed reactor and such that said recycling does not increase asphaltene concentration in the upgraded ebullated bed reactor and/or the vacuum bottoms. operating an ebullated bed reactor using a heterogeneous catalyst to hydroprocess heavy oil at baseline conditions, optionally with recycling of vacuum bottoms and recycle buildup of asphaltenes, the baseline conditions including severity, which is related to operating temperature, throughput, and conversion; adding or forming in situ dispersed metal sulfide catalyst particles in the ebullated bed reactor to yield an upgraded ebullated bed reactor with a dual catalyst system comprised of heterogeneous catalyst and dispersed metal sulfide catalyst particles; operating the upgraded ebullated bed reactor using the dual catalyst system at same or higher severity than when operating the ebullated bed reactor at the baseline conditions to hydroprocess heavy oil to produce hydrocarbon products; subjecting the hydrocarbon products to one or more separation processes including vacuum distillation and separating distillates from vacuum bottoms containing residual metal sulfide catalyst particles; and recycling at least a portion of the vacuum bottoms containing residual metal sulfide catalyst particles into the upgraded ebullated bed reactor without recycle buildup of asphaltenes in the ebullated bed hydroprocessing system, the vacuum bottoms being recycled at a recycle ratio of from 1% to 50% based on the flow rate of recycled vacuum bottoms as a volume percent of the flow rate of fresh heavy oil added to the upgraded ebullated bed reactor and such that said recycling does not increase asphaltene concentration in the upgraded ebullated bed reactor and/or the vacuum bottoms.