in the following paper they describe streamdiffusion and say that instead of the original sequential denoising mode, we batch the denoising steps. We are inspired by the asynchronous
processing in the computer architecture pipeline, and point out that we do not need to wait for the previous denoising
processes to finish then to start the current denoising step.
But how is that possible?

StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation
Akio Kodaira1,∗ Chenfeng Xu1,∗ Toshiki Hazama1,∗ Takanori Yoshimoto2 Kohei Ohno3
Shogo Mitsuhori4 Soichi Sugano5 Hanying Cho6 Zhijian Liu7 Kurt Keutzer1
1UC Berkeley 2University of Tsukuba 3
International Christian University
4Toyo University 5Tokyo Institute of Technology 6Tohoku University 7MIT
{akio.kodaira, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Figure 1. Real-time image-to-image results from camera and screen captures. The first and second columns show examples of AI-assisted
drawing in real-time, and the third column displays the real-time rendering of 2D illustrations from 3D avatars. The fourth and fifth
columns demonstrate the real-time camera filter.
Abstract
We introduce StreamDiffusion, a real-time diffusion
pipeline designed for interactive image generation. Exist-
∗ denotes equal contribution
This work was done when Toshiki was a remote intern at UC Berkeley
ing diffusion models are adept at creating images from text
or image prompts, yet they often fall short in real-time interaction. This limitation becomes particularly evident in
scenarios involving continuous input, such as Metaverse,
video game graphics rendering, live video streaming, and
broadcasting, where high throughput is imperative. To adarXiv:2312.12491v1 [cs.CV] 19 Dec 2023
dress this, we present a novel and simple approach that
transforms the original sequential denoising into the batching denoising process. Stream Batch eliminates the conventional wait-and-interact approach and enables fluid and
high throughput streams. To handle the disparity in frequencies between data input and model throughput, we design a novel input-output queue for parallelizing the streaming process. Moreover, the existing diffusion pipeline uses
classifier-free guidance to enforce the generated results
conditioned on the prompts. We point out that current implementations are inefficient due to the inherent redundancy
of negative conditional denoising. To mitigate the redundant computations, we propose a novel residual classifierfree guidance (RCFG) algorithm that reduces the number
of negative conditional denoising steps to only one or even
zero. Besides, to optimize power consumption, we use
a simple and effective stochastic similarity filtering strategy, which significantly reduces GPU activation frequency,
thereby improving GPU utilization efficiency. Our batching denoising strategy achieves around 1.5x speedup compared to the sequential denoising method at different denoising levels. The proposed (RCFG) leads to up-to 2.05x
speedups compared to the previous conventional classifierfree guidance. The combination of the proposed strategies
and existing mature acceleration tools makes the image-toimage generation achieve up-to 91.07fps on one RTX 4090
GPU, improving the throughputs of AutoPipline developed
by Diffusers over 59.56x. Beyond the pipeline efficiency,
our proposed StreamDiffusion also significantly reduces the
energy consumption by 2.39x on one RTX 3060 GPU and
1.99x on one RTX 4090 GPU for a static scene input, respectively. The code is available at https://github.
com/cumulo-autumn/StreamDiffusion.
1. Introduction
Recently, there has been a growing trend in the commercialization of diffusion models [3, 21, 23] for applications
within the Metaverse, as well as in the realms of online
video streaming and broadcasting. These areas require diffusion pipelines that offer high throughput and low latency
to ensure efficient human interaction. A pertinent example
is the use of diffusion models to create virtual YouTubers.
These digital personas should be capable of reacting in a
fluid and responsive manner to user input.
To advance high throughput and real-time interactive capabilities, current efforts primarily focus on reducing the
number of denoising iterations, such as decreasing from 50
iterations to just a few [16, 17] or even one [13, 28]. The
common strategy is to distill the multi-step diffusion models
into a few steps or re-frame the diffusion process with neural Ordinary Differential Equations (ODE) [14, 26]. Quantization has also been applied to diffusion models [9, 12] to
improve efficiency.
In this paper, we start from an orthogonal direction and
introduce StreamDiffusion, a pipeline-level solution that
enables real-time interactive image generation with high
throughput. We highlight that existing model design efforts can still integrate with our pipeline. Our approach enables the use of N-step denoising diffusion models while
still keeping high throughput and offers users more flexibility in choosing their preferred models.
We leverage a simple strategy in StreamDiffusion: instead of the original sequential denoising mode, we batch
the denoising steps. We are inspired by the asynchronous
processing in the computer architecture pipeline, and point
out that we do not need to wait for the previous denoising
processes to finish then to start the current denoising step,
as shown in Fig. 2. In order to deal with the synchronous
issue of input frequency and U-Net processing frequency,
we leverage a queue strategy to cache the inputs and output. Our pipeline is different from merely asynchronous
processing. We make use of the advantage of GPU parallelism; thus, we can use one U-Net to denoise a batched
noise latent feature. More importantly, the existing diffusion pipeline incorporated classifier-free guidance to emphasize the given prompts in the generated images. Nevertheless, traditional implementations of classifier-free guidance are plagued by excessive and redundant computational
overheads. We introduce an innovative approach termed as
residual classifier-free guidance (RCFG). The novel technique approximates the negative condition with a virtual
residual noise, which allows us to calculate the negative
condition noise only during the initial step of the process.
Further, we point out that maintaining the diffusion models always in an interactive mode is energy-consuming as it
keeps hitting GPU. To reduce the energy, we further apply a stochastic similarity filtering strategy. In the pipeline,
we compute the similarities between continuous inputs and
determine whether the diffusion model should process the
images based on the similarity.
Experiments demonstrate that our proposed StreamDiffusion can achieve up to 91.07fps for image generation on a
one RTX4090 GPU, surpassing the diffusion Autopipeline
from Diffusers team by up to 59.6x. Besides, our stochastic
similarity filtering strategy significantly reduces the GPU
power usage by 2.39x on one RTX 3090GPU and by 1.99x
on one RTX 4090GPU. Our proposed StreamDiffusion is
a new diffusion pipeline that is not only efficient but also
energy-saving.
2. Related works
2.1. Diffusion Models
Diffusion models, as introduced in [8, 25], have sparked
considerable interest in the commercial sector due to their
Figure 2. The concept of Stream Batch. In our approach, instead of waiting for a single image to be fully denoised before processing the
next input image, we accept the next input image after each denoising step. This creates a denoising batch where the denoising steps are
staggered for each image. By concatenating these staggered denoising steps into a batch, we can efficiently process continuous inputs using
a U-Net for batch processing. The input image encoded at timestep t is generated and decoded at timestep t + n, where n is the number of
the denoising steps.
high-quality image generation capabilities and the level of
control they offer over the generated images. These models
have been progressively adapted for various applications,
including text-to-image generation [2, 19, 20], image edition [1, 22], and video generation [4, 5].
A notable advancement in this area is the development of
consistent models [26], which have demonstrated the potential to improve the efficiency of the sampling process with
rarely compromising on image quality. Building on this,
recent studies have focused on enhancing the efficiency of
diffusion models by reducing the number of sampling steps
required, further expanding their practical applicability and
efficiency.
2.2. Accelerating Diffusion Model
Diffusion models are currently limited by their slow speed
in generating outputs. To address this drawback, several strategies have been developed. These include methods that don’t require additional training, like using Ordinary Differential Equation (ODE) solvers, as explored by
[14, 15, 17, 26]. Other techniques involve adaptive step size
solvers as discussed by [10] and predictor-corrector methods noted by [25]. The training-based methods are also
proposed, such as optimized discretization techniques [27],
neural operator design [29], and model distillation [24]. Our
proposed method distinguishes itself significantly from the
approaches mentioned previously. While earlier methods
primarily focus on the low latency of their individual model
designs, our approach takes a different route. We introduce
a comprehensive pipeline-level solution specifically tailored
for high throughputs, which enables efficient interactive diffusion. Our pipeline is designed to seamlessly integrate
most of the low-latency diffusion models discussed above.
By doing so, we achieve a notable increase in speed within
the interactive generation process.
3. StreamDiffusion Pipeline
StreamDiffusion is a new diffusion pipeline aiming for high
throughput. It comprises several key components: the
Stream Batch strategy, Residual Classifier-Free Guidance
(RCFG), an input-output queue, Stochastic Similarity Filter, a pre-computation procedure, and model acceleration
tools with a tiny-autoencoder. We elaborate on the details
below.
Figure 3. Virtual residual noise vectors: The orange vectors depict
the virtual residual noise that starts from the PF ODE trajectory
and points to the original input latent x0
3.1. Batching the denoise step
In diffusion models, denoising steps are performed sequentially, resulting in a proportional increase in the processing
time of U-Net relative to the number of steps. However,
to generate high-fidelity images, it is necessary to increase
the number of steps. To resolve the high-latency generation
in interactive diffusion, we propose a method called Stream
Batch.
The Stream Batch technique restructures sequential denoising operations into batched processes, wherein each
batch corresponds to a predetermined number of denoising
steps, as depicted in Fig. 2. The size of each batch is determined by the number of these denoising steps. This approach allows for each batch element to advance one step
further in the denoising sequence via a single pass through
U-Net. By iteratively applying this method, it is possible
to effectively transform input images encoded at timestep t
into their corresponding image-to-image results at timestep
t+n, thereby streamlining the denoising procedure.
Stream Batch significantly reduces the need for multiple
U-Net inferences. The processing time does not escalate
linearly with the number of steps. This technique effectively
shifts the trade-off from balancing processing time and generation quality to balancing VRAM capacity and generation
quality. With adequate VRAM scaling, this method enables
the production of high-quality images within the span of a
single U-Net processing cycle, effectively overcoming the
constraints imposed by increasing denoising steps.
3.2. Residual Classifier-Free Guidance
Classifier-free guidance (CFG) [7] is an algorithm that enhances the effect of the original conditioning by performing
vector calculations between an unconditioning or a negative conditioning term [6] and the original conditioning term
(Eq. 1).
ϵτi,cfg = ϵτi,c¯ + γ(ϵτi,c − ϵτi,c¯), (1)
where ϵτi,c and ϵτi,c¯ are predicted residual noise at the denoising step τi with conditioning embedding c and negative
conditioning embedding c¯, respectively, and γ is a guidance
scale.
This allows for benefits such as strengthening the effect
of the prompt. However, to compute the negative conditioning residual noise, it is necessary to pair each input latent
variable with the negative conditioning embedding and pass
it through the U-Net at every inference time. To address this
drawback, we introduce Residual Classifier-Free Guidance
(RCFG), which significantly reduces the computational cost
of the additional U-Net inference for the negative conditioning embedding.
Firstly, the encoded input latent x0 can be transferred to
the noise distribution xτ0
as follows,
xτ0 =
√
ατ0 x0 +
p
βτ0
ϵ0, (2)
where ατ0
and βτ0
are values determined by a noise scheduler and ϵ0 is a sampled noise from a Gaussian N (0, I).
Following the implementation of latent consistency model
[26], we can predict the data distribution xˆ0,τi−1,cfg and
generate a next step noise distribution xτi,cfg using the CFG
residual noise ϵτi−1,cfg as follows,
xˆ0,τi−1,cfg =
xτi−1,cfg −
p
βτi−1
ϵτi−1,cfg
√ατi−1
(3)
xτi,cfg =
√
ατixˆ0,τi−1,cfg +
p
βτi
ϵi
, (4)
Where ϵi ∼ N (0, I).
For the next step CFG calculation, instead of computing the next step negative conditioning residual noise ϵτi,c¯,
which requires an additional U-Net computation cost, we
hypothesize the existence of virtual residual noise ϵτi,c¯
′ , and
a virtual negative condition embedding c¯
′
that predicts the
original input image’s latent x0 from xτi,cfg, as shown in
Fig. 3. Similarly to Eq. 3, this virtual residual noise satisfies Eq. 5 to predict the input latent x0.
x0 ≈ xˆ0,τi,c¯
′ =
xτi,cfg −
p
βτi
ϵτi,c¯
′
√ατi
(5)
Given the initial value x0, and the subsequent values
of xτi,cfg obtained sequentially through Eq. 4, the virtual
residual noise ϵτi,c¯
′ can be analytically determined by employing these values with the Eq. 5:
ϵτi,c¯
′ =
xτi,cfg −
√ατix0
p
βτi
(6)
Figure 4. Input-Output Queue: The process of converting input images into a tensor data format manageable by the pipeline, and conversely,
converting decoded tensors back into output images requires a non-negligible amount of additional processing time. To avoid adding these
image processing times to the bottleneck process, the neural network inference process, we have segregated image pre-processing and postprocessing into separate threads, allowing for parallel processing. Moreover, by utilizing an Input Tensor Queue, we can accommodate
temporary lapses in input images due to device malfunctions or communication errors, enabling smooth streaming.
With the virtual residual noise ϵτi,c¯
′ obtained from Eq. 6,
we formulate RCFG as follows,
ϵτi,cfg = δϵτi,c¯
′ + γ(ϵτi,c − δϵτi,c¯
′ ) (7)
where δ is a magnitude moderation coefficient for the virtual residual noise that softens the effect and the approximation error of the virtual residual noise.
RCFG that uses the original input image latent x0 as the
residual term can effectively generate results that diverge
from the original input image according to the magnitude of
the guidance scale γ, thereby enhancing the effect of conditioning without the need for additional U-Net computations.
We call this method Self-Negative RCFG.
Not only to deviate from the original input image x0, but
also to diverge from any negative condition, we can compute the negative conditioning residual noise ϵτ0,c¯ using UNet only onetime for the first denoising step and use this to
predict the negative conditioning denoised data as follows,
xˆ0,τ0,c¯ =
xτ0 −
p
βτ0
ϵτ0,c¯
√ατ0
(8)
In Eq. 6, instead of x0, using xˆ0,τ0,c¯, we can obtain the virtual residual noise ϵτi+1,c¯
′ that can effectively diverge the
generation results from the controllable negative conditioning embedding c¯. We call this Onetime-Negative RCFG.
By combining Eq. 2, Eq. 6, and Eq. 8, both SelfNegative RCFG and Onetime-Negative RCFG can be uniformly represented by the following equation,
ϵτi,c¯
′ =
xτi,cfg −
√ατix0
p
βτi
+
s
ατiβτ0
βτiατ0
(ϵτ0,c¯ − ϵ0) (9)
When ϵτ0,c¯ = ϵ0, Eq. 9 is identical to Eq. 6 and returns
Self-Negative RCFG result.
In contrast to the conventional CFG, which requires
2n computations of U-Net, the Self-Negative RCFG and
Onetime-Negative RCFG necessitate only n and n+1 computations of U-Net, respectively, where n is the number of
the denoising steps.
3.3. Input-Output Queue
The current bottleneck in high-speed image generation systems lies in the neural network modules, including VAE
and U-Net. To maximize the overall system speed, processes such as pre-processing and post-processing of images, which do not require handling by the neural network
modules, are moved outside of the pipeline and processed
in parallel.
In the context of input image handling, specific operations, including resizing of input images, conversion to tensor format, and normalization, are meticulously executed.
To address the disparity in processing frequencies between
the human inputs and the model throughput, we design an
input-output queuing system to enable efficient parallelization, as shown in Fig. 4. This system operates as follows:
processed input tensors are methodically queued for Diffusion Models. During each frame, Diffusion Model retrieves
the most recent tensor from the input queue and forwards
it to the VAE Encoder, thereby triggering the image generation sequence. Correspondingly, tensor outputs from the
VAE Decoder are fed into an output queue. In the subsequent output image handling phase, these tensors are subject to a series of post-processing steps and conversion into
the appropriate output format. Finally, the fully processed
image data is transmitted from the output handling system
Figure 5. Inference pipeline overview: The core diffusion inference pipeline, including VAE and U-Net. By incorporating a denoising
batch and pre-computed prompt embedding cache, sampled noise cache, and scheduler values cache, the inference pipeline is enhanced
in terms of speed and enables real-time image generation. Stochastic Similarity Filter (SSF) is designed to save GPU power usage. It
dynamically gates the pass of the diffusion model. This framework realizes fast and energy-efficient real-time inference.
to the rendering client.
3.4. Stochastic Similarity Filter
When images remain unchanged or show minimal changes,
particularly in scenarios without active user interaction or
static environment, nearly identical input images are often repeatedly fed into the VAE and U-Net. This leads to
the generation of identical or nearly identical images and
unnecessary consumption of GPU resources. In contexts
involving continuous inputs, such instances of unmodified
input images can occasionally occur. To tackle this issue
and minimize unnecessary computational load, we propose
a strategy termed stochastic similarity filter (SSF), as shown
in Fig. 5.
We calculate the cosine similarity between the current
input Image It and the past reference frame Image Iref.
SC (It, Iref) = It · Iref
∥It∥∥Iref∥
(10)
Based on this cosine similarity, we calculate the probability of skipping the subsequent VAE and U-Net processes.
It is given by
P(skip|It, Iref) = max 
0,
SC (It, Iref) − η
1 − η

, (11)
where η is the similarity threshold. This probability decides
whether subsequent processes like VAE Encoding, U-Net,
and VAE Decoding should be skipped or not. If not skipped,
the input image at that time is saved and updated as the reference image Iref for future use. This probabilistic skipping mechanism allows the network to operate fully in dynamic scenes with low inter-frame similarity, while in static
scenes with high inter-frame similarity, the network’s operational rate decreases, conserving computational resources.
The GPU usage is modulated seamlessly based on the similarity of the input images, enabling smooth adaptation to
scenes with varying dynamics.
Note: We emphasize that compared to determine
whether we skip the compute via a hard threshold, the proposed probability-sampling-based similarity filtering strategy leads to a more smooth video generation. Because the
hard threshold is prone to making the video stuck which
hurts the impression of watching video streaming, while the
sampling-based method significantly improves the smoothness.
3.5. Pre-computation
The U-Net architecture requires both input latent variables
and conditioning embeddings. Typically, the conditioning
embedding is derived from a prompt embedding, which remains constant across different frames. To optimize this,
we pre-compute the prompt embedding and store it in a
cache. In interactive or streaming mode, this pre-computed
prompt embedding cache is recalled. Within U-Net, the Key
and Value are computed based on this pre-computed prompt
embedding for each frame. We have modified the U-Net to
store these Key and Value pairs, allowing them to be reused.
Whenever the input prompt is updated, we recompute and
update these Key and Value pairs inside U-Net.
For consistent input frames across different timesteps
and to improve computational efficiency, we pre-sample
Gaussian noise for each denoising step and store it in the
cache. This ensures that while each denoising step has distinct noise, every timestep retains the same noise (xt,τ ̸=
xt,τ+1 and xt+1,τ = xt,τ ). This approach is particularly
relevant for image-to-image tasks.
We also precompute ατ and βτ , the noise strength coefficients for each denoising step τ , defined as:
xt =
√
ατx0 +
p
βτ ϵ (12)
This is a minor point in low throughput scenarios, but at
frame rates higher than 60 FPS, the overhead of recomputing these static values becomes noticeable. If we achieve 60
FPS (i.e. 16.666 ms) with recomputing these static values in
the loop with pre-computed cache, we can achieve 62 FPS
(i.e. 16.066 ms).
We note that we have a specific design of key-value
cache for Latent Consistency Models (LCM). As per the
original paper, we need to compute cskip(τ ) and cout(τ ) to
satisfy the following equation:
fθ(x, τ ) = cskip(τ )x + cout(τ )Fθ(x, τ ). (13)
The functions cskip(τ ) and cout(τ ) in original LCM [17]
is constructed as follows:
cskip(τ ) = σ
2
data
(sτ )
2 + σ
2
data
, cout(τ ) = σdatasτ
p
σ
2
data + (sτ )
2
,
(14)
where σdata = 0.5, and the timestep scaling factor s =
10. We note that with s = 10, cskip(τ ) and cout(τ ) approximate delta functions that enforce the boundary condition
to the consistency models. (i.e., at denoising step τ = 0,
cskip(0) = 1, cout(0) = 0; and at τ ̸= 0, cskip(τ ) = 0,
cout(τ ) = 1). At inference time, there’s no need to recompute these functions repeatedly. We can either pre-compute
cskip(τ ) and cout(τ ) for all denoising steps τ in advance
or simply use constant values cskip = 0, cout = 1 for any
arbitrary denoising step τ .