Looking Backward: Streaming Video-to-Video Translation with Feature Banks

## Abstract

Abstract This paper introduces StreamV2V, a diffusion model that achieves real-time streaming video-to-video (V2V) translation with user prompts.
Unlike prior V2V methods using batches to process limited frames, we opt to process frames in a streaming fashion, to support unlimited frames.
At the heart of StreamV2V lies a backward-looking principle that relates the present to the past.
This is realized by maintaining a feature bank, which archives information from past frames.
For incoming frames, StreamV2V extends self-attention to include banked keys and values, and directly fuses similar past features into the output.
The feature bank is continually updated by merging stored and new features, making it compact yet informative.
StreamV2V stands out for its adaptability and efficiency, seamlessly integrating with image diffusion models without fine-tuning.
It can run 20 FPS on one A100 GPU, being 15 × \times × , 46 × \times × , 108 × \times × , and 158 × \times × faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively.
Quantitative metrics and user studies confirm StreamV2V’s exceptional ability to maintain temporal consistency.

## 1 Introduction

Text-driven video-to-video (V2V) translation, which aims to alter the input video following given prompts, has wide applications, such as creating short videos and more broadly in the film industry.
Most diffusion model based methods (Wu et al., ; Yang et al., ; Ouyang et al., ; Wang et al., ; Khachatryan et al., ; Qi et al., ; Zhang et al., ; Wang et al., ; Chen et al., ; Zhao et al., ; Geyer et al., ; Liang et al., ; Wu et al., ; Singer et al., ) use batches to process recorded videos, as shown in Figure 2(a).
However, batch processing necessitates loading all frames into GPU memory, thereby limiting the video length they can handle, typically up to 4 seconds.
Furthermore, these methods do not accommodate real-time translation and typically require several minutes to process a single 4-second clip.

Figure: Figure 2: (a) Existing V2V methods process frames in batches, restricting them to a limited number of frames. (b) Our StreamV2V processes frames in a streaming fashion, being able to operate on streaming videos in real-time.
Refer to caption: x2.png

This paper works on streaming V2V applications, such as webcam video translation and AI-assisted drawing, where users want to modify the streaming video iteratively.
This necessitates the model to handle input videos of varying lengths and perform real-time translation.
To tackle this challenge, this paper introduces StreamV2V , an approach that processes frames in a streaming fashion, as depicted in Figure 2(b).
Leveraging advancements in one-/few-step diffusion models (Song et al., ; Sauer et al., ; Luo et al., ), StreamDiffusion (Kodaira et al., ) has designed a pipeline for real-time interactive image generation.
However, directly applying StreamDiffusion for V2V tasks leads to noticeable pixel flickering across frames. This is because it treats each frame independently, disregarding the continuity of videos.
In contrast, humans memorize visual elements across frames and depict the current frame based on previous observations.
To synthesize consistent videos, it is critical to integrate a mechanism that can effectively bridge the current frame to its predecessors.

Recent studies have shown that diffusion features (Tang et al., ; Luo et al., ), captured during the U-Net’s forward process, contain rich correspondences between images.
Inspired by this, our StreamV2V maintains a feature bank, which stores the intermediate features of past frames.
For incoming frames, we extend self-attention by incorporating the corresponding stored keys and values.
This can be interpreted as a weighted sum of similar regions across frames via attention, effectively aligning the current frame with previous frames.
Additionally, to ensure the consistency of fine-grained details, we directly fuse the block’s output with similar features from past frames.

The challenge then becomes: *How can we implement this feature bank?*
A straightforward approach might store a constant number of frames, such as employing a sliding window technique.
However, this method is sub-optimal, as it inadvertently discards valuable data when a frame is omitted, and generates redundancy when the stored frames are similar.
To address this, we propose to continuously update the bank by merging redundant features within incoming and stored features.
This allows us to preserve the most representative features while keeping a consistent bank size over time.
Through our experiment, we find the feature bank can be condensed to the size needed to store just one frame.

StreamV2V requires no training or fine-tuning, making it compatible as an add-on with any image diffusion models.
It excels in efficiency, capable of processing high-resolution video (512$\times$512) in real-time at 20 frames per second (FPS) on a single A100 GPU.
This speed surpasses current V2V methods—FlowVid (Liang et al., ), CoDeF (Ouyang et al., ), Rerender (Yang et al., ), and TokenFlow (Geyer et al., )—by factors of 15$\times$, 46$\times$, 108$\times$, and 158$\times$, respectively.
We evaluate our method with quantitative metrics, such as CLIP score (Radford et al., ) and warp error (Lai et al., ), and a user study.
Our findings indicate that users significantly favor our StreamV2V over StreamDiffusion (Kodaira et al., ) (with over 70% win rates) and CoDeF (Ouyang et al., ) (with over 80% win rates). While our method may not yet match the performance of state-of-the-art (SOTA) methods like TokenFlow and FlowVid, its rapid execution opens up new streaming V2V applications. Our contributions are three-fold:

- •
To the best of our knowledge, our approach is one of the first approaches to tackle real-time video-to-video translation for streaming videos.
- •
Our method, StreamV2V, employs a simple yet effective looking-backward principle by maintaining a feature bank to improve consistency.
- •
We develop a dynamic feature bank updating strategy that merges redundant features, ensuring the feature bank remains both compact and descriptive.

## 2 Related Work

### 2.1 Video-to-video translation

Significant progress has been made in the domain of text-guided image-to-image (I2I) translation (Brooks et al., ; Hertz et al., ; Tumanyan et al., ; Mou et al., ), greatly supported by large pre-trained text-to-image diffusion models (Ramesh et al., ; Rombach et al., ; Saharia et al., ).
Similarly, video-to-video (V2V) translation, which aims to generate consistent videos, has attracted substantial interest.
To generate coherent multiple frames, most existing works (Esser et al., ; Wu et al., ; Wang et al., ; Guo et al., ; Chen et al., ; Khachatryan et al., ; Ceylan et al., ; Qi et al., ; Geyer et al., ; Wu et al., ; Liang et al., ; Ku et al., ; Singer et al., ) process batches of frames simultaneously with cross-frame attention mechanisms.
However, as the memory usage increases with increased frames, these methods are typically constrained to about 4 seconds.
Additionally, they tend to rely on expensive DDIM inversion (Song et al., ; Qi et al., ; Geyer et al., ) or optical flow computation (Yang et al., ; Liang et al., ), leading to long processing time.
In contrast, our StreamV2V handles videos in a streaming fashion, allowing for videos of any length.
Compared to concurrent work Live2Diff (Xing et al., ) that requires additional video fine-tuning to train uni-directional temporal attention, our method is training-free and can serve as an add-on with any image diffusion models.

### 2.2 Accelerating diffusion models

While achieving great generation quality, diffusion models are commonly limited by their slow speed due to the need for multiple denoising steps.
Recent advancements have introduced reusing cached features in denoising steps (Ma et al., ; Wimbauer et al., ) and one-/few-step diffusion models through distillation (Song et al., ; Meng et al., ; Luo et al., ; Sauer et al., ; Yin et al., ; Lin & Yang, ).
StreamDiffusion (Kodaira et al., ) proposes a pipeline to leverage these developments for real-time image generation.
However, its application to video without adjustments brings unsatisfactory results.
Leveraging StreamDiffusion’s groundwork, we enhance frame consistency by implementing a backward-looking feature bank.
Our approach introduces a dynamic merging technique for the feature bank, ensuring it remains compact and incurs minimal additional computational cost in comparison to StreamDiffusion.

### 2.3 Feature banks

Long-term feature banks (Wu et al., ; Pan et al., ) have been used in video understanding as supportive context features to help reason the entire video.
Adapting this concept, we employ feature banks to enhance video generation consistency.
To ensure our feature bank remains both informative and compact, we introduce a dynamic feature merging strategy.
While token merging (Bolya et al., ; Bolya & Hoffman, ) is a common method for merging similar features within single images, our approach extends this technique across video frames, differentiating it from traditional within-image operations.

## 3 Preliminary: StreamDiffusion

StreamDiffusion (Kodaira et al., ) leverages the Latent Consistency Model (LCM) (Luo et al., ) to implement a stream batch strategy, enabling the real-time generation of images.
Instead of waiting for one image to be entirely denoised (usually 2-4 steps for LCM), stream batch reformulates sequential denoising steps into batched processes.
This allows simultaneous processing of $S$ images at varying denoising steps, where $S$ is the number of denoising steps.
For instance, at a given timestep $t$ (assuming $t>S$), StreamDiffusion first encodes the image $I_{t}$ with a VAE encoder and adds a certain level of noise.
We denote encoded latent as $z_{t}^{S}$, where the subscript t denotes the frame timestep and the superscript ^S denotes the denoising step.
StreamDiffusion processes latent denoising batch $\{z_{t}^{S},z_{t-1}^{S-1},...,z_{t-S+1}^{1}\}$.
Upon advancing to timestep $t+1$, the model outputs the final latent $z_{t-S+1}^{0}$, which is then decoded into the output image $I^{\prime}_{(t-S+1)}$.
The remaining latent would be denoised one step further, and the latent $z_{t+1}^{S}$ from the new image $I_{t+1}$ would be added to the batch.
We include the diagram of the stream batch (Figure 12) in Appendix A.1.
StreamDiffusion kick-starts the process by initializing the batch with $S$ identical starting images, enabling a warm start. To further accelerate the inference, SteamDiffusion also utilizes the Tiny autoencoder (madebyollin, ) and TensorRT (NVIDIA, ) acceleration.

## 4 StreamV2V

While being real-time, directly applying StreamDiffusion to video-to-video tasks brings unsatisfactory flickering results because each frame is generated independently.
Built upon StreamDiffusion, we introduce a backward-looking mechanism so that the generation of the current frame can reason about the past to bring a consistent output.
This is realized by maintaining a feature bank storing the information of past frames. As shown on the left of Figure 3, StreamV2V fetches the stored features in the bank to process the frame $I_{t}$. We introduce two training-free techniques to leverage the stored features, namely Extended self-Attention (EA) (Section 4.1) and direct Feature Fusion (FF) (Section 4.2)
Lastly, we discuss how to update the compact but informative feature bank by dynamically merging the stored and newly incoming features in Section 4.3.

Figure: Figure 3: Overview of StreamV2V. Left: StreamV2V connects the current frame to the past by maintaining a feature bank that stores the intermediate transformer features. For new incoming frames, StreamV2V fetches the stored features and uses them by Extended self-Attention (EA) and direct Feature Fusion (FF). Middle: EA concatenates the stored keys $K_{fb}$ and values $V_{fb}$ directly to that of the current frame in the self-attention computation (Section 4.1). Right: Operating on the output of transformer blocks, FF first retrieves the similar features in the bank via a cosine similarity matrix, and then conducts a weighted sum to fuse them (Section 4.2). The update method of the feature bank is elaborated in Section 4.3.
Refer to caption: x3.png

### 4.1 Extended self-Attention

We mainly use Stable Diffusion (Rombach et al., ), which is built upon the U-Net architecture.
The model contains multiple encoder and decoder blocks.
Each block comprises a residual convolutional unit and a transformer module, the latter including a self-attention layer, a cross-attention layer, and a feed-forward network.
While it’s been a common practice to inflate the self-attention layer to cross-frame attention in image diffusion (Hertz et al., ; Tewel et al., ; Zhou et al., ) or video diffusion methods (Ho et al., ; Wu et al., ; Khachatryan et al., ; Ceylan et al., ; Qi et al., ; Liang et al., ), they require processing all the frames at the same time in a batch.
We extend the self-attention to accommodate the feature bank, which stores past frame information.
As shown in the middle of Figure 3, for a given video frame $I_{t}$, we obtain the projected queries $Q_{t}\in\mathbb{R}^{n\times d}$, keys $K_{t}\in\mathbb{R}^{n\times d}$, and values $V_{t}\in\mathbb{R}^{n\times d}$ from the intermediate transformer input $x_{t}$. Here, $n$ and $d$ denote the number and dimension of feature tokens, respectively.
Denoting $K_{fb}\in\mathbb{R}^{m\times d}$, and $V_{fb}\in\mathbb{R}^{m\times d}$ as stored keys and values from previous frames, where $m$ indicates the size of the bank, we can write the extended self-attention:

$$ $\text{ExAttn}=\text{softmax}\left(\frac{Q_{t}\cdot\left[K_{t},K_{fb}\right]^{ Tr}}{\sqrt{d}}\right)\left[V_{t},V_{fb}\right]$ (1) $$

$\left[\cdot\right]$ and ^Tr denotes concatenation and transpose operation.
Essentially, this extended self-attention functions as a weighted sum of similar areas across frames, thereby aligning the current frame with its past for improved consistency. (More illustrations can be found in Appendix A.2)
Without specific notification, we extend all the self-attention layers with all denoising times. Further details on updating the feature bank will be explained in Section 4.3.

### 4.2 Feature fusion

While extended self-attention offers significant improvements in consistency, it operates implicitly through attention.
We further introduce an explicit strategy for enhancing fine-grained consistency by directly fusing features based on correspondence.
This is motivated by recent findings that diffusion features (Tang et al., ; Luo et al., ) during the U-Net forward process contain rich correspondences between images.
As shown on the right of Figure 3, for a given video frame $I_{t}$, we obtain the output features for the intermediate blocks $O_{t}\in\mathbb{R}^{n\times d}$, and we also maintain the output features of past frames $O_{fb}\in\mathbb{R}^{m\times d}$.
For the token at position $p$ in $O_{t}$, we seek the closed token at position $q$ in $O_{fb}$, utilizing cosine similarity as described in (Tang et al., ). We denote $O_{t}(p)$ as selecting the token $p$ from $O_{t}$ and $O^{\prime}_{t}$ as the fused output feature.

$$ $O^{\prime}_{t}(p)=(1-\alpha)\,O_{t}(p)+\alpha\,O_{fb}(q),\>\text{where}\,q= \text{arg max}\left(O_{t}(p)\cdot O_{fb}^{Tr}\right)$ (2) $$

Where $\alpha$ is a hyperparameter to identify the strength of fusion, which is usually set to 0.75. Intuitively, this direct feature fusion aims to enhance consistency by merging similar regions from past frames to the current frame.
In some cases, the current frame introduces novel regions which are absent in past frames.
To prevent misalignment, we generate a mask based on a predefined similarity threshold.
Specifically, we generate the mask by calculating the cosine similarities between each feature in the current output and all features stored in the bank.
For each feature, we identify the most similar stored feature and calculate the similarity score.
If the score is lower than a predefined threshold (set at 0.9), it indicates that we cannot find a suitable match in the feature bank.
In such cases, we mask out this position, meaning this feature remains unchanged and is not fused with others from the bank.
Our analysis further indicates that the location of feature fusion across various network blocks significantly impacts overall performance.
Specifically, we observed that limiting feature fusion to the low-resolution decoder blocks results in optimal performance enhancements.
For a comprehensive comparison and further insights, refer to Appendix A.3.

### 4.3 Updating the feature bank with dynamic merging

Figure: Figure 4: Naive queue vs our dynamic merging (DyMe). DyMe has a more compact and informative feature bank.
Refer to caption: x4.png

Denoting all transformer intermediate features of frame $I_{t}$ as $T_{t}=\{K_{t},V_{t},O_{t}\}$, where $K_{t}$ and $V_{t}$ are projected keys and values of self-attention input, $O_{t}$ is the output of transformer block.
An initial method for creating a feature bank is to store features from a constant number of frames.
As depicted in Figure 4 (a), after deciding the maximum number of frames which is set to 2, the bank operates like a queue: as new frames arrive, the oldest frames are popped out to make space for the newcomers.
Yet, this method encounters two primary limitations: (1) the continuous removal of the oldest frames restricts the feature span to a brief temporal window, and (2) the redundant features in the bank incur extra storage and processing costs.

To create a compact and informative bank, we propose to dynamically merge (DyMe) the stored features and newly coming features.
As shown in Figure 4 (b), at timestep 2, we merge $T_{1}$ and $T_{2}$ into $T_{\{1,2\}}$, which has the same size as $T_{1}$.
By applying DyMe, we can get condensed bank $T_{\{1,2,3,4\}}$ which contains the information for all 4 frames, yet occupies only half the space of a naive queue.
We follow the efficient bipartite matching technique (Bolya et al., ; Bolya & Hoffman, ) to do the merging.
In more detail: (1) we first concatenate features of the current frame (green boxes) and features stored in the bank (gray boxes); (2) we then randomly partition the features into two sets source (src) and destination (dst) of equal size; (3) For each feature vector $f_{\text{src}}$ in the src set, we identify the most similar feature vector $f_{\text{dst}}$ in the dst set using the cosine similarity metric defined as follows: $\text{sim}(f_{\text{src}},f_{\text{dst}})=\frac{f_{\text{src}}\cdot f_{\text{
dst}}}{|f_{\text{src}}||f_{\text{dst}}|}$.
In this expression, $\cdot$ represents the dot product, and $|\cdot|$ denotes the norm of a vector.
(4) Once we find the most similar features $f_{\text{dst}}$ of $f_{\text{src}}$, we proceed to merge the features from src into dst. This is achieved by averaging the values of each matched pair:$f_{\text{dst}}^{\text{new}}=\frac{f_{\text{src}}+f_{\text{dst}}}{2}$. This step ensures that the features in dst are updated to reflect a blend of both the original and the matched features from src.
Our experiments show that our dynamically merged feature bank brings better performance-speed trade-off than the naive queue-based bank.
Empirically, we find the feature bank can be condensed to the size of storing just one frame.
For more details, please refer to Section 5.4.2.

## 5 Experiments

### 5.1 Implementation details

We built our method on StreamDiffusion (Kodaira et al., ) with Latent Consistency Model (Luo et al., ).
By default, we use 4-step LCM without the classifier-free guidance (Ho & Salimans, ).
We update the feature bank every 4 frames.
The underlying image-to-image method is SDEdit (Meng et al., ), with an initial noise strength of 0.4.
Unlike some methods (Yang et al., ; Liang et al., ) which use frame interpolation to generate high FPS video, our StreamV2V generates all frames in the same pipeline.

Following FlowVid (Liang et al., ), we build our user study by selecting 19 object-centric videos from the DAVIS trainval 2017 dataset (Pont-Tuset et al., ), covering diverse subjects such as humans and animals.
We reuse 67 prompts from  (Liang et al., ), ranging from stylization to object swaps, for these videos.
We conduct a thorough comparison with state-of-the-art V2V methods such as Rerender (Yang et al., ), CoDeF (Ouyang et al., ), TokenFlow (Geyer et al., ), and FlowVid (Liang et al., ), utilizing their official codes under default settings.
We report both qualitative comparison 5.2, and quantitative metrics 5.3, such as CLIP score (Radford et al., ), warp error (Lai et al., ), and user preference, to verify the effectiveness of our method.

Figure: Figure 5: Qualitative comparison with representative V2V models. Prompt is ’A pixel art of a man doing a handstand on the street’. Our method stands out in terms of prompt alignment and overall frame consistency. We highly encourage readers to refer to video comparisons in our supplementary videos.
Refer to caption: x5.png

### 5.2 Qualitative results

In Figure 5, we qualitatively compare our StreamV2V with several representative V2V methods.
Starting with our per-frame baseline StreamDiffusion (Kodaira et al., ), which treats each frame independently.
StreamDiffusion often results in noticeable flickering, such as the background flowers and the dancer’s legs.
CoDeF (Ouyang et al., ) tends to produce outputs with significant blurriness, especially when there is a big motion within the input video, which fails in the construction of the canonical image.
Rerender (Yang et al., ) fails to keep tracking the clothing color in the dance, which fluctuates between brown and blue.
TokenFlow (Geyer et al., ) occasionally struggles to follow the prompt, such as transforming the video into pixel art.
FlowVid (Liang et al., ) maintains good prompt alignment but similarly struggles with color consistency in clothing and suffers from blurriness in the final frame.
In contrast, our method stands out in terms of editing capabilities and overall video quality, demonstrating superior performance over these methods.
Moreover, our StreamV2V can handle arbitrary lengths of videos as showcased in the Appendix A.4.

### 5.3 Quantitative results

**Table 1: Quantitative metrics comparison. We report the CLIP score and warp error to indicate the consistency of generated videos. We bold the best result and underline the second best.**
|  | StreamDiffusion | CoDeF | Rerender | TokenFlow | FlowVid | StreamV2V (ours) |
| --- | --- | --- | --- | --- | --- | --- |
| CLIP score $\uparrow$ | 95.24 | 96.33 | 96.20 | 97.04 | 96.68 | 96.58 |
| Warp error $\downarrow$ | 117.01 | 116.17 | 107.00 | 114.25 | 111.09 | 102.99 |

#### 5.3.1 Evaluation metrics

CLIP score Following previous research (Liang et al., ), we first utilize CLIP (Radford et al., ) to evaluate the consistency of generated videos. Specifically, we first get the CLIP image embeddings for all the video frames, then we measure the mean cosine similarity across all sequential frame pairs.
Our evaluation, detailed in Table 1, includes an analysis of 67 video-prompt pairs from the DAVIS dataset.
TokenFlow shows superior performance in maintaining temporal consistency which aligns with the findings from our user study.
StreamDiffusion, which is a per-frame image model, achieves the worst score.
Our StreamV2V ranks in third place regarding CLIP score. Taking the consideration that our StreamV2V is significantly faster than these V2V methods (Section 5.3.3), our method could have a good performance-speed trade-off.

Warp error
We also propose to use warp error (Lai et al., ) as a measure of temporal consistency following previous research (Ceylan et al., ; Geyer et al., ).
We first compute the optical flow between consecutive frames in input video using a pre-trained RAFT flow estimator  (Teed & Deng, ).
Then we warp the frame in the transferred video to the next using the estimated flow.
This allows for the evaluation of pixel discrepancies between the warped frame and the target counterpart.
We calculate the average mean squared pixel error over the un-occluded regions as warp error.
As shown in Table 1, our StreamV2V achieves the best warp error among all the methods.

#### 5.3.2 User study

Figure: Figure 6: User study comparison. The win rate indicates the frequency our StreamV2V is preferred compared with certain counterpart.
Refer to caption: x6.png

We further conducted a human evaluation to compare our method with five notable works. The user study involves 67 video-prompt pairs from DAVIS.
We adopt a Two-alternative Forced Choice (2AFC) protocol used in (Yang et al., ; Geyer et al., ), where participants are shown our result and a counterpart result, and are asked to identify which one has the best quality, considering both temporal consistency and text alignment.
For each comparison, feedback was gathered from at least three different participants.
The final win rates are shown in Figure 7.
We find users significantly favor our StreamV2V over StreamDiffusion baseline (over 70% win rates) and CoDeF (over 80% win rates).
It is important to note that while our method does not outperform the more advanced V2V methods such as Rerender, FlowVid, and TokenFlow, StreamV2V primarily aims to achieve real-time video transfer, rather than beating the state-of-the-art V2V methods.
In practice, we recommend using StreamV2V for applications where speed is a critical requirement, such as real-time webcam translation and draw rendering. For other uses that don’t require long video processing or speed, such as creating short video content, slower methods like FlowVid might yield better results.
We have made all videos from the user study available on our project page for further examination.
More details about the user study can be found in Appendix A.5.

#### 5.3.3 Pipeline runtime

We benchmark the runtime with a 512 $\times$ 512 resolution video containing 120 frames (4 seconds video with FPS of 30) in Figure 7.
For StreamV2V, we conducted ten runs, discarded the top and bottom two results, and averaged the remaining six.
The runtime for FlowVid, CoDeF, Rerender, and TokenFlow is taken from the FlowVid paper (Liang et al., ). FlowVid (Liang et al., ) and Rerender (Yang et al., ) first generate key frames, then use frame interpolation methods (Jamriska, ; Huang et al., ) to generate non-key frames. We choose the keyframe interval of 4 for these two methods, following (Liang et al., ).
Two other methods, CoDeF (Ouyang et al., ) and TokenFlow (Geyer et al., ), both require per-video preparation. Specifically, CoDeF involves training a model for reconstructing the canonical image, while TokenFlow requires a 500-step DDIM inversion process to acquire the latent representation.
Unlike all these methods requiring two-stage processing, our StreamV2V produces all the frames in one single stage.
We continue to use xFormers (Lefaudeux et al., ) for fair comparisons with existing methods. Our StreamV2V processes videos in 9 seconds with the 4-step LCM, making it significantly faster than current approaches.
It’s notable that while existing V2V methods use an A100-80GB GPU for runtime measurement, we only use an A100-40GB GPU. Despite this, StreamV2V remains significantly faster.
Further acceleration is possible with fewer steps, though this may slightly affect performance. For more details, see Appendix 13.

### 5.4 Ablation study

We ablate several of our key designs in this section. The evaluation set contains 18 in-house videos with cartoon-style prompts.

#### 5.4.1 Extended self-Attention (SA) and Feature Fusion (FF)

As shown in Figure 8, the StreamDiffusion baseline, which doesn’t have EA or FF, produces inconsistent outcomes. Noticeable flickering can be observed in the man’s hands and hair and the strips on the clothes.
After introducing the EA (the third column), the consistency is much improved, with the warp error decreased from 85.2 to 74.0. However, some artifacts persist in the man’s hand.
After further introducing FF (the last column), we have the most consistent result, with the lowest warp error of 73.4.
We also isolate the effect of FF in the fourth column, we lower the warp error from 85.2 to 80.4 with the introduction of FF.

0.68 {paracol}2 {leftcolumn}

Figure: Figure 8: Ablation on Extended self-Attention (EA) and Feature Fusion (FF). Warp error is averaged within 18 videos.
Refer to caption: x8.png

**Table 2: Ablation on different feature banks. No bank indicates the StreamDiffusion baseline. Queue represents the naive first-in-first-out queue. Our Dynamic Merging (DyMe) maintains a compact and informative bank.**
| Bank type<br>(size) | Time<br>(ms) | Mem<br>(GiB) | Warp<br>error $\downarrow$ |
| --- | --- | --- | --- |
| no bank (0) | 60 | 3.68 | 85.2 |
| queue (1) | 70 | 4.03 | 76.4 |
| queue (2) | 79 | 4.17 | 74.8 |
| queue (4) | 100 | 4.77 | 72.4 |
| DyMe (1) | 75 | 4.07 | 73.4 |

Figure: Figure 9: Continuous text-to-image generation with feature bank. The images from left to right are the intermediate generation as we type continuously. The caption of each column image is a segment of the prompt (separated by the line). (a) Per-frame LCM baseline has severe flickering even with slight one or two-word modification. (b) Feature bank would provide a much smoother transition. We highly encourage readers to refer to video comparisons in our supplementary videos.
Refer to caption: x9.png

#### 5.4.2 Dynamic merging (DyMe) bank

We study different feature banks in Table 2. In the case of no bank, which represents the StreamDiffusion baseline, it needs 60 ms to process a 512$\times$512 frame on a single A100 GPU. However, its warp error is considerably high at 85.2.
For queue-based banks, we test sizes of 1, 2, and 4.
As expected, increasing bank size raises inference time but reduces warp error. Our dynamic merging (DyMe) bank achieves a running speed of 75 ms, faster than the queue bank of size 2, while maintaining a lower warp error.
We also report the memory usage obtained via the nvidia-smi command.
The introduced DyMe bank only adds a very small 10% more GPU memory overhead to StreamDiffusion.

### 5.5 Extend feature bank to continuous image generation

The concept of the proposed feature bank, which links the current with the past, is expected to be broadly applicable beyond video-to-video. To demonstrate its versatility, we validate its effectiveness in a continuous text-to-image generation task.

Continuous or real-time image generation allows users to produce images instantly when the prompt is being typed.
However, most current frameworks generate each frame independently, resulting in flickering transitions between frames.
Figure 9(a) illustrates per-frame generation in the LCM (Luo et al., ).
The images from left to right reflect the intermediate results as we type continuously.
Even with slight one or two words, the layout of the image can shift dramatically.
By incorporating a feature bank as in Figure 9(b), the model can reference previous frames, enabling smoother transitions.

### 5.6 Limitations

Figure: Figure 10: Limitations of StreamV2V. (a). StreamV2V fails to alter the person within the input video into Pope or Batman. (b). StreamV2V can produce inconsistent output, as seen in the girl for Anime style and the backpack straps for Van Gogh style.
Refer to caption: x10.png

Despite achieving notable improvements, our model encounters certain limitations.
First, our StreamV2V can fail to alter the video with the provided prompts.
As shown in Figure 10(a), when we try to transfer the man to Pope or Batman, our model struggles to change the input video.
This may be due to the use of SDEdit (Meng et al., ) as the primary image editing method, which has limited ability to significantly alter objects. We also find that increasing initial noise strength can improve editability but at the cost of introducing more pixel flickering.
Another limitation is that StreamV2V can produce inconsistent output, especially for videos with rapid movements of the camera or the object.
This is also the major reason why our StreamV2V still cannot match the performance of state-of-the-art FlowVid and TokenFlow.
As demonstrated in Figure 10(b), when converting the video into anime style, the appearance of the girl in the background changes significantly. Similarly, when applying Van Gogh style, there are noticeable changes to the backpack straps of the man.

## 6 Conclusion

In this paper, we present StreamV2V: a diffusion model that can perform real-time video-to-video translation for streaming videos.
The key to StreamV2V lies in a look-backward mechanism that enables the current frame to reason the past to ensure consistency.
We propose a feature bank to store the intermediate features for past frames and reuse them in the current frame generation via extended self-attention and direct feature fusion.
Furthermore, we propose dynamic merging, a method to make the bank compact and informative.
Our evaluation shows that StreamV2V is an order of magnitude faster than existing video-to-video methods and can produce temporally consistent outputs.

## Broad Impacts

Our research aims to synthesize consistent videos with significant efficiency, intended for beneficial uses. However, like many generative models, it may also be exploited for fraudulent or other harmful purposes. We strongly oppose any use of our technology to create misleading or harmful content. Furthermore, we acknowledge that the videos produced by our model contain subtle artifacts, which are easily identifiable by humans.

#### Acknowledgments

We would like to express sincere gratitude to Hung-Yueh Chiang, Dennis Menn, and Wen Liu for their valuable discussion and feedback.

Feng Liang and Diana Marculescu were supported in part by the ONR Minerva program, iMAGiNE - the Intelligent Machine Engineering Consortium at UT Austin, and a UT Cockrell School of Engineering Doctoral Fellowship.

## Appendix A Appendix

### A.1 Diagram of stream batch

Figure: Figure 11: Stream batch of StreamDiffusion. It enables the processing of $S$ images with different denoising steps in a batch ($S$ is set to 3).
Refer to caption: x11.png

### A.2 Heatmap visualization of extended self-attention

Given an anchor point (marked in red) in frame 4, we extract its query tensor from the last layer of the UNet up-block.
We also obtain and store key tensors from the preceding frames 1, 2, and 3.
We then calculate the dot product between the anchor query and stored key tensors.
The most similar position is marked in red point in frames 1, 2, and 3.
This shows that the anchor point in frame 4 is highly correlated with the same goggle spot in earlier frames, indicating that extended self-attention functions as a weighted sum of similar areas across frames, thereby aligning the current frame with its past for improved consistency.

Figure: Figure 12: Heatmap of extended self-attention. The anchor point (marked as red) in frame 4 has a very high correlation with the same goggles spot in previous frames.
Refer to caption: x12.png

### A.3 Feature fusion positions

While feature fusion significantly enhances temporal consistency, it may result in blurry artifacts. As demonstrated in Figure 13, incorporating feature fusion in the layers makes the cat’s face appear blurry.
We believe that doing feature fusion at high-resolution features would average the features across frames, resulting in a diminishing of details.
Thus, we suggest applying feature fusion only to low-resolution features, specifically within the middle block and the first two blocks of the upper block. As displayed in the right column of Figure 13, this approach preserves higher-quality details compared to using all layers.

Figure: Figure 13: Ablation on feature fusion (FF) layers. Applying FF to all layers sometimes results in blurry artifacts, as seen in the cat’s face. We propose to only apply FF to low-resolution layers, specifically the middle block (mid) and the first two blocks of the upper block (up01).
Refer to caption: x13.png

As outlined in Section 5.1, our StreamV2V utilizes LCM (Luo et al., ) as the accelerated diffusion model. LCM can perform denoising in 4, 2, or 1 steps, and we examine these variations in Figure 14. Reducing the number of steps decreases the inference time per frame from 75 ms to 41 ms. However, this also leads to a decline in video quality. Unless otherwise specified, we use the 4-step LCM to achieve optimal performance.

Figure: Figure 14: Ablation on different denoising steps. While using fewer denoising steps would accelerate the inference time for every frame, we do observe a certain level of quality drop if we use only 1 step.
Refer to caption: x14.png

### A.4 More results on long video

While most existing methods typically handle up to 4 seconds of video, StreamV2V can scale to arbitrary lengths thanks to the streaming processing and feature bank.
As demonstrated in Figure 15, our approach handles a video exceeding 1000 frames (over 30 seconds) while maintaining consistent face swapping or style transfer throughout.

Figure: Figure 15: Long video ($>$ 1000 frames) generation. Our StreamV2V can handle arbitrary length of videos without consistency degradation.
Refer to caption: x15.png

### A.5 More details of user study

Figure: Figure 16: User study Interface. Each user is asked three questions about temporal consistency, prompt alignment and overall preference.
Refer to caption: extracted/5847109/figures/user_study_demo.png

As described in Section 5.3.2, we conducted a user study to compare our method with existing approaches. Figure 16 illustrates the interface presented to each participant. The instructions given to users were as follows: ”The left/right video is labeled as A/B. You will be asked three questions to compare these two videos. If you perceive minimal differences between the videos, whether they are both good or bad, please select ’draw’.

We find that the preference is very subjective and there are huge variances between users. We report the mean and standard deviation among at least three users in Table 3.
It is worth noting that, instead of providing numerical metrics for comparison, the study aims to offer a first-impression comparison between the two models. The results, whether indicating one model is better, worse, or on par with another, are generally consistent. This information can guide our decision-making, particularly when we need to balance speed and accuracy in model selection. In practice, we recommend using StreamV2V for applications where speed is a critical requirement, such as real-time webcam translation and draw rendering. For other uses, such as creating short video content, slower methods like FlowVid might yield better results.

**Table 3: Comparison of StreamV2V against various models. We also report the mean and standard deviation (std) for each set of users.**
| Model | StreamV2V wins<br>(mean $\pm$ std) | Draws<br>(mean $\pm$ std) | StreamV2V loses<br>(mean $\pm$ std) |
| --- | --- | --- | --- |
| vs. StreamDiffusion | $71.1\pm 10.2$ | $25.3\pm 13.3$ | $3.6\pm 5.2$ |
| vs. CoDeF | $81.3\pm 20.1$ | $13.2\pm 12.1$ | $5.5\pm 9.3$ |
| vs. Rerender | $30.2\pm 15.1$ | $31.4\pm 7.8$ | $38.4\pm 18.9$ |
| vs. FlowVid | $19.8\pm 9.5$ | $21.3\pm 8.5$ | $58.9\pm 18.0$ |
| vs. TokenFlow | $19.3\pm 9.0$ | $13.4\pm 9.7$ | $67.3\pm 13.4$ |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - What is the size of the feature bank used in the experiments? Does the size of the feature bank affect the performance of StreamV2V?
- Can the proposed method be applied to previous text-to-video approaches to improve the performance of long video generation?
soundness: 3
strengths: - The motivation behind StreamV2V is clear and addresses an essential limitation in existing video-to-video methods that require batch processing and are constrained by GPU memory.
- The implementation is designed for practical efficiency, with a clear description of feature fusion and dynamic feature bank updates, ensuring minimal memory overhead and ease of replication.
- The method provides flexible control over generating different styles without requiring model fine-tuning, allowing users to adapt the output to various visual aesthetics, which makes it suitable for creative applications.
summary: This paper presents StreamV2V, a real-time streaming video-to-video (V2V) translation method leveraging a backward-looking feature bank to maintain temporal consistency across video frames. Unlike traditional batch-processing approaches, StreamV2V processes incoming video frames in a streaming fashion, which allows it to handle an unlimited number of frames without fine-tuning. The method introduces a feature bank that stores intermediate features from past frames, extended self-attention, and direct feature fusion to improve temporal consistency. Furthermore, a dynamic feature bank updating strategy is implemented to reduce redundancy and optimize efficiency. The experimental results show that StreamV2V can achieve real-time processing at 20 FPS on a single A100 GPU, significantly outperforming existing methods like FlowVid, CoDeF, Rerender, and TokenFlow in terms of speed, with competitive consistency performance.
weaknesses: - A significant concern is that using the feature bank to generate higher resolution could lead to a drastic increase in memory usage.
- The DyMe (Dynamic Merging) mechanism lacks detailed pseudocode to illustrate the workflow, which would help in understanding and replicating the process.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Please clarify the points mentioned in weaknesses above.  Particularly:

- How are the magic constants determined and have you studied the impact?  If so, what did you find?

- Please provide the details of the user study.

- Please articulate the design choices, particularly SDEdit.
soundness: 4
strengths: - The core idea of the paper makes sense and is clearly presented.
- The method achieves good results, both in terms of computational time (50 ms per frame) and quality (using objective metrics and subjective user studies).
- Limitations and weaknesses against SOTA methods are presented.
summary: StreamV2V is an online approach (as opposed to batch / offline) for generating continuous video-to-video translation output without limits to video length and runs at 20 fps on a single A100 GPU.  The core idea behind StreamV2V is a feature bank from past frames that is retrieved and fused to ensure temporal coherence and consistency in frame generation.
weaknesses: - Some magic constants are reported without explanation (e.g., threshold of 0.90, max number of frames of 2, update feature bank every 4 frames).  How sensitive is your methods to these parameters?  What are the impact of higher or lower values? 
- The user study is presented without the necessary details.  How many participants? Age group? Gender? Prior experience with generative videos?  The authors should refer to papers reporting user studies to see what is the norm of doing so (e.g., see papers from CHI)
- The choice of SDEdit (which is 3 years old) should be explained -- why not newer methods?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Refer to the first question in weakness.
soundness: 3
strengths: This work is well-motivated and addresses an important problem. The method proposed is technically sound and practically feasible. The method achieves real-time V2V translation speed with a translation quality on par with the SOTA methods. The experiments, especially the ablation studies, are convincing enough to show the system-level contribution of the work.
summary: This paper presents a real-time zero-shot method for streaming video-to-video translation based on diffusion models. The key innovation is the adoption and maintenance of a dynamic feature bank which significantly increases the cross-frame consistency in the resultant videos. Although Steam Diffusion is existing work and there are a huge amount of memory updating algorithms in the field of video understanding and visual tracking, this reviewer appreciates the soundness and practical value of the proposed approach.
weaknesses: There are still some artifacts in the result videos. This reviewer wonders whether a training-free method could eventually achieve a satisfactory performance. This will need some discussions. 
Component wise, the innovation of this work is not very high. But as stated above, this reviewer appreciates the system-level contributions of the work. 
In quantitative metrics comparison, the CLIP score is not quite differentiating.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors claim that the method can be used as an add-on with any image generation model without any training or fine-tuning. This claim needs further elaboration. How is it that the method can be integrated to any translation model without any fine-tuning? How do the authors decide which features to store in the feature bank and which features to use in feature fusion? Could some fine-tuning improve the performance of the overall model? If not, why? If so, why haven't the authors done that so that overall performance could be improved?
soundness: 3
strengths: The paper has the following novel contributions: 1) videos can be translated in a streaming fashion without the need to process long sequences of video frames, 2) The past information of previous frames is stored in a feature bank and used during the generation of current frame without the need to fine-tune the existing image generation model, 3) Extended self-attention and feature fusion techniques are introduced to provide temporal consistency between consecutive frames. The most important contributions are the streaming video generation capability and the ability to use the method as an add-on with any image generation model without any training or fine-tuning.
summary: The paper describes a video-to-video translation approach  that generates video frames in a streaming fashion. The method is based on StreamDiffusion for batch processing of frames and uses SDedit for generating each frame. The method needs no training or fine-tuning can be used together with any image diffusion model. Although the results are superior to some recent video translation models, they are inferior to the results of TokenFlow and FlowVid. However the method excels in its inference time.
weaknesses: The proposed method is fast and efficient, but it lags behind state-of-the-art  video translation models in CLIP score and subjective evaluations. The level of novelty is marginal: the paper does not propose a new video translation architecture, but merely develops an efficient feature storage, update and fusion strategy for temporal consistency. The overall approach is based on the existing work of StreamDiffusion and SDEdit.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. how does the proposed framework still outperform other video-to-video translation techniques for variations of diffusion models? Do different models with the proposed framework still outperform other video-to-video translation techniques?

2. what does the running-time comparison look like on CPU Inferences rather than GPU inferences?
soundness: 3
strengths: 1. this paper addresses the challenge of real-time video-to-video translation via the streamV2V framework with backward-looking and training-free techniques. Since streamV2V is training-free and fine-tuning-free compared to other video-to-video techniques, the proposed approach is feasible to be deployed to practical applications.

2. the streamV2V framework enhances coherence between adjacent translated video frames through the extended self-attention layer and a proposed feature fusion approach for intermediate features of the current and the past frames. Due to the extended self-attention layer and the feature fusion approach, streamV2V effectively reuses the intermediate features from the past video frames to generate coherent translated video frameworks between adjacent frames.

3. the streamV2V stores projected keys, projected values, and outputs of intermediate features of the past video frames through a feature bank with a dynamic merging mechanism. With the dynamic feature bank, the streamV2V framework trivially reuses historical features to align the outputs of intermediate blocks of the current and the past frames.
summary: The paper presents StreamV2V, a diffusion model for real-time, streaming video-to-video (V2V) translation using user prompts. It achieves efficiency by maintaining a continually updated feature bank of past frames, which it references to enhance incoming frames without needing fine-tuning. StreamV2V seamlessly integrates with image diffusion models and operates at 20 FPS on an A100 GPU, making it significantly faster than competing models. User studies and quantitative metrics validate StreamV2V's strong temporal coherence and adaptability.
weaknesses: 1. the proposed streamV2V framework fails to perform video-to-video translation for high-quality video generation in some scenarios while a prompt is provided.

2. the proposed streamV2V framework fails to perform video-to-video translation when input video includes rapid motion movements.

3. to test the generalization of the proposed framework, the manuscript misses experimentation comparisons for variations of difusion models.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. The paper proposes a novel approach to streaming video-to-video translation, introducing a feature bank mechanism to address temporal consistency, enabling each frame in the video stream to reference features from past frames in an innovative manner.

2. This method achieves efficient real-time processing without model fine-tuning, outperforming many comparable methods in experimental results.

3. StreamV2V requires no training or fine-tuning and seamlessly integrates with existing image diffusion models, demonstrating the potential for a wide range of video editing tasks.

4. The paper includes extensive quantitative experiments and user studies, validating its performance.

5. This paper is well written and easy to follow. The supplementary materials also include comprehensive experimental details, content, and discussions, demonstrating the effectiveness of the proposed method.
summary: This paper introduces StreamV2V, a real-time video-to-video translation model that uses a feature bank and backward-looking mechanism to ensure temporal consistency. StreamV2V achieves significantly faster speeds than existing methods while maintaining consistent outputs.
weaknesses: 1. Does DyMe Bank still have advantages in video quality compared to queue-based banks when dealing with longer videos, more complex video content, or video motion?

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: I would like to understand the rationale behind first concatenating, then randomly selecting features for matching and updating during the feature bank update process. Were other methods tested in the experiments, such as retaining a specific proportion of past and current features for matching and updating? Would the performance change if most of the randomly selected features came from either past frames or the current frame?
soundness: 3
strengths: 1.This paper addresses a critical need in real-time video-to-video (V2V) translation, especially for streaming applications, which is highly relevant to the field. 
2.The paper is well-organized, with clear explanations of both the limitations in existing methods and the motivations behind the proposed backward-looking feature bank approach. The authors’ solution is explained in a logical and accessible manner, supporting reader understanding.
summary: This paper presents a study on real-time video-to-video (V2V) translation for streaming input using diffusion models. The authors identify limitations in batch-processing V2V methods, which are constrained by GPU memory and unsuitable for continuous, long-duration videos. In response, they propose StreamV2V, a diffusion model framework that integrates a feature bank, allowing it to maintain frame-to-frame consistency by leveraging stored features from past frames. This architecture enables StreamV2V to operate at 20 FPS on a single A100 GPU, significantly faster than existing V2V methods, and to support streaming inputs of indefinite length. Experimental results confirm that StreamV2V achieves competitive consistency and temporal stability, with favorable feedback from user studies compared to other V2V models. This paper is well-organized, and the methods and results are clearly presented.
weaknesses: 1.The model sometimes struggles to maintain temporal consistency when handling videos with significant camera or object movement, which may impact the overall quality.
2.It may have limited capacity for major visual transformations, such as altering complex object characteristics, which may restrict its applicability in highly dynamic editing tasks.

Meta-review: This paper proposed SteamV2V, a diffusion model that could do real-time streaming video-to-video translation with user prompts. Authors achieved this by maintaining a feature bank. The proposed solution is faster than previous work while maintaining temporal consistency.

7 reviewers unanimously thought this paper passed the acceptance bar. 

The strengths of this paper are: 1) motivation is clear and problem is important; 2) methods are flexible and suitable for creative applications and achieves good results; 3) experimental results are solid. 

weaknesses are: 1) using the feature bank to generate higher resolution could lead to a drastic increase in memory usage; 2) Some magic constants are reported without explanation; 3) The user study is presented without the necessary details; 4) The choice of SDEdit (which is 3 years old) should be explained -- why not newer methods; 5) There are still some artifacts in the result videos; 6) lags behind state-of-the-art video translation models in CLIP score and subjective evaluations; 7) The model sometimes struggles to maintain temporal consistency when handling videos with significant camera or object movement. 

During the rebuttal, authors addressed reviewers' concerns. Some reviewers raised their confidence or score.

Given these, AC decided to accept the paper.

Decision: Accept (Poster)