Looking Backward: Streaming Video-to-Video Translation with Feature Banks
Authors: Feng Liang
Akio Kodaira
Chenfeng Xu
Masayoshi Tomizuka
Kurt Keutzer
Diana Marculescu
Venue: ICLR 2025 Poster (2025)
Keywords: Streaming video translation
diffusion models
feature banks
Primary area: generative models
TL;DR: We present StreamV2V to support real-time video-to-video translation for streaming input.

Abstract: This paper introduces StreamV2V, a diffusion model that achieves real-time streaming video-to-video (V2V) translation with user prompts. 
  Unlike prior V2V methods using batches to process limited frames, we opt to process frames in a streaming fashion, to support unlimited frames.
  At the heart of StreamV2V lies a backward-looking principle that relates the present to the past. 
  This is realized by maintaining a feature bank, which archives information from past frames.
  For incoming frames, StreamV2V extends self-attention to include banked keys and values, and directly fuses similar past features into the output.
  The feature bank is continually updated by merging stored and new features, making it compact yet informative.
  StreamV2V stands out for its adaptability and efficiency, seamlessly integrating with image diffusion models without fine-tuning.
  It can run 20 FPS on one A100 GPU, being 15$\times$, 46$\times$, 108$\times$, and 158$\times$ faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. 
  Quantitative metrics and user studies confirm StreamV2V's exceptional ability to maintain temporal consistency.

Published as a conference paper at ICLR 2025
LOOKING BACKWARD: STREAMING VIDEO-TO-VIDBO TRANSLATION WITH FEATURE BANKS
Feng Liang"
Akio Kodaira 2 Kurt Keutzer UC Berkeley
Chenfeng Xu
Masayoshi Tomizuka
Diana Marculescu
UT Austin
[jeffliang, dianam) edu, fakio. kodaira, xuchenfenng&aerrkeeey
webcam
drawing stream
Draw rendering
Face swap
Stylization
stream
Edit prompt:
Elon Musk
Ni Smith
Doodle art
Claymation
Edit prompt: Sunset beach
Figure 1: We present Stream to support real-time video-to-video translation for streaming input. For webcam input, our Stream V2V supports face swap (e.g., to Elon Musk) and video stylization (e.g. to doodle art). Additionally, Stream provides drawing rendering capabilities, enabling iterative creation. We encourage readers to check our video results in the supplementary materials.
ABSTRACT
This paper introduces Stream V2V, diffusion model that achieves real-time stream- ing video-to-viden (V2V) translation with user prompts. Unlike prior V2V methods using batches to process a limited number frames, we opt to process frames in streaming fashion, to support an unlimited number of frames. At the heart of Stream V2V lies backward-looking approach that relates the present the past. This is realized by maintaining feature bank that archives information from past frames. For incoming frames, Stream extends self-attention to include banked keys and values, and directly fuses similar past features into the output. The feature bank is continually updated by merging stored and new features, making compact yet informative. Stream V2V stands out for its adaptability and efficiency, seam- lessly integrating with image diffusion models without fine-tuning. Stream V2V can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x faster than Flow CoDeF, Rerender, and TokenFlow, respectively. Quantitative metrics and user studies confirm Stream V2V's exceptional ability maintain temporal consistency. Demo, code, and models are available on the project page
INTRODUCTION
Text-driven iido-to-viieo (V2V) translation, which aims to alter the input video following given
prompts, has wide applications, such creating short videos, and more broadly in the film industry.
Most diffusion model based methods (Wu et al. 20236 Yang et al.; 2023 Ouyang et al.
2023
Wang
al. 2023a Khachatryan et al. 2023 Qi al.
2023 Zhang et al. 2023 Wang al.
2023b:
Chen

---

Published as a conference paper at ICLR 2025
et al. 2023 Zhao et al., 2023 Geyer et al. 2023 Liang al. 2023 Wu et al. 2023a Singer et al.
2024 use batches to process recorded videos, as shown in Figure 2 a). However, batch processing necessitates loading all frames into GPU memory, thereby limiting the video length they can handle, typically up to 4 seconds. Furthermore, these methods do not accommodate real-time translation and typically require several minutes process single 4-second clip.
This paper targets streaming V2V applications, such as webcam video translation and Al-assisted drawing, where users want to modify the streaming video iteratively. This necessitates the model to handle imput videos of varying lengths and perform real-time translation. To tackle this challenge, we introduce EtreamVV. an approach that processes frames in streaming fashion, as shown in Fig- ure bb). Leveraging advancements in one-/few-step dif- fusion models (Song al. [023 [auuer et al. 2023 Luo
Existing V2V methods
Stream V2V (ours)
(a) Batch processing
(b) Stream processing
ct al. 2023b). StreamDiftusion (Kodaira et al. 2023) has
Batch process: O(N) memory cost
designed pipeline for real-time interactive image genera- tion. However, directly applying StreamDiffusion for V2V tasks leads noticeable pixel flickering across frames. This is because StreamDiffusion treats each frame inde- pendently, disregarding the continuity of videos. In con- trast, humans implicitly memorize visual elements across frames and see the current frame links with past ob- servations. To generate consistent videos, it is critical to integrate a mechanism that can effectively bridge the current frame to its predecessors.
Memory cost
Our stream process: O(1 memory cost
A100 80G memory bound
2
Processing vidco longth (minutes)
(c) Memory consumption comparsion
Figure 2: (a) Existing V2V methods process frames in batches, restricting them to limited number of frames. (b) Our Stream V2V framework processes frames streaming fashion, can oper- ate on streaming videos in real-time. (c) Batch processing requires O(N) mem- ory for the video length N, whereas our Stream V2V only needs O(1) memory regardless of video length.
Recent studies have shown that diffusion features (Tang et al. 2024: Luo et al. 2023a) captured during U-Net's forward process contain rich correspondences between im- ages. Inspired by this, our StV maintains feature bank, which stores the intermediate features of past frames. For incoming frames, rating the corresponding stored keys and values. This can be interpreted as a weighted sum of similar regions across frames via attention, effectively aligning the current frame with previous frames. Additionally, to ensure the consistency of fine-grate ane-grained details, we directly fuse the block's 's output with similar features from past frames.
extend self-attention by incorpo-
The challenge then becomes: How can we implement this feature bank? A straightforward approach might store constant number of frames, such as employing a sliding window technique. However, this method is sub-optimal, as it inadvertently discards valuable data when a frame is omitted, and generates redundancy when the stored frames are similar. To address this, we propose to continuously update the bank by merging redundant features within incoming and stored features. This allows us to preserve the most representative features while keeping a consistent bank size over time. Through our experiment, we find the feature bank can be condensed to the size needed to store just one frame.
requires no training fine-tuning, making it compatible an add-on with any image diffusion models. It excels in efficiency, capable of processing high-resolution video (512x512) in real-time at 20 frames per second (FPS) on single A100 GPU. This speed surpasses current Vid (Liang et al., 2023), CoDeF (Ouyang et al. 2023), Rerender (Yang et al.) 2023), and TokenFlow (Geyer et al. 2023 by factors of 15x, 46x, 108x, and 1,58x, respec- tively. We evaluate our method with quantitative metrics, such as CLIP score (Radford et al. 2021) and warp eтror (Lai et al. 2018), and a user study. Our findings indicate that users significantly favor our Stream V2V over Stream (Kodaira et al. (203)) (with over 70% win rates) and CoDeF (Ouyang et al. 2023) (with over 80% win rates). While our method may not yet match the performance of state- of the (SOTA) methods like TokenFlow and Flow its rapid real-time execution opens up new venues for streaming V2V applications.
Stream
V2V methods-Fiie
Our contributions are three-fold: (1) To the best of our knowledge, our approach is the first approach to tackle real-time video-to-video translation for streaming videos. (2) Our method, Stream V2V,
2

---

Published as a conference paper at ICLR 2025
employs simple yet effective looking-backwee principle by maintaining feature bank to improve consistency. (3) We develop a dynamic feature bank updating strategy that merges redundant features, ensuring the feature bank remains both compact and descriptive.
RELATED WORK
2.1
VIDEO-TOVVDDR TRANSLATION
Significant progress has been made in the domain of text- sott-guiddd image-to- to-iiageeeee (12l) transla-
tion (Brooks et al., 2023 Hertz al., [222 Tumanyan al.] [223]] [ou et al. 2023), greatly
supported by large pra-te text-to-image diffusion models (Ramesh et al. 2022 Rombach et al.
2022 Saharia et al.) 2022). Similarly, video-to-video (V2V) translation, which
to generate
consistent videos, has attracted substantial interest as well. To generate coherent multiple frames,
most existing works (Esser et al. 2023 Wu al.) 2023b Wang et
2023a Guo et al.
2023
Chen et al. [223] Khachatryan et al. Ceylaan et al. |0233] Qi et al.
2023 Geyer et al. 2023
Wu et al. 2023aj Liang et al.) 2023] Ku et al. 2024) (Siggrr al. (024) process batches of frames simultaneously with cross-frame attention mechanisms. However, as the memory usage increases
with an increased number of frames, these methods are typically constrained to about 4 seconds
length. Additionally, they tend to rely on expensive DDIM inversion (Song et al. 2020 Qi et al.
2023; Geyer et al., 2023) or optical flow computation (Yang et al,, 2023; Liang et al. [2023), ) leading
to long processing time. In contrast, our Stream V2V handles videos in real-time and in streaming fashion, allowing for processing videos of any length. Compared to concurrent work Live2Diff (Xing
al. 2024) that requires additional video fine-tuning to train uni-directional temporal attention, our method is training-free and can serve as an add-on for any image diffusion model.
2.2 ACCELERATING DIFFUSION MODELS
While achieving great generation quality, diffusion models are commonly limited by their slow speed due to the need for multiple denoising steps. Recent advancements have introduced reusing cached
features in denoising steps Ma al. 2023 Wimbauer et al. 2023) and one-/few-step diffusion
models through distillation (Song et al. 2023 Meng et al. 2023
Luo et al. [2236]] Sauer et al. 2023
Yin et al. 2023 Lin Yang 2024). StreamDiffusion (Kodaira et al. (022)) proposes a pipeline to leverage these developments for real-time image generation. However, its application to video without adjustments brings unsatisfactory results. Leveraging StreamDiffusion' 's groundwork, we enhance frame consistency by implementing a backward-looking feature bank. Our approach introduces dynamic merging technique for the feature bank, ensuring it remains compact and incurs minimal additional computational cost in comparison to StreamDiffusion.
2.3 FEATURE BANKS
Long-term feature banks (Wu et al. 2019 Pan et al. (021) have been used in video understanding as supportive context features help reason the entire video. Adapting this concept, we employ feature banks to enhance video generation consistency. To ensure our feature bank remains both informative and compact, we introduce a dynamic feature merging strategy. While token merging (Bolya et al.) 2022; Bolya & Hoffiman| 2023) is common method for merging similar features within single images, our approach extends this technique across video frames, differentiating it from traditional within-image operations.
BACKGROUND: STREAMDIFFUSION
StreamDiffusion (Kodaira et al. 2023) leverages the Latent Consistency Model (LCM) (Luo et al. 2023b) to implement stream batch strategy, enabling the real-time generation of images. Instead of waiting for one image to be entirely denoised (usually 2-4 steps for LCM), stream batch reformulates sequential denoising steps into batched processes. This allows simultaneous processing of S images at varying denoising steps, where S is the number of denoising steps. For instance, at a given timestep t (assuming t S), StreamDiffusion first encodes the image I, with variational autoencoder (VAE) and adds a certain level of noise. We denote encoded latent as 25 where the subscript, denotes the frame timestep and the superscript denotes the denoising step. StreamDiffusion processes latent s+1) Upon advancing timestep 1, the model outputs the
denoising batch {2g :
S-1
3

---

Published as a conference paper at ICLR 2025
Feature
Input frames
bank
Output frames
EA: Extended self-Attention
Feature Fusion
Fetch
transformer input Xr
Feature bank
block out Feature bank
EA
EA
FF
0.
Update
Matching w/ similarity
save
V
Qc
Kg
Kyn
V/D
4
Fetch
Tbb
EA
EA
FF
Scaled Dot-Product Attention
Weighted
c+1
Update save
l++1
transfomeer output
fused block
Figure 3: Overview of Stream V2V. Left: StreamVVV connects the current frame to the past by maintaining a feature bank that stores the intermediate transformer features. For new incoming frames, Stream fetches the stored features and uses them by Extended self-Attention (EA) and direct Feature Fusion (FF). Middle: EA concatenates the stored keys K fs and values Vys directly that of the current frame in the self-attention computation (Section 4.1). Right: Operating on the output of transformer blocks, FF first retrieves the similar features in the bank via a cosine similarity matrix, and then conducts weighted sum fuse them (Section 4.2 The update method of the feature bank is elaborated in Section 4.3
final latent =0 2f-5+1- which is then decoded into the output image f((-S41)- The remaining latent would be denoised one step further, and the latent = from the new image I would be added to the batch. We include the diagram of the stream batch (Figure [2) in Appendix|A. A StreamDiffusion kick-starts the process by initializing the batch with S identical starting images, enabling warm start. To further accelerate the inference, SteamDiffusion also utilizes the Tiny autoencoder (madebyollin, 2023) and TensorRT (NVIDIA 2024 acceleration.
4 STREAMV2V
While being real-time, directly applying StreamDiffusion to video-to-video generation tasks brings unsatisfactory flickering results because each frame is generated independently. Built upon Stream]>- iffusion, we introduce a kackward-looking mechanism so that the generation of the current frame can reason about the past bring a consistent output. This realized by maintaining a feature bank storing the information of past frames. As shown on the left of Figure 3] StreamVVV fetches the stored features in the bank to process the frame It. We introduce two training-free techniques to leverage the stored features, namely Extended self-Attention (EA) (Section (.11 and direct Feature Fusion (FF) (Section 4.2) Lastly, we discuss how to update the compact but informative feature bank by dynamically merging the stored and newly incoming features in Section 4.3
4.1 EXTENDED SELF- ATTENTION
We mainly use Stable Diffusion (Rombach et al. 2022), which is built upon the U-Net architecture. The model contains multiple encoder and decoder blocks. Each block comprises residual convolu- tional unit and a transformer module, the latter including self-attention layer, a cross-attention layer, and feed-forward network. While it's been a common practice to inflate the self-attention layer to cross-frame attention in image diffusion (Hertz et al. [023 Tewel et al., 2024 Zhou et al. (0224) or
video diffusion methods (Ho et al. 2022 Wu et al. 2023b Khachatryan et al. 2023] Ceylan et al..
2023 Qi et al. 2023 Liang et al. 2023). these approaches require processing all the frames at the
same time in a batch. We extend the self-attention to accommodate the feature bank, which stores past frame information. As shown in the middle of Figure for given video frame It obtain the projected queries Q, E Rxo keys K, E [ and values V, n from the intermediate transformer input 24. Here, n and d denote the number and dimension of feature tokens, respectively.
4

---

Published as a conference paper at ICLR 2025
Denoting yt RTxd, and Vy Rmxd stored keys and values from previous frames, where m indicates the size of the bank, we can write the extended self-attention:
kvl Q2 (Ke, (0.
ExAttn softmax
[Ve, Vyal
(I)
Vd
[)] and Tr denotes concatenation and transpose operation. Essentially, this extended self-attention functions as a weighted sum of similar areas across frames, thereby aligning the current frame with its past for improved consistency. (More illustrations can be found in Appendix[A.22 We extend all
the self-attention layers with all denoising times. Further details on updating the feature bank are included in Section
4.2 FEATURE FUSION
While extended self-attention offers significant improvements in consistency, it operates implicitly through attention. We further introduce an explicit strategy for enhancing fine-grained consistency by directly fusing features based on correspondence. This is motivated by recent findings that diffusion features (Tang et al. 2024 Luo et al., 2023a) during the U-Net forward process contain rich correspondences between images. As shown on the right of Figure 3 for a given video frame 4,, we obtain the output features for the intermediate blocks O₂ E IR 1x and we also maintain the output features of past frames On E Rm For the token at position p in Oz, we seek the closed token at position q in Os.. utilizing cosine similarity as described by Tang et al. (2024). We denote (r(p) as selecting the token p from O and Oj the fused output feature:
(p) = (1 (1- O.(p) (p(o)) where q arg max (O.(p) 07,))
where a is hyperparameter identify the strength of fusion, which is usually set to 0.75. Intuitively, this direct feature fusion aims to enhance consistency by merging similar regions from past frames to the current frame. In some cases, the current frame introduces novel regions which are absent in past frames. To prevent misalignment, we generate a mask based on a predefined similarity threshold. Specifically, we generate the mask by caleulating the cosine similarities between each feature in the current output and all features stored in the bank. For each feature, we identify the most similar stored feature and calculate the similarity score. If the score is lower than a predefined threshold (set at 0.9), it indicates that we cannot find a suitable match in the feature bank. In such cases, we mask out this position, meaning this feature remains unchanged and not fused with others from the bank. Our analysis further indicates that the location of feature fusion across various network blocks significantly impacts overall performance. Specifically, we observed that limiting feature fusion to the low-resolution decoder blocks results in optimal performance enhancements. For comprehensive comparison and further insights, refer to Appendix A.3
4.3 UPDATING THE FEATURE BANK WITH DYNAMIC MERGING
Bank
We denote all transformer intermediate features of frame I as T, = K,, Vz, Ot), where K, and Vú are projected keys and values of self-attention input, O, is the output of transformer block. An initial approach for creating a feature bank is to store features from a constant number of frames. As depicted in Figure 4 (a), after deciding the maximum number of frames (which is set 2), the bank operates like a queue: as new frames arrive, the oldest frames are popped out to make space for the newcomers. Yet, this method encounters two primary limitations: (1) the continuous removal of the oldest frames restricts the feature span to brief temporal window, and (2) the redun- dant features in the bank incur extra storage and processing costs.
Append
Append
Append
DyMe
Append
DyMe
Append
DyMe
(a). Naive queue
(b). Dynamic merging
Figure 4: Naive queue our dynamic merging (DyMe). DyMe has a more compact and informative feature bank.
To create a compact and informative bank, we propose to dynamically merge (DyMe) the stored features and newly
5

---

Published as a conference paper at ICLR 2025
coming features. As shown in Figure 4 (b), at timestep 2, merge T1 and T2 into (1.2), which has the same size as Ti. By applying DyMe, we can the get condensed bank T1122,3.4) which contains the information for all 4 frames, yet occupies only half the space of naive queue. We follow the efficient bipartite matching technique (Bolya et al. 2022 Bolya & Hotfman] 2023) to do the merging. In more detail: (1) first concatenate features of the current frame (green boxes) and features stored in the bank (gray boxes); (2) we then randomly partition the features into two sets source (sre) and destination (dst) of equal size: (3) For each feature vector fare in the src set, we identify the most similar feature vector fdu in the dst set using the cosine similarity metric defined as follows: simefare fas) = Ihmillaa fas In this expression, represents the dot product, and | denotes the norm of vector. (4) Once we find the most similar features fas of fan, proceed merge the features from src into dst. This is achieved by averaging the values of each matched pair: Luct This step ensures that the features in dst are updated to reflect a blend of both the original and the matched features from Our experiments show that our dynamically merged feature bank brings better performance-speed trade-off than the naive queue-based bank. Empirically, we find the feature bank can be condensed to the size of storing just one frame. For more details, please refer Section 5.4.2 We also visualize the effect of DyMe in Appendix A.5.I
5
EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
We built our method on Stream (Kodaira et al. 2023) with Latent Consistency Model (Luo al. 2023b). By default, we use a 4-step LCM without the classifier-free guidance (Ho & Salimans 2022). We update the feature bank every 4 frames. The underlying image-to-image method is
SDEdit (Meng al. (021), with an initial noise strength of 0.4. Unlike some methods (Yang et
2023 Liang et al. (0233) which use frame interpolation to generate high FPS video, our Stream V2V
generates all frames the same pipeline.
Following TokenFlow (Geyer et al. 2023) and Flow (Liang et al. 2023). we build our user study by selecting 19 object-centric videos from the DAVIS trainval 2017 dataset (Pont-Tuset et al. 2017] covering diverse subjects such as humans and animals. We reuse 67 prompts from (Liang et al. 2023), ranging from stylization to object swaps, for these videos. We conduct a thorough comparison with statc-of-the-art V2V methods such as Rerender (Yang et 2023), CoDeF (Ouyang et al. 2023), TokenFlow (Geyer et al. 2023), and Flow (Liang et al. (023). utilizing their official codes under default settings. We report both qualitative comparison 5.2 and quantitative metrics [5.3] such as CLIP score (Radford et al. 2021), warp error (Lai et al. 2018). and user preference, to verify the effectiveness of our method.
5.2 QUALITATIVE RESULTS
In Figure 5 we qualitatively compare our Stream with several representative V2V methods, starting with our per-frame baseline StreamDiffusion (Kodaira et al. (023). which treats each frame independently. StreamDifffssion often results noticeable flickering, such as the background flowers and the dancer's legs. CoDeF (Ouyang et al. (203)) tends to produce outputs with significant blurriness, especially when there a big motion within the input video, which fails in the construction of the canonical image. Rerender (Yang et al. 2023) fails to keep tracking the clothing color in the dance, which fluctuates between brown and blue. TokenFlow (Geyer et al. 2023) occasionally struggles to follow the prompt, such transforming the video into pixel art. Flow Vid (Liang et al., 2023) maintains good prompt alignment but similarly struggles with color consistency in clothing and suffers from blurriness in the final frame. In contrast, our method stands out in terms of editing capabilities and overall video quality, demonstrating superior performance over these methods. Moreover, our StreamV22 V2V can handle arbitrary lengths of videos showcased in the Appendix A.6
5.3
QUANTITATIVE RESULTS
5.3.1
EVALUATION METRICS
CLIP score Following previous research (Geyer et al.) 2023 Liang et al.) 2023). we first utilize CLIP (Radford al. 2021) to evaluate the consistency of generated videos. Specifically, we first get

---

Published as a conference paper at ICLR 2025
StreamDiifusion (Per-frame)
StreamV2V
Input video
CoDeF
Rerender
TokenFlow
FlowVid
OUrss)
Figure 5: Qualitative comparison with representative V2V models. Prompt is A pixel art of a man doing handstand on the street', Our method stands out in terms of prompt alignment and overall frame consistency. We highly encourage readers to refer to video comparisons in our supplementary videos.
Table Quantitative metries comparison. We report the CLIP score and warp error to indicate the consistency of generated videos. We bold the best result and underline the second best.
StreamDiffusion
CoDeF
Rerender
TokenFlow
Flow
Stream (ours)
CLIIP score f
95.24
96.33
96.20
97.04
96.68
96.58
Warp error
117.01
116.17
107.00
114.25
111.09
102.99
the CLIP image embeddings for all the video frames, then we measure the mean cosine similarity across all sequential frame pairs. Our evaluation, detailed in Table includes an analysis of 67 video-prompt pairs from the DAVIS dataset. TokenFlow shows superior performance in maintaining temporal consistency which aligns with the findings from our user study. StreamDiffusion. which is per-frame image model, achieves the worst score. Our Stream ranks in third place regarding CLIP score. Taking the consideration that our Stream V2V is a real-time stream processing method for unlimited length videos which is significantly faster than these batch processing, limited length video V2V methods (Section 5.3.2). our method is shown to have a good performance-speat trade-off.
2018 measure of temporal consistency following previous research (Ceylan et al.) 2023 Geyer et al. 2023). We first compute the optical flow between consecutive frames input video using pre-trained RAFT flow estimator (Teed) & Deng 2020). Then we warp the frame in the transferred video to the next using the estimated flow. This allows for the evaluation of pixel discrepancies between the warped frame and the target counterpart. We calculate the average mean squared pixel error over the un-occluded regions as warp emror. As shown in Table [ our StreamV2y V2V achieves the best warp error among methods.
Warp error We also propose
warp LLi
5.3.2 PIPELINE RUNTIME
benchmark the runtime with a 512 512 resolution video containing 120 frames (4 seconds
video with FPS of 30) in Figure 6 For Stream we conducted ten runs, discarded the top and bottom two results, and averaged the remaining six. The runtime for Flow CoDeF, Rerender, and TokenFlow is taken from the Flow paper (Liang al. 2023). Flow Vid (Liang al. 2023) and Rerender (Yang et al. 2023) first generate key frames, then use frame interpolation methods (Jamriska 2018 Huang al. 2022 to generate non-key frames. We choose the keyframe interval of 4 for
7

---

Published as a conference paper at ICLR 2025
StreamV2V inference
15.8
Streum V2V
Stream V2V
15
Flow Vid key-frame
draw
loses
FlowVid fiume interp.
CoDeF train
5.4
71.1
25.3
3.0
StreamDift
Time (minutes)
CoDeF inference
10.8
10
Rerender keg-frame gen.
81.3
5.5
CoDeF
Rerender interp.
DDIM
5.6
Stream V2V
30.2
31.4
38.4
Rerender
TokenFlow inférence
10.4
4.6
19.8
21.3
58.9
Flow
19.3
13.4
67.3
TokenFlow
s9c
1,2
1.5
3.5
5.2
50 Win rate (%)
100
Stream (eV(rs) Vlow
CoDeF
Rerender
Figure 7: User study comparison. The win rate indicates the frequency our StreamV2V preferred compared with certain counterpart.
Figure 6: Runtime breakdown on one A100 GPU of generating 4-second 512x512 reso- lution video with 30 FPS.
these two methods, following (Liang et al. 2023). Two other methods, CoDeF (Ouyang et al. 2023) and TokenFlow (Geyer al., (023). both require per-video preparation. Specifically, CoDeF involves training a model for reconstructing the canonical image, while TokenFlow requires a 500-step DDIM inversion process to acquire the latent representation. Unlike all these methods requiring two-stage processing, our Stream V2V produces all the frames in single stage. We continue to use xFormers (Lefaudeux al., 2022) for fair comparison with existing methods. Our Stream V2V processes videos in 9 seconds with the 4-step LCM, making it significantly faster than current approaches. We note that while existing V2V methods use A100-80GB GPU for runtime measurement, we only use an A A100-40GB GPU. Despite this, Stream remains significantly faster. Further acceleration possible with fewer steps, though this may slightly affect performance. For more details, see Appendix A.4
5.3.3 USER STUDY
We further conducted TokenFlow [Geeer et al. (2023) and Flow Vid Liang et al.] (2023), we use the DAVIS dataset PPnn Tuset al. (2017) with 67 video-prompt pairs. We adopt a Two-alternative Forced Choice (2AFC) protocol used in (Yang et al. 2023 Geyer et al.) 2023), where participants are shown our result and counterpart result, and are asked identify which one has the best quality, considering both temporal consistency and text aligmment. For each comparison, feedback was gathered from at least three different participants. The final win rates are shown Figure 7 We find users significantly favor our Stream V2V over StreamDiffussin baseline (over 70% win rates) and CoDeF (over 80% win rates). important note that while our method does not outperform the more advanced V2V methods such as Rerender, Flow and TokenFlow, Stream V2V primarily aims to achieve real-time video transfer, rather than beating the state-of-the-art V2V methods, none of which support real-time processing and unlimited video length. Moreover, videos as showcased in Appendix A. while Flow Vid and TokenFlow are limited to 4 seconds due to the memory limit (illustrated in Figure Cc). We have made all videos from the user study available on our project page for further examination. More details about the user study can be found in Appendix A.7
study to compare method with five notable works. Following
Stream V2V deal with half-minute long
5.4 ABLATION STUDY
We ablate several of our key designs in this section. The evaluation set contains 18 in-house videos with cartoon-style prompts.
5.4.1 EXTENDED SELF- ATTENTION (SA) AND FEATURE FUSION (FF)
As shown in Figure the StreamDiffusion baseline, which doesn't have EA FF, produces inconsistent outcomes. Noticeable flickering can be observed in the man's hands, hair, and the strips on his clothes. After introducing the EA (the third column), the consistency is much improved, with
8

---

Published as a conference paper at ICLR 2025
the warp decreased from 85.2 to 74.0. However, some artifacts persist in the man's san's hand. After further introducing FF (the last column), we have the most consistent result, with the lowest warp error of 73.4. We also isolate the effect of FF in the fourth column, lower the warp error from 85.2 to 80.4 with the introduction of FF
wio EA, wia FF Warp Error: 85.2
wi EA, wio FF Emor: 74.0
wio EA, w/ FF Warp Frror: 80.4
wi EA, wi FF Fovor: 73.4
Table Ablation on different feature banks, No bank indi- cates the StreamDiffusion base- line. Queue represents the naive first-i-ffirt-oou queue, Our Dy- namic Merging (DyMe) main- tains a compact and informative bank.
Input video
Bank type (size)
Time (ms)
Mem (GiB)
Warp error
bank (0)
60
3.68
85.2
queue (1)
70
4.03
76.4
queue (2)
79
4.17
74.8
queue (4)
100
4.77
72.4
Figure 8: Ablation on Extended self-Attention (EA) and Feature
DyMe (1)
75
4.07
73.4
Fusion (FF). Warp error is averaged within 18 videos.
wio foet bank
(b). Fext bank
nasterplece man yetlow] suit. Its holding standing 1n front bookshelf.1
crown
head
Figure 9: Continuous text-to-image generation with feature bank. The images from left right are the intermediate generation as we type continuously. The caption of each column image is segment of the prompt (separated by the line). (a) Per-frame LCM baseline has severe flickering even with slight one two-word modification. (b) The feature bank provides a much smoother transition. We highly encourage readers to refer to video comparisons in our supplementary materials.
5.4.2 DYNAMIC MERGING (DYME) BANK
We study different feature banks in Table The case of bank, which represents the StreamDiffusion baseline, needs 60 ms to process a 112x512 frame on single A100 GPU. However, its warp error considerably high 85.2. For queue- based banks, we test sizes of 1, 2, and As expected, increasing bank size increases inference time but reduces warp error. Our dynamic merging (DyMe) bank achieves a running speed of 75 ms, faster than the queue bank of size 2, while maintaining lower warp error. We also report the memory usage obtained via the nvidia-smi command. The introduced DyMe bank only adds a very small 10% GPU memory overhead when compared to StreamDiffluion.
5.5 EXTEND FEATURE BANK TO CONTINUOUS IMAGE GENERATION
The concept of the proposed feature bank, which links the current with the past, is expected to be broadly applicable beyond video-to-video. To demonstrate its versatility, we validate its effectiveness in continuous text-to-image generation task.

---

Published as a conference paper at ICLR 2025
Edit prompts:
Edit prompts:
Input video
Pope
Input video
style
Van Gogh Sty le
(a) Fail edit the video as prompted
(b) Inconsistent output
Figure 10: Limitations of Stream V2V. (a). Stream fails to alter the person within the input video into Pope or Batman. (b). Str can produce inconsistent output, as seen in the girl for Anime style and the backpack straps for Van Gogh style.
Continuous or real-time image generation allows users to produce images instantly when the prompt being typed. However, most current frameworks generate each frame independently, resulting in flickering transitions between frames. Figure ([a) illustrates per-frame generation in the LCM (Luo ctt al. 2023b). The images from left to right reflect the intermediate results as we type continuously. Even with slight one or two words, the layout of the image can shift dramatically. By incorporating feature bank as in Figure b)), the model can reference previous frames, enabling smoother transitions.
5.6 LIMITATIONS
Despite achieving notable improvements, our model encounters certain limitations. First, our StreamVVV can fail to alter the video with the provided prompts. As shown in Figure [o]a), when try to transfer the man to Pope B Batman,
model struggles to change the input video.
This may be due to the use of SDEdit (Meng et al. 202T) as the primary image editing method, which has limited ability significantly alter objects. We also find that increasing initial noise strength can improve editability but at the cost of introducing more pixel flickering. Another limitation is that Stream V2V can produce inconsistent output, especially for videos with rapid movements of the camera or the object. This is also the major reason why our Stream V2V still cannot match the performance of state-of-the-at FlowVis and TokenFlow. As demonstrated in Figure (o),,, when converting the video into anime style, the appearance of the girl in the background changes significantly. Similarly, when applying Van Gogh style, there are noticeable changes to the backpack straps of the man.
CONCLUSION
In this paper, we present Stream V2V: a diffusion model that can perform real-time video-to-video translation for streaming videos. The key to Stream V2V lies look-backward mechanism that enables the current frame to reason the past to ensure consistency. We propose a feature bank to store the intermediate features for past frames and reuse them the current frame generation via extended self-attention and direct feature fusion. Furthermore, the bank compact and informative. Our evaluation shows that Stream V2V is an order of magnitude faster than existing iide-to-vvdeo methods and can produce temporally consistent outputs.
propose dynamic merging, method to make
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our research focuses on real-time video-to-video translation, aiming to unleash people's creativity for good applications. It is not designed to create harmful misleading content, such fraud or deepfake. However, like other related image/video generation models, it could still potentially be misused for deepfake humans. We strongly oppose any use cases that create fraudulent or harmful content using our model. Currently, the videos generated by our Stream V2V still contain human- perceivable artifacts, as shown our video demos. Stream V2V demonstrates the possibility of using diffusion models to achieve real-time texi-prompted video-to-video, but the overall editing performance is far from confusing people. However, when technology evolves, one day we may reach point where generated videos are indistinguishable from real ones. We, as community, need to investigate more research into detecting generated videos.
REPRODUCIBILITY STATEMENT
Our code is reproducible and can be implemented based on the method description Section 4 4as well as implementation details in Section 5.1 We have open-sourced our codes and models.
ACKNOWLEDGMENT
This research was supported in part by ONR Minerva program, NSF CCF Grant No. 2107085, iMAGINE the Intelligent Machine Engineering Consortium at UT Austin, and UT Cockrell School of Engineering Doctoral Fellowship.
REFERENCES
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp.
2023. Daniel Bolya, Cheng- Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 2022.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392 18402, 2023.
Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2023.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guice guided video synthesis with diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 7346-7356, 2023.
Michal Geyer, Omer Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10371 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint 0:Xi::2307.047255. 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint 22X.::200..01626, 2022.
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. arXiv preprint 1322 2023.

---

Published as a conference paper at ICLR 2025
Jonathan Ho and Tim Salimans. Classifier- frese diffusion guidance. arXiv preprint 1rX1v:2207.12598 2022.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03456 2022.
Zhewei Huang, Tianyuan Zhang. Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Ondrej Jamriska. Ebsymth: Fast example- based image synthesis and style transfer. github. wtt.co///jaaaa wacoo//jaaissiio http://amaiaaaabaaaaeeeee 2018.
https://
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Tex22vico-zeer: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 2023.
Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streamdiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint arXiv:2312.1249)) 2023.
Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play framework for any video-to-video editing tasks. arXiv preprint arXiv:2403. /4468, 2024.
Wei-Sheng Lai, Jin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming- Hssan Yang. Learning blind video temporal consistency. In Proceedings of the European conference computer vision (ECCV), pp. 170-185, 2018
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https://github... http///hhh..ccccccaaeeeeceeeeeee x oformers] 2022.
Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang. Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. arXiv preprint arXiv:2312.17666 2023.
Shanchuan Lin and Xiao Yang. Animatedif-lighhning: Cross-model diffusion distillation. arXiv preprint 1XX1:22403.12066 2024.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence, In Advances in Neural Information Processing Systems, 2023a.
Simian Luo, Yiqin Tan, Longho Huang. Jian Li, and Hang Zhao. Latent consistency models:
Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04377. 2023b.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. arXiv preprint arXi::2312.00858, 2023.
madebyollin. Tiny autoencoder for stable diffusion. Eaesd 2023. GitHub repository.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0 01077, 2021.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14297-14306 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08455 2023.
12

---

Published as a conference paper at ICLR 2025
NVIDIA. Tensont: High performance deep learning inference library. httpa://developet nvidia. com.l//sccttttt.rtttt......... 2024.
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. arXiv preprint 23X.::23088.792666 2023.
Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li. Actor-context- actor relation network for spatio-temporal action localization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 464-474, 2021.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:/704.000 00675, 2017.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint N3V::2333.09335 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprini arXiv:2311. 2023.
Uriel Singer, Amit Zohar, Yuval Kirstain, Shelly Sheynin, Adam Polyak, Devi Parikh, and Yaniv Taigman. Video editing via factorized diffusion distillation. arXiv preprint arXiv:2403.09333 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. Computer EssionEECVV 2020: 16th European Conference, Glasgow. UK, August 23-28. 2020. Proceedings, Part 1l 16, pp. 402-419 Springer, 2020.
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-fii free consistent text-to-image generation. arXiv preprint arXiv:2402.03286 2024.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-pla diffusion features for text-driven saage-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
13

---

Published as a conference paper at ICLR 2025
Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang. and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17999 2023a.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.0222 02018, 2023b.
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang. Sam Tsai, Jonas Kohler, al. Cache me if you can: Accelerating diffusion models through block caching. arXiv preprint arXiv:23/2.03209, 2023.
Bichen Wu, Ching- Yao Chuang, Xiaoyan Wang, Yichen Jia, Kapil Krishnakumar, Tong Xiao, Feng Liang, Licheng Yu, and Peter Vajda. Fairy: Fast parallelized instruction-guider video-to-video synthesis. arXiv preprini arXiv:2312.33825 2023a.
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 284-293, 2019.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 7623-7633, 2023b.
Zhening Xing, Gereon Fox, Yanhong Zeng, Xingang Pan, Mohamed Elgharib, Christian Theobalt, and Kai Chen. Live2diff: Live stream translation via uni-directional attention video diffusion models. arXiv preprint 2xXiv:2407,08701, 2024.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video: Zero-shot text-guided video-to-video translation. arXiv preprint arXiv:2306.07954 2023.
Tianwei Yin, Michaël Gharbi, Richard Zhang. Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprini arXiv:2311.1883 188828 2023.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023.
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. Controlvideo: Adding conditional control for one shot text-to-video editing. arXiv preprint arXiv:2305.17996 2023.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Con- sistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01432 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.I DIAGRAM STREAM BATCII
VAE encodeing adding noise
Denoising batch
VAE decoding
Output images
1
2
@
l
1 1
2
i-1
2
1
Input images
1
11
Figure 11:
Stream batch of StreamDiffussin The approach enables the processing of images with different denoising steps in batch (S is set to 3).
A.2 HEATMAP VISUALIZATION OF EXTENDED SELF-ATTENTION
Given an anchor point (marked in red) in frame 4, we extract its query tensor from the last layer of the UNet up-block. We also obtain and store key tensors from the preceding frames 1, 2, and
3. We then calculate the dot product between the anchor query and stored key tensors. The most similar position is marked by the red point in frames in frame 4 is highly correlated with the same goggle spot in earlier frames, indicating that extended self-attention functions as weighted sum of similar areas across frames, thereby aligning the current frame with its past for improved consistency.
and This shows that the anchor point
Frame
Frame
Frame
Frame 4
Heatmap of extended self-attention. The anchor point (marked in red) in frame has a very high correlation with the same goggles spot in previous frames.
Figure 12:
A.3 DETAILS FEATURE FUSION
A.3.1 FEATURE FUSION VISUALIZATION
To further verify the intuition of feature fusion, we visualize the output feature from the last layer of the UNet up-block. begin by concatenating the features from all frames and applying principle component analysis (PCA) for visualization. The second row of Figure 13 shows the visualization of the first three principal components. We observe that similar concepts tend to appear in similar colors, indicating their similarity in feature space. For example, features corresponding to the goggles (blue box) and pants (red box) exhibit consistent coloring across frames, highlighting the potential of feature fusion average them.
15

---

Published as a conference paper at ICLR 2025
Input frame
Feature after PCA
Frame
Frame 2
Frame 3
Figure 13: Feature Fusion Illustration. For input frames, PCA is applied to diffusion features extracted from all frames, and the first three principal components are visualized (second row). Features like the goggles (blue box) and pants (red box) show similar colors across frames, indicating consistent features.
A.3.2 FEATURE FUSION POSITIONS
While feature fusion significantly enhances temporal consistency, it may result in blurry artifacts. As demonstrated in Figure 14 incorporating feature fusion in the layers makes the cat's face appear blurry. We believe that doing feature fusion high-resolution features would average the features across frames, resulting in diminishing of details. Thus, we suggest applying feature fusion only to low-resolution features, specifically within the middle block and the first blocks of the upper block. As displayed in the right column of Figure [4] this approach preserves higher- details compared using all layers.
Input video
FF in all layers
FF in mid-up01
Figure 14: Ablation on feature fusion (FF) layers. Applying FF to layers sometimes results in blurry artifacts, as seen in the cat's 's face. We propose only apply FF to low-resolution layers, specifically the middle block (mid) and the first two blocks the upper block (up01).
16

---

Published as a conference paper at ICLR 2025
denoising sleps 75 per frame
demoiising steps per frame
demoising step per frame
denoising steps per frume
2 denoising steps 50 ms per
denoising step per frame
Input video
Edit pronots:
Elon Nusk
Edit pronpts: Claynation
Figure 15: Ablation on different denoising steps. While using fewer denoising steps would accelerate the inference time for every frame, we do observe a certain level quality drop if use only 1 step.
Input frame
After dynamic merging
Frame
Frame
Frame
Frame
Figure 16: Visualization of Dynamic Merging. Similar patches across frames merged, showcas-
ing that the DyMe bank can maintain a compact yet informative feature bank.
A.4 DETAILS OF DIFFUSION STEPS
As outlined in Section 5.1 our Stream V2V utilizes LCM (Luo et al. 2023b) as the accelerated diffusion model. LCM perform denoising in 4, or 1 steps, and we examine these variations in Figure [5] Reducing the number of steps decreases the inference time per frame from 75 ms to 41 ms. However, this also leads decline in video quality. Unless otherwise specified, we use the 4-step LCM to achieve optimal performance.
A.5 MORE DETAILS DYNAMIC MERGING
A.5.1 VISUALIZATION OF DYNAMIC MERGING
As discussed in Section 4 the Dynamic Merging (DyMe) bank merges redundant information across time. Figure 16 illustraass this effect. For better visualization, we applied the merging operation 10 times to the output features from the last layer of the UNet up-block, where merged features
17

---

Published as a conference paper at ICLR 2025
marked in the same color. We observe that key elements, such as the person, sky, and snow, grouped consistently, demonstrating the effectiveness of dynamic merging.
A.5.2 PSEUDO CODE OF DYNAMIC MERGING
import torch import torch.nn. functtioal as
def dynamic_merge (current_ fcature_bank):
Dynamic merging (DyMe) to create compact feature bank.
Args:
current frame (Tensor): Features of the current frame (shape: IN, DIJ.
feature bank (Tensor) Existing feature bank (shape: IN, DJ).
12
Returns:
Tensor: Updated feature bank with dynamic merging.
Step Concatenate current frame features and feature bank all features torch.cat ([current_frame, feature_bank] dim-0) Shape: [(2,, DJ
Step 2: Randomly partition features into source (src) and destinal nattionion (dat)
19
num num_features = a11_features.sizeel
30
permuted_indices - torch. barchhrandpern (nuu_features)
sre indices = permuted indiuteeedddiieiiiiiiiiie inum features // 21
dst indices permuted indices num features // 2:1
23
src_set all features srrc [sre_ indices] det_set all_features [dst_indices
Shape: (N, D]
24
$ Shape: [N, D]
25
step 3: Find the most similar features between and dst Compute cosine similarity
28
similarity_matrix lecosine_imiiarrite (src_set. (ttp://ssssstt.suueeeeee) dst_ .unsqueeze(0), din-2)
29
max mwx_similaritiss, indices similarity.matrix max (dim=l)
31
Step 4: Merge sre into dst by averaging matched pairs for i, match_ idx in enumerate (matched_indiccss] dst_set [match_idx] (dst_set [match_idx] src_set [i]) /
32
33
updated feature bank dst set return updated feature bank
36
A.6 MORE RESULTS ON LONG VIDEO
While most existing methods typically handle up seconds of video, StreamV2V can scale to arbitrary lengths thanks to the streaming processing and feature bank. As demonstrated in Figure [7] our approach handles video exceeding 1000 frames (over 30 seconds) while maintaining consistent face swapping or style transfer throughout.
A.7 MORE DETAILS OF USER STUDY
conducted user study to compare our method with existing approaches. Figure 18 illustrates the interface presented to each participant. The instructions given to users were follows: The left/right video is labeled A/B. You will be asked three questions to compare these two videos. If you perceive minimal differences between the videos, whether they are both good bad, please select 'draw".
As described in Section 5.3.3
18

---

Published as a conference paper at ICLR 2025
Claymation Elon Musk
Ml
Frume i O
Frume # 250
Frume // 450
Frame @ 650
Frame // 1000
Figure 17: Long video (> 1000 frames) generation. Our Stream can handle arbitrary length of videos without consistency degradation.
Video Comparison
beeakdance- dare. Impresiinist_ Peressionst___mpp_ Lipp4
Trange An aypeetienn Li
CAO B.G
CAOK Cre
Figure 18: User study Interface. Each user is asked three questions about temporal consistency, prompt alignment and overall preference.
The participants all college students aged 20-30. The ratio of male: female participants is roughly 2:1. The participants don't have hands-on rescarch/engineerinc may have heard of the concept of generative AI. We find that user preference very subjective and there are huge variances between users. We report the mean and standard deviation among at least three users Table
experience with generative videos but
It is worth noting that, instead of providing numerical metrics for comparison, the study aims to offer a first-impression comparison between the two models. The results, whether indicating one model is better, worse, par with another, are generally consistent. This information can guide our aecision-making particularly when we need to balance speed and accuracy in model selection. In practice, recommend using Stream V2V for applications where speed is critical requirement, such as real-time webcam translation and draw rendering. For other uses, such as creating short video content, slower methods like Flow might yield better results.
Table 2: Comparison of Stream against various models. We also report the mean and standard deviation (std) for each set of users.
Strcam V2V wins (mean + std)
Draws (mean + std)
Stream V2V loses (mean + std)
Model
StreanDiffusson
71.1 1: 10.2
25.3 1 13.3
3.6 1 5.2 5.5 + 9.3
CoDeF
81.3 = 20.1
13.2 12.1
Rerender
30.2 15.1
31.4 7.8
38.4 18.9
vs. Flow Vid
19.8 = 9.5
8.5
58.9 18.0
Ps. TokenFlow
19.3 + 9.0
13.4 9.7
67.3 13.4
19

---

Published as a conference paper at ICLR 2025
A.8 MORE ANALYSIS AND ABLATION STUDIES
A.8.1 MEMORY CONSUMPTION WITH LARGER VIDEO RESOLUTION
We tested the memory footprint under different resolutions with one 24-GB A5000 GPU. As shown in 3 we can run Stream V2V up to a high resolution of 1536 x 1536. However, we did see steady increase in the relative memory overhead of the DyMe bank. We conjecture the reason for this increase is that the baseline diffusion pipeline has specific optimization regarding memory usage, while DyMe implementation doesn't t.. Some potential solutions are (1) storing features with key layers rather than all the layers; and (2) implementing memory optimization techniques for the DyMe data structure. We will consider these in our future work.
Table 3: Memory Usage Comparison. We report the memory usage (in GiB) of the baseline and Stream V2V across different resolutions, along with the relative memory overhead increase.
Resolution
Mem of Baseline (GiB)
Mem of Stream (GiB)
Relative Memory Overhead Inerease
512 512
4.87
5.34
+ 10%
768 768
5.59
7.29
+ 30%
1024 x 1024
6,62
9.86
+ 49%
1536 x 1536
9.52
22.47
+ 136%
A.8.2 SIMILARITY THRESHOLD OF FEATURE FUSION
We ablate the effect of similarity threshold in feature fusion as in Table 4 The similarity threshold of 0.9 yields the lowest average warp error of 19 videos. We have worse results when the threshold is set to 1.0 (no features are fused) or 0.0 (all features are fused).
Table 4: Effect of similarity threshold. We analyze the impact of varying similarity thresholds on the average warp error videos. The best result, achieved with our default value of 0.9, is highlighted.
Similarity Threshold
Warp Error +
0.0
74.0
0.8
73.5
0.9 (default)
73.4
1.0
74.0
A.8.3 BANK UPDATING INTERVAL
We ablate the effect of bank updating intervals Table 5 The interval brings us the best performance. During practice, may want to adjust this value according to our videos: videos with larger motion may require a more frequent update.
Table 5: Effect of bank updating interval. We report the average warp error of videos for different bank updating intervals. The best result and method are highlighted.
Bank Updating Interval
Warp Eror
78.1
75.3
(default)
73.4
72.4
16
72.5
A.8.4 SAMPLING STRATEGY OF DYME UPDATING
Our DyMe bank needs to derive source (src) and destination (dst) sets from current and stored past features. We ablated three choices in Table 6 (1) Randomly sample (our default setting): (2) Uniformly grid sample: After concatenation, features at even locations are used as the source, and features odd locations are used the destination; and (3) Split sample: Use the current features as
20

---

Published as a conference paper at ICLR 2025
Table 6: Effect of sample strategy. We evaluate warp error across different sampling strategies. The best result is highlighted.
Sample Strategy
Warp Error +
73.4 Uniform Grid
Random (default)
73.7
Split
74.2
the destination, and the stored past features as the source. Split sampling has the worst warp erros, performs more like queue bank, in which the current features dominate the bank. Compared to uniform grid sampling, random sampling proved more general and achieved the lowest warp error. Thus, random sampling was chosen the default strategy.
21


Reviews:
[{"review_id": "woL1cJRxpF", "reviewer": "Reviewer_cUnp", "summary": "This paper presents StreamV2V, a real-time streaming video-to-video (V2V) translation method leveraging a backward-looking feature bank to maintain temporal consistency across video frames. Unlike traditional batch-processing approaches, StreamV2V processes incoming video frames in a streaming fashion, which allows it to handle an unlimited number of frames without fine-tuning. The method introduces a feature bank that stores intermediate features from past frames, extended self-attention, and direct feature fusion to improve temporal consistency. Furthermore, a dynamic feature bank updating strategy is implemented to reduce redundancy and optimize efficiency. The experimental results show that StreamV2V can achieve real-time processing at 20 FPS on a single A100 GPU, significantly outperforming existing methods like FlowVid, CoDeF, Rerender, and TokenFlow in terms of speed, with competitive consistency performance.", "questions": "- What is the size of the feature bank used in the experiments? Does the size of the feature bank affect the performance of StreamV2V?\n- Can the proposed method be applied to previous text-to-video approaches to improve the performance of long video generation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation behind StreamV2V is clear and addresses an essential limitation in existing video-to-video methods that require batch processing and are constrained by GPU memory.\n- The implementation is designed for practical efficiency, with a clear description of feature fusion and dynamic feature bank updates, ensuring minimal memory overhead and ease of replication.\n- The method provides flexible control over generating different styles without requiring model fine-tuning, allowing users to adapt the output to various visual aesthetics, which makes it suitable for creative applications.", "weaknesses": "- A significant concern is that using the feature bank to generate higher resolution could lead to a drastic increase in memory usage.\n- The DyMe (Dynamic Merging) mechanism lacks detailed pseudocode to illustrate the workflow, which would help in understanding and replicating the process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gv7IkyHzdL", "reviewer": "Reviewer_geFz", "summary": "StreamV2V is an online approach (as opposed to batch / offline) for generating continuous video-to-video translation output without limits to video length and runs at 20 fps on a single A100 GPU.  The core idea behind StreamV2V is a feature bank from past frames that is retrieved and fused to ensure temporal coherence and consistency in frame generation.", "questions": "Please clarify the points mentioned in weaknesses above.  Particularly:\n\n- How are the magic constants determined and have you studied the impact?  If so, what did you find?\n\n- Please provide the details of the user study.\n\n- Please articulate the design choices, particularly SDEdit.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The core idea of the paper makes sense and is clearly presented.\n- The method achieves good results, both in terms of computational time (50 ms per frame) and quality (using objective metrics and subjective user studies).\n- Limitations and weaknesses against SOTA methods are presented.", "weaknesses": "- Some magic constants are reported without explanation (e.g., threshold of 0.90, max number of frames of 2, update feature bank every 4 frames).  How sensitive is your methods to these parameters?  What are the impact of higher or lower values? \n- The user study is presented without the necessary details.  How many participants? Age group? Gender? Prior experience with generative videos?  The authors should refer to papers reporting user studies to see what is the norm of doing so (e.g., see papers from CHI)\n- The choice of SDEdit (which is 3 years old) should be explained -- why not newer methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mn79tipG9P", "reviewer": "Reviewer_KRkM", "summary": "This paper presents a real-time zero-shot method for streaming video-to-video translation based on diffusion models. The key innovation is the adoption and maintenance of a dynamic feature bank which significantly increases the cross-frame consistency in the resultant videos. Although Steam Diffusion is existing work and there are a huge amount of memory updating algorithms in the field of video understanding and visual tracking, this reviewer appreciates the soundness and practical value of the proposed approach.", "questions": "Refer to the first question in weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is well-motivated and addresses an important problem. The method proposed is technically sound and practically feasible. The method achieves real-time V2V translation speed with a translation quality on par with the SOTA methods. The experiments, especially the ablation studies, are convincing enough to show the system-level contribution of the work.", "weaknesses": "There are still some artifacts in the result videos. This reviewer wonders whether a training-free method could eventually achieve a satisfactory performance. This will need some discussions. \nComponent wise, the innovation of this work is not very high. But as stated above, this reviewer appreciates the system-level contributions of the work. \nIn quantitative metrics comparison, the CLIP score is not quite differentiating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6uo5bCOmY", "reviewer": "Reviewer_7fxd", "summary": "The paper describes a video-to-video translation approach  that generates video frames in a streaming fashion. The method is based on StreamDiffusion for batch processing of frames and uses SDedit for generating each frame. The method needs no training or fine-tuning can be used together with any image diffusion model. Although the results are superior to some recent video translation models, they are inferior to the results of TokenFlow and FlowVid. However the method excels in its inference time.", "questions": "The authors claim that the method can be used as an add-on with any image generation model without any training or fine-tuning. This claim needs further elaboration. How is it that the method can be integrated to any translation model without any fine-tuning? How do the authors decide which features to store in the feature bank and which features to use in feature fusion? Could some fine-tuning improve the performance of the overall model? If not, why? If so, why haven't the authors done that so that overall performance could be improved?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has the following novel contributions: 1) videos can be translated in a streaming fashion without the need to process long sequences of video frames, 2) The past information of previous frames is stored in a feature bank and used during the generation of current frame without the need to fine-tune the existing image generation model, 3) Extended self-attention and feature fusion techniques are introduced to provide temporal consistency between consecutive frames. The most important contributions are the streaming video generation capability and the ability to use the method as an add-on with any image generation model without any training or fine-tuning.", "weaknesses": "The proposed method is fast and efficient, but it lags behind state-of-the-art  video translation models in CLIP score and subjective evaluations. The level of novelty is marginal: the paper does not propose a new video translation architecture, but merely develops an efficient feature storage, update and fusion strategy for temporal consistency. The overall approach is based on the existing work of StreamDiffusion and SDEdit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8usJwQMfIb", "reviewer": "Reviewer_DwFq", "summary": "The paper presents StreamV2V, a diffusion model for real-time, streaming video-to-video (V2V) translation using user prompts. It achieves efficiency by maintaining a continually updated feature bank of past frames, which it references to enhance incoming frames without needing fine-tuning. StreamV2V seamlessly integrates with image diffusion models and operates at 20 FPS on an A100 GPU, making it significantly faster than competing models. User studies and quantitative metrics validate StreamV2V's strong temporal coherence and adaptability.", "questions": "1. how does the proposed framework still outperform other video-to-video translation techniques for variations of diffusion models? Do different models with the proposed framework still outperform other video-to-video translation techniques?\n\n2. what does the running-time comparison look like on CPU Inferences rather than GPU inferences?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. this paper addresses the challenge of real-time video-to-video translation via the streamV2V framework with backward-looking and training-free techniques. Since streamV2V is training-free and fine-tuning-free compared to other video-to-video techniques, the proposed approach is feasible to be deployed to practical applications.\n\n2. the streamV2V framework enhances coherence between adjacent translated video frames through the extended self-attention layer and a proposed feature fusion approach for intermediate features of the current and the past frames. Due to the extended self-attention layer and the feature fusion approach, streamV2V effectively reuses the intermediate features from the past video frames to generate coherent translated video frameworks between adjacent frames.\n\n3. the streamV2V stores projected keys, projected values, and outputs of intermediate features of the past video frames through a feature bank with a dynamic merging mechanism. With the dynamic feature bank, the streamV2V framework trivially reuses historical features to align the outputs of intermediate blocks of the current and the past frames.", "weaknesses": "1. the proposed streamV2V framework fails to perform video-to-video translation for high-quality video generation in some scenarios while a prompt is provided.\n\n2. the proposed streamV2V framework fails to perform video-to-video translation when input video includes rapid motion movements.\n\n3. to test the generalization of the proposed framework, the manuscript misses experimentation comparisons for variations of difusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ocJWGJQs4R", "reviewer": "Reviewer_wGbG", "summary": "This paper introduces StreamV2V, a real-time video-to-video translation model that uses a feature bank and backward-looking mechanism to ensure temporal consistency. StreamV2V achieves significantly faster speeds than existing methods while maintaining consistent outputs.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel approach to streaming video-to-video translation, introducing a feature bank mechanism to address temporal consistency, enabling each frame in the video stream to reference features from past frames in an innovative manner.\n\n2. This method achieves efficient real-time processing without model fine-tuning, outperforming many comparable methods in experimental results.\n\n3. StreamV2V requires no training or fine-tuning and seamlessly integrates with existing image diffusion models, demonstrating the potential for a wide range of video editing tasks.\n\n4. The paper includes extensive quantitative experiments and user studies, validating its performance.\n\n5. This paper is well written and easy to follow. The supplementary materials also include comprehensive experimental details, content, and discussions, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. Does DyMe Bank still have advantages in video quality compared to queue-based banks when dealing with longer videos, more complex video content, or video motion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8A0PuvldtK", "reviewer": "Reviewer_bNTf", "summary": "This paper presents a study on real-time video-to-video (V2V) translation for streaming input using diffusion models. The authors identify limitations in batch-processing V2V methods, which are constrained by GPU memory and unsuitable for continuous, long-duration videos. In response, they propose StreamV2V, a diffusion model framework that integrates a feature bank, allowing it to maintain frame-to-frame consistency by leveraging stored features from past frames. This architecture enables StreamV2V to operate at 20 FPS on a single A100 GPU, significantly faster than existing V2V methods, and to support streaming inputs of indefinite length. Experimental results confirm that StreamV2V achieves competitive consistency and temporal stability, with favorable feedback from user studies compared to other V2V models. This paper is well-organized, and the methods and results are clearly presented.", "questions": "I would like to understand the rationale behind first concatenating, then randomly selecting features for matching and updating during the feature bank update process. Were other methods tested in the experiments, such as retaining a specific proportion of past and current features for matching and updating? Would the performance change if most of the randomly selected features came from either past frames or the current frame?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.This paper addresses a critical need in real-time video-to-video (V2V) translation, especially for streaming applications, which is highly relevant to the field. \n2.The paper is well-organized, with clear explanations of both the limitations in existing methods and the motivations behind the proposed backward-looking feature bank approach. The authors’ solution is explained in a logical and accessible manner, supporting reader understanding.", "weaknesses": "1.The model sometimes struggles to maintain temporal consistency when handling videos with significant camera or object movement, which may impact the overall quality.\n2.It may have limited capacity for major visual transformations, such as altering complex object characteristics, which may restrict its applicability in highly dynamic editing tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)