TokenFlow: Consistent Diffusion Features for Consistent Video Editing
Authors: Michal Geyer
Omer Bar-Tal
Shai Bagon
Tali Dekel
Venue: ICLR 2024 poster (2024)
Keywords: Computer Vision Applications
Video Editing and Synthesis
Text-to-Image Diffusion Models
Text-driven Video Editing
Primary area: generative models

Abstract: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

Under review as a conference paper at ICLR 2024
TOKENFLOW: CONSISTENT DIFFUSION FEATURES FOR CONSISTENT VIDEO EDITING
Anonymous authors
Paper under double-blind review
"A robot spinning shiny silver ball"
"A cheetak in the Sahara desert"
Bellt
"A Van Gogk portrait"
"A wolf in Machu Pichu"
Figure I:
TokenFlow enables consistent, high-quality semantic edits of real-world videos. Given an input video (top row), our method edits it according to a target text prompt (middle and bottom rows), while preserving the semantic layout and motion in the original scene.
ABSTRACT
The generative AI revolution has recently expanded to videos. Nevertheless, cur- rent state-of-the-ee video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power text-to-image diffusion model for the task of text-driven video editing. Specifically, given source video and method generates high-quality video that adheres the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspon- dences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to- image editing method. We demonstrate oftte-ofttheart editing results variety of weal-world videos.
target text-prompt,
INTRODUCTION
The evolution of text-to-image models has recently facilitated advances in image editing and con- tent creation, allowing users to control various proprieties of both generated and real images. Nev- ertheless, expanding this exciting progress to video is still lagging behind. A surge of large-scale text-to-video generative models has emerged, demonstrating impressive results in generating clips solely from textual descriptions. However, despite the progress made this area, existing video models are still in their infancy, being limited in resolution, video length, or the complexity of video dynamics they can represent.
this paper, harness the power of state-of-the-art pre-trained to--iag--ige model for the task of text-driven editing of natural vidcos. Specifically, our goal is to generate high-quality videos that adhere to the target edit expressed by an input text prompt, while preserving the spatial layout and motion of the original video. The main challenge in leveraging image diffusion model for video editing to ensure that the edited content consistent across all video frames ideally, each physical point in the 3D) world undergoes coherent modifications across time. Existing and concurrent video editing methods that based on image diffusion models have demonstrated that global appearance coherency across the edited frames can be achieved by extend- ing the self attention module to include multiple frames (Wu et al. 2022 Khachatryan et al.] [023b]] Ceylan et al. 2023 Qi et al.) 2023). Nevertheless, this approach is insufficient for achieving the desired level of temporal consistency, motion in the video is only implicitly preserved through the

---

Under review as a conference paper at ICLR 2024
attention module. Consequently, professionals or semi crofessoffssionll users often resort to elaborate video editing pipelines that entail additional manual work. In this work, we propose a framework to tackle this challenge by explicitly enforcing the original inter- frame correspondences on the edit. Intuitively, natural videos contain redundant information across frames, e.g., depict similar appear- ance and shared visual elements. Our key observation is that the internal representation of the video in the diffusion model exhibits similar properties. That is, the level of redundancy and temporal consistency of the frames in the RGB space and in the diffusion feature space are tightly correlated. Based on this observation, the pillar of our approach is to achieve consistent edit by ensuring that the features of the edited video are consistent across frames. Specifically, we enforce that the edited features convey the same inter-frame correspondences and redundancy as the original video features. To do so, we leverage the original inter-frame feature correspondences, which are readily available by the model. This leads to an effective method that directly propagates the edited diffusion features based on the original video dynamics. This approach allows us to harness the generative prior of state-of-thh-arr image diffusion model without additional training or fine-tuning, and can work in conjunction with an off-the-shelf diffusion-based image editing method (e.g.. |Meng al. (2022); HHertz et al. (2022); ZZhng & Agrawala (2023): Tumanyyn et al. (2023)).
To summarize, we make the following key contributions:
A technique, dubbed TokenFlow, that enforces semantic correspondences of diffusion fea-
tures across frames, allowing to significantly increase temporal consistency in videos gen- erated by text-to-image diffusion model.
Novel empirical analysis studying the propricties of diffusion features across video.
State-of-the-ar editing results on diverse videos, depicting complex motions.
2 RELATED WORK
Text-driven image & video synthesis Seminal works designed GAN architectures to synthesize images conditioned on text embeddings (Reed al., [016]] Zhang et al. 2016). With the ever- growing scale of vision-language datasets and pretraining strategies (Radford et al,, 2021: Scchh- mann al. 2022), there has been remarkable progress in text-driven image generation capabilities. Users can sytnesize high-quality visual content using simple text prompts. Much of this progress is also attributed to diffusion models (Sohl-Dicksten) et al, 2013: CCoitoou et al. 2022 Dhariwal & Nichol, 2021 Ho et al. 2020; Nichol & Dhariwal (021) which have been established as state- of-the-art text-to- image generators (Nichol al. 2021, Saharia et al. 2022 Ramesh et al.) 2022 Roobach et al. 2022] Sheynin et 20,2 Bar- et al. 2023). Such models have been extended for text-to-video generation, by extending 2D architectures to the temporal dimension (c.g.. using temporal attention Ho et al. (202bb) and performing large-scale training on video datasets (Ho) ctt al.) 2022a| Blattmann et al. 2023; Singer et al., (022). Recently, Gen-l (Esser et al., 2023) tai- lored a diffusion model architecture for the task of vIdeo editing. by conditioning the network on structure/appearance representations. Nevertheless, due to their extensive computation and memory requirements, existing video diffusion models are still in infancy and are largely restricted to short clips, or exhibit lower visual quality compared to image models. On the other side of the spectrum, a promising recent trend of works leverage pre-trained image diffusion model for video synthesis tasks, without additional training (Fridman et al. 2023 Wu et al. 2022: Lee et al. 2203a Qi et al. 2023). Our work falls into this category, employing pretrained ---agee diftusion model for the task of video editing, without any training or finetuning.
Consistent video stylization A common approach for video stylization involves applying image editing techniques (e.g., style transfer) on frame-by-frame basis, followed by prot-proessiing stage to address temporal inconsistencies in the edited video (Lai et al. (2018)); [ei et al. (2020) 2023). Although these methods effectively reduce high-frequency temporal flickering, they are not designed to handle frames that exhibit substantial variations in content, which often occur when applying text-based image editing techniques (Qi al. (023). Kasten et al. (202T) propose to de- compose a video into a set of 2D atlases, each provides a unified representation of the background or of foreground object throughout the video. Edits applied to the 2D atlases are automatically mapped back to the video, thus achieving temporal consistency with minimal effort. [Bar-Tal et al. (202)); Lee et al. (2023b) leverage this representation to perform text-driven editing. However, the atlas representation is limited to videos with simple motion and requires long training, limiting the applicability of this technique and of the methods built upon it. Our work is also related to classi- cal works that demonstrated that small patches in a natural video extensively repeat across frames (Shahar et al. 2011]] Cheung et al. 2005). and thus consistent editing can by simplified by editing subset of keytrames and propagating the edit across the video by establishing patch correspon- dences using handcrafted features and optical flow (Ruder et al. 2016: Jamriška al. 2019) or by
2

---

Under review as a conference paper at ICLR 2024
Original
Per-frame editing
Ours
Sample Frames
a-faliee Features (PCA)
Figure 3: Diffusion features across time, Left: Given an input video (top row), we apply DDIM inversion on each frame and extract features from the highest resolution decoder layer cg. We apply PCA on the features (i.e., output tokens from the self-attention module) extracted from frames and visualize the first three components (second row). We further visualize x-( slice (marked red on the original frame) for both RGB and features (bottom row). The feature representation is consistent across time corresponding regions are encoded with similar features across video. Middle: Frames and feature visualization for an edited video obtained by applying an image editing method (Tumanyan et al. (2023)) on each frame: inconsistent patterns RGB are also evident the feature space (e.g.. on the dog's body). Right: Our method enforces the edited video to convey the same level feature consistency as the original video, which translates into a coherent and high-quality edit RGB space.
training a patch-based GAN Texler et al. 2020). Nevertheless, such propagation methods strug- gle to handle videos with illumination changes, or with complex dynamics. Importantly, they rely on user provided consistent edit of the keyframes, which remains labor-intensive task yet to be automated. Yang et al. (2023) combines keyframe editing with a propagation method by Jamriška jet al.] (2019). They edit keyframes using a text-to-image diffusion model while enforcing optical flow constraints on the edited keyframes. However, since optical flow estimation between distant frames is not reliable, their method fails consistently edit keyframes that are far apart (as seen in our Supplementary Material - SM), and as result, fails to consistently edit most videos. Our work shares a similar motivation as this ap- proach that benefits from the temporal redundan- cies natural videos. We show that such redun- dancies are also present the feature space of a text-to-image diffusion model, and leverage this property achieve consistency.
Source
(b) Warped Source. Target
(c) Nearest-Nee Nearest-Nighhbor Hield
Recomstructed Target
Target
Controlled generation via diffusion features manipulation Recently, a surge of works demonstrated how text-to-image diffusion mod- els can be readily adapted to various editing and generation tasks, by performing simple opera-
Target II
tions the intermediate feature representation of the diffusion network (Chefer et al.) 2023
Hong et al.) 2022] Ma et al.) 2023] Tumanyan et al. 2023 Hertz et al. 2022 Patashnik et al. 2023
Figure 2: Fine-grained feature correspondences. Features (i.c., output tokens from the self-attention modules) extracted from of a source frame are used to reconstruct nearby frames. This is done by: (a) swapping each feature in the target by its nearest fea- ture in the source, in all layers and all generation time steps, and (b) simple warping in RGB space, using a nearest neighbour field (c), computed between the source and target features extracted from the highest resolution decoder layer. The target faithfully re- constructed, demonstrating the high level of spatial gramularity and shared content between the features.
Cao al. 2023). Luo al. (2023); Zhang et al. (2023) demonstrated semantic ap- pearance swapping using diffusion feature corre- spondences. Heerz al. (2022) observed that by manipulating the cross-attention layers, is pos- sible to control the relation between the spatial layout of the image to each word in the text. Plug- and-Play Diffusion (PnP. Tumanyan al. (2023) analyzed the spatial features and the self-atiention maps and found that they capture semantic information at high spatial granularity. Tune-A- Video (Wu et al. 2022) observed that by extending the self-attention module to operate more than sin-
3

---

Under review as a conference paper at ICLR 2024
Input Video
Compute NN rield
DDIM inversion
Extract tokens
Noisy Video
Joint editing
(II) TokenFlow propagation
Denoised
Sampled keyframes
J.
Z0g
T
F
Extended Attention
"cotountul painking""
'oolourful fuul
pallniing""
Figure 4: TokenFlow pipeline. Top: Given an input video I, we DDIM invert each frame, extract its tokens, ie.. output features from the self-attentio modules, from cach timestep and layer, and compute inter-frame features correspondences using a nearest-
(NN) search. Bottom: The edited video is generated as follows: at each denoising step i. (I) we sample keyframes from the noisy video J. and jointly edit them using an iitended-attentiin
block; the set of resulting edited tokens is Thase- (II) We propagate the edited tokens across the video according to the pre-computed correspondences of the original vidco features. To denoise J.. we feed each frame to the network, and replace the generated tokens with the tokens obtained from the propagation step (II).
gle frame, it is possible to generate frames that share common global appearance, Qi et al. (2023); Ceylan al. (2023); Khachatryan et al. (2023a); [Shin et al. (2023) ; Liu et al. 2023) leverage this property to achieve globally-coherent video edits. Nevertheless, as demonstrated
Sec. 5] inflat-
ing the self-attention module is insufficient for achieving fine-grained temporal consistency. Prior and concurrent works either compromise visual quality, exhibit limited temporal consistency. In this work, we also perform video editing via simple operations the feature space of pre- pre-trained text-to-image model, we explicitly encourage the features the model to be temporally consistent through TokenFlow.
PRELIMINARIES
Diffusion Models Diffusion probabalistic models (DPM) (Sohl-Dickstein et 2015 Croitoru cet 2022; Dhariwal Nichol 2021; Ho et al. 2020: Nichol & Dhariwal 202 )) are class of generative models that aim to approximate data distribution through a progressive denosing process. Starting from Gaussian i.id noisy image Fy NN, the diffusion model eg. gradually denoises it, until reaching clean image 2n drawn from the target distribution q. DPM can learn conditional distribution by incorporating additional guiding signals, such as text conditioning.
Song et al. (2020) derived DDIM, deterministic sampling algorithm given an initial noise Zr. By applying this algorithm in the reverse order (a.k.a. DDIM inversion) starting from the clean Zo. allows to obtain the intermediate noisy images {z )T used to generate it.
Stable Diffusion Stable Diffusion (SD) (Rombach al. 2022) is prominent text-to-image dif- fusion model that operates in a latent image space. A pretrained encoder maps RGB images to this space, and a decoder decodes latents back to high-resolution images. In more detail, SD is based U-Net architecture (Ronneberger al. 2015), which comprises of residual, self-attention, and cross-attention blocks. 'he residual block convolves the activations from a previous layer, while cross-attention manipulates features according to the text prompt. In the self-attention block, fea- tures projected into queries Q. keys K, and values V. The Atrention operation [Vaswani] al. 2017) computes the affinities between the d-dimensional projections Q, K to yield the output of the layer:
QKT
A V where A Artention(Q:Kt and Attention(Q:Ae KQ Softmax
(1)
Vd
4

---

Under review as a conference paper at ICLR 2024
Imput
Input video
"A Van Gogh portrait"
"Ice sculpture of car"
marble sculpture"
"Sand sculp of car on beack"
Iideo
Input
"A robotic wolf*
"Maui from Moana Movie"
"A colourful polygonal illustration"
"A Pixar animation
Figure 5: Results. Sample results our method. refer the reader to our webpage and SM for more examples and full-video results.
4 METHOD
I"] and a text prompt describing the target edit, our goal is to generate edited video J = [J] ..... J"] that adheres to the text P, while preserving the original motion and semantic layout of To achieve this, our framework leverages a pretrained and fixed text-to-image diffusion model es.
Given input video [T'
Naïvely leveraging eg for video editing, by applying image editing method on each frame inde- pendently (e.g., Hertz al. (2022;; Tumanyan et (2023) Meng al. (2022); Zhang & Agrawala (2023), results in content inconsistencies across frames (e.g., Fig. 3 middle column). Our key finding is that these inconsistencies can be alleviated by enforcing consistency among the internal diffusion features across frames, during the editing process.
Natural videos typically depict coherent and shared content across time. We observe that the internal representation of natural videos in Eg has similar properties. This is illustrated in Fig. 3 where we visualize the features extracted from given video (first column). As seen, the features depict shared and consistent representation across frames, i.e., corresponding regions exhibit similar repre- sentation. We further observe that the original video features provide fine-grained correspondences between frames, using a simple nearest neighbour search (Fig 2). Moreover, we show that these corresponding features are interchangeable for the diffission model we can faithfully synthesize one frame by swapping its features by their corresponding ones in a nearby frame (Fig ( Nevertheless, when an edit is applied to each frame individually. the consistency of the features breaks (Fig. 3 middle column). This implies that the level of consistency of in RGB space is corre- lated with the consistency the internal features of the frames. Hence, our key idea is to manipulate the features of the edited video to preserve the level consistency and inter-frame correspondences of the original video features.
As illustrated in Fig. our framework, dubbed TokenFlow, alternates at each generation timestep between two main components: (i) sampling a set keyframes and jointly editing them according to P; this stage results in shared global appearance across the keyframes, and (ii) propagating the fea- tures from the keyframes all of the frames based the correspondences provided by the original

---

Under review as a conference paper at ICLR 2024
Impul
"A rejnbow-fextured dog"
metal sculpture"
*An origami stork"
TAV
Gmn11
Ted2
Video
Fate-
Revendar
Figure 6: Comparison. We compare our method against Tune-A Video (TAV, Wu et al. 2022)), PnP- Diffusion (Tumanyan et 2023) applied per frame, Gen-1 (Esser et al. (023), Text2 Video-Zero (Khachatryan) ct al. 2023a) and Fate-Zero (Qi et al. 2023). We refer the reader to our supplementary material for full-vidco
A-Vieo
comparisons.
video features; this stage explicitly preserves the consistency and fine-grained shared representation of the original video features. Both stages are done in combination with image editing technique 6g (e.g, TTumanyan et al. (023)). Intuitively, the benefit of alternating between keyframe editing and propagation is twofold: first, sampling random keyframes at each generation step increases the robustness to particular selection. Second, since each generation step results in more consistent features, the sampled keyframes in the next step will be edited more consistently.
Pre-processing: extracting diffusion features. Given input video I, apply DDIM in- version (see Sec. 3) on each frame I', which yields a sequence of latents (.i,...,_7]] For each generation timestep f, we feed the latent æ; of each frame E [m]] to the model and extract the tokens ((x{) from the self-attention module every layer in the network en (fig. top). We will later use these tokens to establish fnter-frame correspondences between diffusion features.
4.1 KEYFRAME SAMPLING AND JOINT EDITING
Our observations imply that given the features of single edited frame, frames by propagating its features their corresponding locations. Most videos, however, can not be represented by single keyframe. To account for that, consider multiple keyframes, from which obtain set of features (tokens), Tases that will later be propagated to the entire video. Specifically, at each generation step, we randomly sample a set of keyframes {J*) Nee in fixed frame intervals (see SM for details). We joinly edit the keyframes by extending the self- attention block to simultaneously process them (Wu al. 2022). thus encouraging them to share global appearance. In more detail, the input to the modified block are the self-attention features from all feyframes {Q' Jea [K') Jien: (Vi) hhe where Q' K', V are the queries, keys, and values of frame E K, K ......).. The keys of all frames are concatenated, and the extended-attentee attentioo is:
generate the next
Q' K .... K'
ExtAttn (
,... K")) Softmax
Vd

---

Under review as a conference paper at ICLR 2024
The output of the block for frame is given by:
((J") A [V
via] where Á ExtAttn QQ: [
k)))
Intuitively, each keyframe queries all other keyframes, and aggregates information from them. This results in roughly unified appearance in the edited frames (Wu et al. 2022, Khachatryan et al. 2023b Ceylan et al. 2023 Qi et al. 2023). We define T Tasse = {o(J") hie for each layer in the
network (Ing. bottom middle).
4.2 EDIT PROPAGATION VIA TOKENFLOW
Given T buse. we propagate across the video based the token correspondencee extracted from the original video. At each generation step t, we compute the nearest neighbor (NN) of each original frame's tokens, (bx{), and its two adjacent keyframes' tokens, ((x;" ), <bx{²) where i+ is the index of the closest future keyframe, and i- the index of the closest past keyframe. Denote the resulting NN fields 7t -
7 ') arg min D (6(("))))) d( lal)
Where p, q spatial locations in the token feature map, and D is cosine distance. For simplicity, we omit the generation timestep t; our method applied in all time-steps and self-attention layers. Once we obtain y we use it to propagate the edited frames' tokens Tase to the rest of the video, by linearly combining the tokens in T Thsse corresponding to each spatial location p and frame i:
F i, = uey ( ((J*)))" [x] + (1 (1-w;) o ((J')) >1n [p]]
(5)
Where ((J=) E T base and wy E (0, 1) scalar proportional to the distance between frame and its adjacent keyframes (see SM), ensuring a smooth transition. Note that F also modifies the tokens of the sampled keyframes. That is, modify the self-attention blocks to output linear combination of the tokens in T Tasse for all frames, including the keyframes, according to the original video token correspondences.
Overall algorithm We summarize video editing algorithm in Alg. [ We first perform DDIM inversion on the in- put video I and extract the sequence of noisy latents {at)T= for all frames i [n] (fig 4 top). We then denoise the video, alternating between keyframes editing and TokenFlow propagation: At each generation step f, we randomize k keyframe indices, and denoise them using an image editing technique (e.g.. Tumanyan et al. (2023); [Mng et al.) 2022); [Zang & Agrawala (223)) com- bined with extended attentooon (Eq 3 Fig.
Algorithm TokenFlow editing
Input:
D Input Video b Target lext prompt
P
Diffusion-based image editing technique
| DDIM-Inv([") Vi [n], E [1]] JJ.....JJ J < x
(x (0(2.)
For ( = T,..., do
K. = (i (in....,
ix] 6 sample keyframe indices
F, g/ Vi [n] compute NN field
(J W/-idee t- 6 (e]
The + oi [J/ ]Sek) extract keyframes' tokens
J + ég [J:; TokenFlow(FF, (TTbe))
4 (I)). We then denoise the entire video
T. by combining the image-editing tech-
Output: J [Jb J
nique with TokenFlow (Eq. 5 Fig. (II)
at every self-attention block in every layer of the network. Note that each layer includes residual
connection between the input and output of the self-attention block, thus performing TokenFlow at each layer is necessary.
5 RESULTS
We evaluate our method on DAVIS videos (Pont-Tuset et al. 2017) and on Internet videos depicting animals, food, humans, and various objects in motion. The spatral resolution of the videos is 384 x 672 512 x 512 pixels, and they consist of 40 to 200 frames. We various text prompts on each video to obtain diverse editing results. Our evaluation dataset comprises of text-video pairs. We utilize PnP-Diffusion (Tumanyan et al. 2023) as the frame editing method, and we use the same hyper-parameters for all our results. PnP-Diffusion may fail to accurately preserve the structure of each frame due to inaccurate DDIM inversion (see Fig. middle column, right frame: the dog's head is distorted), Our method improves robustness to this, as multiple frames contribute to the generation of each frame in the video. Our framework can be combined with any diffusion-based image editing technique that accurately preserves the structure of the images; results with different

---

Under review as a conference paper at ICLR 2024
image editing techniques (e.g. Meng et al. (2022 Zhang & Agrawala (2023) ) are available in the SM. Fig. andd I show sample frames from the edited videos. Our edits are temporally consistent and adhere to the edit prompt. The man's 's head is changed to Van-Gogh or marble (top left); importantly, the man's identity and the scene's 's background are consistent throughout the video. The patterns of the polygonal wolf (bottom left) are the same across time: the body is consistently orange while the chest is blue. We refer the reader to the SM for implementation details and video results.
Baselines. We compare our method to state- ael--the-rrt art, and concurrent works: (i) Fate-Zero (Qi (Khachatryan et al. (023b), that utilize a text-to-image model
it al., (2023) and (ii)
for video editing using self-attention inflation. (1i1) Re-render Video (Yang al., 2023) that edits keyframes by adding optical flow optimization to self-attention inflation of an image model, and then propagates the edit from the keyframes to the rest of the video using an off-the-shelf propagation method. (iv) Tune-a- Video |Wu ct al.) 2022) that fine-tunes the text-to-image model on the given test video. (v) Gen-1 (Esser al. 2023). ). a video diffusion model that was trained on large-scale image and video dataset. (v1) Per-trame diffusion- based image editing baseline,
(Tumanyan et al. 2023). We additionally consider the two following baselines: (i) Text2LIVE (Bar- lal et al. 2022 which utilize layered video representation (NLA) (Kasten et al. 202T) and perform test-time training using CLIP losses. Note that NLA requires foreground/backgroun
separation masks and takes ~ 10 hours to train. (ii) Applying PnP-Diffusion on a single keyframe and propagating the edit
to the entire video using Jamriska et al. (2019).
5.1 QUALITATIVE EVALUATION
Fig. ploovidess a qualitative comparison of our method prominent baselines; please refer SM for the tull videos. Our method (bottom row) outputs videos that better adhere to the edit prompt while maintaining temporal consistency of the resulting edited video, while other methods struggle to meet both these goals. Tune-A- Video (second row) inflates the 2D image model into video model, and fine-tunes it to overfit the motion of the video; thus, it is suitable for short clips. For long videos it struggles to capture the motion resulting with meaningless edits, e.g., the shiny metal sculpture. Applying PnP for each frame independently (third row) results in exquisite edits adhering to the edit prompt but, as expected, lack any temporal consistency. The results of Gen-l (fourth row) also suffer from some temporal inconsistencies (the beak of the origami stork changes color). Moreover, their frame quality is significantly worse than that of text-to-image diffusion model. The edits of Video-odoo and Fate-Zero (fifth and sixth row) suffer from severe jittering these methods rely heavily on the extended attention mechanism to implicitly encourage consistency. The results of Rerender-a-Ve Video exhibit notable long-range inconsistencies and artifacts arising primarily from their reliance on optical flow estimation for distant frames (c.g. keyframes), which is known to be sub-optimal (See our video results the SM: when the wolf turns its head, the nose color changes). We provide qualitative comparison to Tex12LIVE and to a RGB propagation baseline in the SM.
5.2 QUANTITATIVE EVALUATION
We evaluate our method in terms of: (i) edit fidelity measured by comput- ing the average similarity between the CLIP embedding (Radford et al.] 2021) of each edited trame and the target text prompt; (ii) temporal con- sistency. Following Ceylan et al. (2023; Lai et al.] (2018a), tempo- ral consistency is measured computing the optical flow of the original video using Teed & Deng (020). warping the edited trames according to it, and measuring the warping error, and (b) a user study; We adopt a Two-alternative Forced Choice (2AFC) protocol suggested in Kolkin et al. (2019): Park et al. (2000). where participants are shown
We evaluate our method in temporal consistency by computing warp-error and conducting a user study, and in fidelity to the target text prompt using CLIP similarity. See Sec. for more details.
Table 1:
Warp-err + User preference
CLIP
(x10 3)
of our method
score f
LDM recon. PiP--iiitusion
2.0
0.23
(a)
I1.3
94%
0.33
Text2Video-Ze
12.5
78%
0.33
Tune-a-Video
30.0
82%
0.31
Fate-Zero
6.9
71%
0.32
Genl
70%
0.32
Kerender-a-Vides
1.8
71%
0.32
Ours w joint attention
5.9
90%
0.33
Ours w/o rand keyframes Ours
3.7
0.33
3.0
0.33
the input video, ours and bascline result, and are asked to determine which video is more tem- porally consistent and better preserves the motion of the original video. The survey consists of 2000-3000 1000-300 judgments per baseline obtained using Amazon mechanical turk. We note that warping- error could not be measured for Genl since their product platform does not output the same number of input frames. Table [ compares our method baselines. Our method achieves the highest CLIIP
8

---

Under review as a conference paper at ICLR 2024
"A tractor
Figure 7: Limitations. Our method edits the video according to the feature correspondences of the original video, hence it cannot handle edits that requires structure deviations.
score, showing a good fit between the edited video and the input guidance prompt. Furthermore, our method has a low warping error, indicating temporally consistent results. We note that Re-render- a- a-Video optimizes for the warping error and uses optical flow to propagate the edit, and hence has the lowest warping error; However, this reliance on optical flow often creates artifacts and long- range inconsistencies which are not reflected in the warping error. Nonetheless, they are apparent in the user study, that shows users significantly favoured our method over all baselines in terms of temporal consistency. Additionally, we consider the reference baseline of passing the original video through the LDM auto-encoder without performing editing (LDM recon.). This baseline provides an upper bound on the temporal consistency achievable by LDM auto-encoder. As expected, the CLIP similarity of this baseline is poor as it does not involve any editing. However, this baseline does not achieve zero warp error either due to the imperfect reconstruction of the LDM outo-encoder, which hallucinates high-frequency information.
We further evaluate our correspondences and video representation by measuring the accuracy of video reconstruction using TokenFlow. Specifically, we reconstruct the video using the same pipeline of our editing method, only removing the keyframes editing part. Table reports the PSNR and LPIPS distance of this reconstruction, compared to vanilla DDIM reconstruction. As seen, TokenFlow reconstruction slightly improves DDIM inversion, demonstrating robust frame represen- tation. This improvement can be attributed to the keyframe randomization; increases robustness to challenging frames since each frame is reconstructed from multiple other frames during the gener- ation. Notably, our evaluation focuses on accurate correspondences within the feature space during generation, rather than RGB frame correspondences evaluation, which not essential to our method.
5.3
ABL ABLATION STUDY
First, we ablate the use of TokenFlow, Sec. 4.2 for en- forcing temporal consistency. In this experiment, we replace TokenFlow with extended attention (Eq. 3 and compute between each frames of the edited video and the keyframes (w joint attention). Second, we ablate the randomizing of the keyframe selection at each genera- tion step (w/o random keyframes). In this experiment, we use the same keyframe indices (evenly spaced time) across the generation. Table [ shows the quan- titative results of our ablations, the resulting videos be found in the SM. As seen, TokenFlow ensures higher Ours degree of temporal consistency, indicating that solely re- lying on the extension of self-attention to multiple frames is insufficient for achieving fine-grained temporal consistency. Additionally, fixing the keyframes creates an artificial partition of the video into short clips between the fixed keyframes, which reflects poorly on the consistency of the result.
We reconstruct the video using the TokenFlow pipeline, excluding keyframe editing. We evaluate the TokenFlow represen- tation with PSNR and LPIPS metries. Our reconstruction improves vanilla DDIM inver- sion, highlighting the robusteness of Token- Flow representation.
Table 2:
PSNR t
LPIPS
LDM recon.
31.13
0.03
25.32
-0.11
25.74
0.13
DISCUSSION
We presented a new framework for text-driven video editing using an image diffusion model. We study the internal representation of a video in the diffusion feature space, and demonstrate that consistent video editing can be achieved via consistent diffusion feature representation during the generation. Our method outperforms existing baselines, demonstrating a significant improvement in temporal consistency. As for limitations, our method is tailored to preserve the motion of the original video, and as such, it cannot handle edits that require structural changes (Fig 7 Moreover, our method is built upon a diffusion- based image editing technique to allow the structure preservation of the original frames. When the image-editing technique fails to preserve the structure, our method enforces correspondences that are meaningless in the edited frames, resulting in visual artifacts. Lastly, the LDM decoder introduces some high frequency flickering (Blattmann et al., 2023). A possible solution for this would be to combine our framework with an improved decoder (e.g.. Blatmann et al. (2023). Zhu et al. (2023)). ). We note that this minor level of flickering can be easily climinated with exiting post-proces
deflickering (see SM). Our work shed new light on the internal representation of natural videos in the space of diffusion models (c.g., temporal redundancies). and how they can be leveraged for enhancing video synthesis. We believe it can inspire future research in harnessing image models for video tasks, and for the design of text-to-vide models.

---

Under review as a conference paper at ICLR 2024
REFERENCES
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layored image and video editing. In European Conference Computer Vision, pp. 707-723. Springer, 2022.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.00 08/00///333 2023.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Kansten Kreis, Align your latents: High-resolution vidco synthesis with latent diffusion models. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Mingdeng Cao, Xintao Wang. Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactri: Tuming- free mutual self-attention control for consistent image synthesis and editing, 2023.
Duygu Ceylan, Chun-Har Paul Huang, and Niloy Jyoti Mitra. Pix2video: Video editing using image diffusion. ArXiv, abs/2303. 12688, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Danicl Cohen-Or. Attend-and-excite Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13820 2023.
Cheung, B.J. Frey, and N. Jojic. Vidco epitomes. 2005 JEEE Computer Sociery Conference on Computer Vision and Pattern Recognition (CVPR (05), 2005.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. arXiv preprint arXiv:2209.0777 047477 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021.
Patrick Esser, Johnathan Chiu. Parmida Atighehchian, Jonathan Granskog, Anastasis Germanidis. Structure and content- video synthesis with diffusion models. arXiv preprint arXiv:2302.88011, 2023.
Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene gener- ation. arXiv preprint arXiv:2302.011 133, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum. Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint Stti:2200.016221 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffiusion probabilistic models. Advances in Neural
Information Processing Systems, 2020.
Jonathan Ho. William Chan, Chitwan Saharia. Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02308.. 2022a.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.033:5 2022b.
Susung Hong. Gyuseong Lee, Wonscok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. orXiv preprint arXiv:2210.009: 2022.
Ondrcj Jamriška, Šárka Sochorová, Ondicj Texler, Michal Lukáë, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sykora. Stylizing video by example, ACM Transactions on Graphics, 2019.
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing.
ACM Transactions on Graphics (TOG), 2021.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Tex(2vieeo-zero: Text-to-image diffusion models are zero-shot video generators. ArXiv, 133399 2023a.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Tex(2video-zero Text-to-image diffusion models are zero-shot video generators. arXiv preprint 2303 2023b.
Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEFE/CVP Conference on Computer Vision and Pattern Recognition, pp. 10051-10060, 2019.
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In European Conference on Computer Vision, 2018a.
10

---

Under review as a conference paper at ICLR 2024
Wei-Sheng Lai, Jia-Bin Huang. Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming- Yang. Learning blind vidco temporal consistency. In Proceedings the European conference computer vision (ECCV), pp. 170- 170-185, 20.
Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang, Shape-aware text- driven layered video editing. arXiv preprint 2rXiv:2301.13773 2023a.
Yao-Chih Lee, Ji-Ze Genevieve lang Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang. Shape-aware text-driven layered video editing demo. arXiv preprint arXiv:2301.33177 2023h.
Chenyang Lei, Yazhou Xing. and Qifeng Chen. Blind video temporal consistency via deep video prior. In Advances in Neural Information Processing Systems, 2020.
Chenyang Lei, Xuanchi Ren, Zhaoxiang Zhang, and Qifeng Chen. Blind video deflickering by neural filtering with a flawed atlas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2023.
Shaoteng Liu, Yuecheng Zhang. Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross- attention control. ArXiv, abs/2303.0476. 2023.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. arXiv, 2023.
Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung. Directed diffusion: Direct control of object placement through attention guidance. arXiv preprint arXiv:2300. 13153, 2023.
Chenlin Meng. Yutong Hc, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text text- suided diffusion models. arXiv preprint arXiv:2112.10741 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Inter- national Conference Machine Learning, pp. 8162-8171. PMLR, 2021.
Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. In Advances in Neural Information Processing Systems, 2020.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models. arXiv preprint arXiv:2303. 1306, 2023.
Jordi Pont-Tuset, Federico Perazzi. Sergi Caelles, Pablo Arbeláez, Alex The 2017 davis challenge on video object segmentation. arXiv preprint 17Xiv:1704.00675,, 2017.
and Luc Van Gool.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen, Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.09533 2023.
Alec Radford, Jong Woek Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.66122 2022.
Scott Reed, Zeynep Akata, Xinehen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee, Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069, PMLR, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 2015.
Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox, Artistic style transfer for videos. In Pattern Recognition 38th German Conference (GCPR), 2016.
11

---

Under review as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11005.1187... 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large- scale dataset for training next generation image-text models. ArXiv, abs/2210.08402, 2022.
Oded Shahar, Alon Faktor, and Michal Trani. Space-time super-resolution from single video. In CVPR 2011, 2011.
Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. arXiv preprint arXiv:2204.02845 2022.
Chachun Shin, Heeseung Kim, Che Hyun Lee, Sang gil Lee, and Sung-Hoop Hoon Yoon. Edit-a-video: Single video editing with object-aware consistency. ArXiv, abs/2303.07945, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An. Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16. Springer, 2020.
Ondiej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Šárka Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora. Interactive video stylization using few-shot patch-based training. ACM Trans- actions on Graphics, 2020.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion fcatures for text-driven mmage-to-image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune- aivdeo One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565, 2022.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video: Zero-shot text-guided video-to- video translation, 2023.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5908-5916, 2016.
Junyi Zhang, Charles Hermann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspon- dence. arxiv preprint ar05v: 15347, 2023.
Lvmin Zhang and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Zixin Zhu, Xuelu Feng, Dongdong Chen. Jianmin Bao, Le Wang. Yinpeng Chen, Lu Yuan, and Gang Hua. Designing a better asymmetric vqgan for stablediffusion, 2023.
12

---

Under review as a conference paper at ICLR 2024
Table 3: We report average runtime in seconds, of running and competing methods video
of frames.
TAV
Texe2video-zero
Rerender-a-video
fatezero
PnP
ours (preprocess)
ours (sampling)
ours (total)
2684
198
285
349
208
50
187
237
We provide additional implementation details below. We refer the reader to the HTML. file attached to our Supplementary Material for video results.
A IMPLEMENTATION DETAILS
StableDiffusion. We Stable Diffusion as pre-trained text-to-image model; use the StableDiffusion---22- checkpoint provided via official HuggingFace webpage
DDIM inversion. In all of our experiments, we use DDIM deterministic sampling with 50 steps. follow [Tumanyan et al. (2023) and use DDIM inversion with classifier- free guidance scale of I and 1000 forward steps; and extract the self-attention input tokens from this process similarly to Qi et al. (2023).
For inverting the video,
Runtime. Since don't compute the attention module most video frames (i.e., only com- pute the self-attention output on the keyframes) our method is efficient in run-time, and the sampling of the video reduces the time of per-frume editing by 20%. The inversion process with 1000 steps is the main bottleneck of our method in terms of run-time, and in many cases significantly smaller amount steps is suffieicent (e.g. 50). Table 3 reports runtime comparisons using 50 steps in all methods. Notably, our sampling time is indeed Taster than that of per-frame editing (PnP).
slyper-araameers. In equation 3 set w. to be:
= (d4 + d_))
where d4 = || - + 11, i-
where a is sigmoid function, 8 and i are the future and past neighboring keyframes of respec- tively.
For sampling the edited video we set the classifier-fre guidance scale to 7.5. At each timestep, we sample random keyframes in frame intervals of 8. We note that using less keyframes (i.c., increas- ing the interval size) results in (i) runtime decreases (the joint keyframe editing step requires less memory, and is faster), and (ii) temporal consistency is improved (since the same tokens are shared across more frames). Nevertheless, too few keyframes will result in inaccurate correspondences which may result in artefacts.
Baselines. For running the baseline Tune-a-video (Wu et al. 2022) we used their official repos- itory. For Gen-1 (Esser et al. 2023) we used their platform on Runaway website. This platform outputs video that not in the same length and frame-rate as the input video; therefore, could not compute the warping error on their results. For text-to-video-zerr (Khachatryan al.] (023b) we used their official repository. with their depth conditioning configuration. For Pate-Zere (Qi et al. 2023) with used their official repository, and verified the run configurations with the authors.
13


Reviews:
[{"review_id": "yPYl5Y2a0Y", "reviewer": "Reviewer_FeQk", "summary": "The paper presents a method for text-based video editing, called TokenFlow. TokenFlow utilizes a pre-trained text-to-image diffusion model without the need for finetuning or video training data. Independently using text-based image editing techniques on frames will produce temporal artifacts. The paper proposed a method to improve the temporal consistency. More specifically, the method uses extended attention to edit several keyframes and then propagates the keyframe features to all the frames based on a Nearest Neighbour field. The Nearest Neighbour field is computed based on features of DDIM inversion.", "questions": "Are self-attention features the only features that are replaced by features of neighboring frames? Have you tried to replace some other features such as ResBlock features or attention masks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The video editing results are impressive, the temporal consistency is pretty good.\n- The analysis and visualization of UNet features on video tasks are helpful for future research on video generation.\n- The idea of TokenFlow is novel. Based on the ablation study and qualitative results in the supplemental material, TokenFlow is also very critical to good temporal consistency. \n- The paper reads well and is easy to follow.", "weaknesses": "Although it's not necessary, it will be helpful to compare TokenFlow with Pix2Video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHOUetCg1P", "reviewer": "Reviewer_vSnb", "summary": "This paper proposes TokenFlow for text-driven video editing, aiming to generate a temporally consistent video that adheres to the text prompt while preserving the spatial structure/motion of the source video. Specifically, TokenFlow leverages a pre-trained text-to-image diffusion model to extract features/tokens of each video frame, compute latent patch correspondence between neighboring frames, and temporally propagate the key-frame tokens to other frames during the diffusion process. Qualitative and quantitative results show that the TokenFlow performs similarly to prior methods in terms of edit fidelity (CLIP similarity) while achieving higher temporal consistency (warping error and user study).", "questions": "Q1: The paper mentions that TokenFlow is more computationally efficient. What is the overall runtime to generate a new video and how is it compared to the methods listed in Table 1?\n\nQ2: The ablation study on keyframe sampling only covers fixed and random sampling. It would be good to also ablate on sampling interval (tradeoff between computation overhead and temporal smoothness). I’m also curious if a dynamic keyframe sampling scheme would further improve the results, especially for occlusion cases.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: Sensible model design\nAlthough the ideas of 1) using text-to-image diffusion model for video generation and 2) using latent feature flow for temporal consistency are not new, the proposed framework combines these components sensibly. The simplicity of this method also makes it compatible with existing video editing methods and more efficient than most prior arts.\n\nS2: Temporally consistent results\nThe visual results show a significant improvement from prior methods in terms of temporal consistency of both texture and structural details.\n\nS3: Good writing\nThe paper is well-written and easy to follow. I find the illustrations and algorithm pseudo-code quite helpful to understand the framework.", "weaknesses": "W1: Novelty \nThe novelty of the proposed framework is slightly limited, considering that the key components (keyframe sampling, feature aggregation and propagation across frames) are introduced in prior works. Also, it is unclear which part of Section 4.1 is newly proposed in the paper and which is borrowed from other works. It would be great if the authors can elaborate on the main differences from prior methods and specify the novel components/modifications. \n\nW2: Limited structural deviation \nAs shown in Figure 7, TokenFlow outputs strictly follow the structural layout of the source video, which might limit its generative capability/application. I’m wondering if there is a way to relax the temporal consistency constraint around object boundaries, so that one can find the desired tradeoff between temporal consistency and structural editing (maybe by tuning some hyper-parameters).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QXYqDQTlPd", "reviewer": "Reviewer_6aWr", "summary": "The paper aims to address the problem of consistent video editing. The proposed method is based on text-to-mage diffusion models, and the task is to convert a source video with a target text prompt into a new video that associates with the target text while preserving the motion of the source video. The emphasis is on producing consistent frames as naively applying an image-based text-to-image diffusion model would generate individually good-quality frames, but when they are put together, it would jointly result in an inconsistent video. The key idea proposed in this paper to solve the problem is called TokenFlow, which enforces the edited internal representation of the diffusion process to preserve the inter-frame correspondences of the original video. The approach is simple and the results shown in the paper and the supplementary material look quite good.", "questions": "* What would happen if the target of editing is partially occluded for a few frames in the video?\n* The supplementary material shows some results of per-frame editing using ControlNet. Is it possible for ControlNet to be used not only for editing but also for providing optical flow guidance? If so, how would it differ from TokenFLow?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. \nThe proposed method is simple and lightweight. It is built on an existing diffusion-based image editing method and does not need to fine-tune the model. The \"TokenFLow Editing\" algorithm is easy to implement (code available in the supplementary material). It directly utilizes Stable Diffusion, DDIM inversion, and PnP-Diffusion, and the TokenFlow procedure just requires computing the nearest-neighbor fields for token feature maps. \nFurther, as mentioned in the summary in Sec. 1 of the paper, state-of-the-art editing results are one of the main contributions of this work. Indeed, the edited videos presented in the supplementary results exhibit better consistency than other methods' outputs.  \n\n2. \nA helpful finding from this work is that the internal features offer a shared and consistent representation across frames, and the corresponding features are interchangeable for the diffusion model. The spatial and semantic properties of diffusion features are also mentioned in the concurrent work \"Emergent Correspondence from Image Diffusion\" as DIFT, proposed by Tang et al.; however, they focus more on matching different images and show some results for edit propagation in image editing and for video label propagation on DAVIS and JHMDB instead of enforcing consistency in video editing. While it is not necessary to empirically compare TokenFlow with DIFT, it would be helpful to highlight the differences and the shared ideas.\nNevertheless, these findings of diffusion features provide a promising direction to revisit prior ideas like *Image Analogies* and *PatchMatch*.", "weaknesses": "1. \nThe results in the paper and the supplementary material mainly demonstrate the visual effect of video style transfer. For more general video editing tasks, one might expect to see some results of motion-based or composition-based video editing. Since the proposed method relies on the feature correspondences in the original video, it seems not trivial if one would like to modify the TokenFlow for motion-based editing. \n\n2. \nRegarding the quantitative evaluation:\n- The *edit fidelity* measured by CLIP score does not provide useful/discriminative information. It might also need to include a user study on the visual quality and fidelity.\n- The *temporal consistency* measured by optical flow and warping might over-penalize edits that change in shape and tend to favor edits that involve only color/texture changes.\n\n\n3. \nMinor typos:  \n-  \"to operate on more **then** a single frame\"\n-  **keyframess'**\n- The first words after Eqs. (4) \\& (5) are in upper case: **Where**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WkPpRLusH", "reviewer": "Reviewer_Mqtb", "summary": "In this paper, the authors propose a framework, TokenFlow, for video editing task.\nTokenFlow runs in a correspondence-propagation manner, i.e., first seeks for the correspondences across different frames, jointly edits the keyframes, and then propagates features to ensure the temporal consistency. \nCompared to prior arts, TokenFlow shows a better temporal consistency and competitive editing fidelity.", "questions": "- Features & RGB images: In Figure 3, authors show that TokenFlow improves the consistency in feature level. However, can the feature level consistency ensure the RGB output consistency? \n- Correspondence ablation: Why using NN for the correspondences instead of using optical flow? Can pixel-level correspondences like dense optical flow be used for this token-based framework? Can we apply downsampled optical flow maps to the feature maps? \n- Occlusion: Is the current framework be able to handle some extreme cases, like occlusion? For example, a dog running through some poles but sometimes the dog is occluded by pole. \n- Video length: What is the maximal length TokenFlow can handle? \n- What is the post-procssing deflickering that is used in the SM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Approach:\n- Global consistency. TokenFlow utilizes a joint editing and feature propagation via NN feature correspondences. Compared to attention-based methods, this way is more explicit and tends to keep a global consistency across different frames in a video. \n- Compatible with other image-based editors. TokenFlow seems to be able to work with other diffusion-based image editors. \n\nExperiments & validation:\n- Proposed method is intuitive. First doing joint editing and then propagating the features makes sense. \n- From qualitative results, TokenFlow improves the temporal consistency and preserves fair fidelity. \n- Instead of using other pixelwise correspondences (e.g., dense flow or pixelwise trajectory), the authors propose to use nearest neighbor (NN) to find the correspondence. This seems new to me. \n\nWriting & presentation:\n- The paper is well-written and easy to follow.", "weaknesses": "Experiments:\n- Compared to other baselines, like Text2LIVE and Gen1, TokenFlow still shows some \"flickering\" when there are high-frequency patterns. For example, in \"Comparisons to Baselines\" SM, in the first example \"running dog\", the ground has severe flickering compared to Gen 1. Also in the third example \"cutting bread\", there is more flickering in the bread and background compared to Gen 1. Same thing also happened in the comparison to Text2LIVE in \"Additional Qualitative Comparisons\". Why is TokenFlow not able to maintain the consistency for the high-frequency patterns?\n- Based on the previous point, I think the authors could consider analyzing the reason behind the flickering and include it in **Limitations**.\n- How do different image editors affect the results? Specifically,  the comparison w/ PnP + propagation. The authors mention that they have an additional comparison with PnP-Diffusion + propagation in the **last sentence, Section 5 Baselines**, but this part seems missing either in the main paper or in the SM. \n- It would be great if the authors could also include runtime comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)