FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing
Authors: Yuren Cong
Mengmeng Xu
christian simon
Shoufa Chen
Jiawei Ren
Yanping Xie
Juan-Manuel Perez-Rua
Bodo Rosenhahn
Tao Xiang
Sen He
Venue: ICLR 2024 poster (2024)
Keywords: diffusion model
video editing
text-to-video
Primary area: generative models
TL;DR: This paper presents a training-free framework for high consistent text-to-video editing by integrating optical flow into attention modules.

Abstract: Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts.
A major challenge in this task is to ensure that all frames in the edited video are visually consistent. 
Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention.
Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch and therefore cause inconsistency in the edited video. 
In this paper, for the first time, we introduce optical flow into the attention module in diffusion model's U-Net to address the inconsistency issue for text-to-video editing.
Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other in the attention module,  thus improving the visual consistency in the edited videos. 
Additionally, our method is training-free and can be seamlessly integrated into any diffusion based text-to-video editing methods and improve their visual consistency.
Experiment results on existing text-to-video editing benchmarks show that our proposed method achieves the new state-of-the-art performance. In particular, our method excels in maintaining the visual consistency in the edited videos.

Published as a conference paper at ICLR 2024
FLATTEN: OPTICAL FLOW-GUIDED ATTENTION FOR CONSISTENT TEXT-TO- VIDEO EDITING
Yuren Cong 1.2x Mengmmeng Xu², Christian Simon", Shoufa Chen', Chen", Jiawei Ren Yanping Xie², Juan-Manuel Perez-Rua', Bodo Rosenhahn', Tao Xiang", , Sen He?
LLibbiiz University Hannover, Meta AI, 'The University of Hong Kong, "Nanyang Technological University
Source video
"A cat, Van Gogh style."
"A woolen toy cat."
Edited videos
"A tiger."
Our method generates visually consistent videos that adhere to different types (style, texture, and category) of textual prompts while faithfully preserving the motion in the source video.
Figure 1:
ABSTRACT
Text-to-video editing aims to edit the visual appearance of source video con- ditional on textual prompts. major challenge in this task to ensure that all frames in the edited video are visually consistent. Most recent works apply ad- vanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into apatio-tempont attention. Although temporal context added through spatio-temporal attention, it may introduce some irrelevant infor- mation for each patch and therefore cause inconsistency in the edited video. In this paper, for the first time, we introduce optical flow into the attention module in the diffusion model's 's U-Net to address the inconsistency issue for text-to-video edit- ing. Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other the attention module, thus improving the visual consistency in the edited videos. Additionally, our method is training- free and can be seamlessly integrated into any diffusion-based text-to-video edit- ing methods and improve their visual consistency. Experiment results existing text-to- video editing benchmarks show that our proposed method achieves the new state-of-the-an performance. In particular, our method excels in maintain- ing the visual consistency in the edited videos. The project page is available at https: htt.:/latttttiiiiee wewettteee-dio-ddeeeeeee-- ingg github.io/
be
'Work done during internship Meta AI.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Short videos have become increasingly popular on social platforms in recent years. To attract more attention from subscribers, people like to edit their videos to be more intriguing before uploading them onto their personal social platforms. Text-to-video (T2V) editing, which aims to change the visual appearance of video according to a given textual prompt, can provide a new experience for video editing and has the potential to significantly increase flexibility, productivity, and efficiency. It has, therefore, attracted a great deal of attention recently (Wu et al., 2022; Khachatryan et al., 2023; Qi et al., 2023; Zhang et al., 2023; Ceylan al., 2023; Qiu et al., 2023; Ma et al., 2023).
A critical challenge in text-to-video editing compared to text-to-image (T2I) editing is visual consis- tency, i.c., the content in the edited video should have smooth and unchanging visual appearance throughout the video. Furthermore, the edited video should preserve the motion from the source video with minimal structural distortion. These challenges are expected to be alleviated by using fundamental models for text-to-video generation (Ho et al., 2022a; Singer et al., 2022; Blattmann al., 2023; Yu et al., 2023). Unfortunately, these models usually take substantial computational resources and gigantic amounts of video data, and many models are unavailable to the public.
Most recent works (Wu et al., 2022; Khacha- tryan et al., 2023; Qi et al.. 2023; Zhang et al., 2023; Ceylan et al., 2023) attempt to extend the existing advanced diffusion models for text- to-image generation to text-to-video editing model by inflating spatial self-attention into spatio-temporal self-attention. Specifically, the features of the patches from different frames in the video are combined in the extended spatio- temporal attention module, as depicted in Fig- ure 2. By capturing spatial and temporal con- text in this way, these methods require only few fine-tuning steps or even no training to ac- complish T2V editing. Nevertheless, this sim- ple inflation operation introduces irrelevant in- formation since each patch attends to all other patches in the video and aggregates their fea- tures in the dense spatio-temporal attention. The irrelevant patches in the video can mislead the attention process, posing threat to the con- sistency control of the edited videos. As a result, these approaches still fall short of the visual consistency challenge in text-to-video editing.
F
Optical
Spatial attention
F2
F3
F
F
Spatio-temporal attention
Flow-guided attention
Figure 2: Illustration of spatial attention, spatio- temporal attention, and our flow-guided attention. The patches marked with the crosses attend to the colored patches and aggregate their features. Fi indicates the feature map of the k-th video frame.
In this paper, for the first time, we propose FLATTEN, a novel (optical) FLow-guided ATTENtion that scamlessly integrates with text-to-image diffusion models and implicitly leverages optical flow for text-to-vides editing to address the visual consistency limitation in previous works. FLATTEN enforces the patches on the same flow path across different frames to attend to each other in the attention module, thus improving the visual consistency of the edited video. The main advantage of our method is that enables the information to communicate accurately across multiple frames guided by optical flow, which stabilizes the prompt-generated visual content of the edited videos. More specifically, we first use a pre-trained optical flow prediction model (Teed & Deng, 2020) to estimate the optical flow of the source video. The estimated optical flow is then used to compute the trajectories of the patches and guide the attention mechanism between patches on the same trajectory. Meanwhile, we also propose an effective way to integrate flow-guided attention into the existing diffusion process, which can preserve the per-frame feature distribution, even without any training- We present a T2V editing framework utilizing FLATTEN techniques such as DDIM inversion (Mokady et al., 2023) and feature injection (Tumanyan et al., 2023). We observe high-quality and highly consistent text-to-video editing, shown in Figure 1. Furthermore, our proposed method can be easily integrated into other diffusion-baase text-to-video editing methods and improve the visual consistency of their edited videos.
foundation and employing T2I editing
The contributions of this work are as follows: (1) We propose a novel flow- www-guided attention (FLAT- TEN) that enables the patches the same flow path across different frames to attend to each other
2

---

Published as a conference paper at ICLR 2024
during the diffusion process and present framework based on FLATTEN for high-quality and highly consistent T2V editing. (2) Our proposed method, FLATTEN, can be easily integrated into existing text-to-vides editing approaches without any training fine-tuning to improve the visual consistency of their edited results. (3) We conduct extensive experiments to validate the effectiveness of our method. Our model achieves the new state-of-the- af-t performance on existing text-to-video editing benchmarks, especially in maintaining visual consistency.
2 RELATED WORK
Image and Video Generation Image generation is a popular generative task in computer vision. Deep generative models, e.g., GAN (Karras al., 2019; Kang et al., 2023) and auto-regressive Transformers (Ding et al., 2021; Esser et al., 2021; Yu et al., 2022) have demonstrated their capacity. Recently, diffusion models (Ho et al., 2020; Song et al., 2020a;b) have received much attention due to their stability. Many T21 generation methods based on diffusion models have emerged and achieved superior performance (Ramesh et al., 2021; 2022; Saharia et al., 2022; Balaji al., 2022). Some of these methods operate in pixel space, while others work in the latent space of an auto-encoder.
Video generation (Le Moing et al., 2021; Ge et al., 2022; Chen et al., 2023a; Cong et al., 2023; Yu et al., 2023; Luo et al., 2023) can be viewed as an extension of image generation with ad- ditional dimension. Recent video generation models (Singer al., 2022: Zhou al., 2022; Ge et al., 2023) attempt to extend successful text-to-image generation models into the spatio-tempor domain. VDM (Ho et al., 2022b) adopt a spatio-temporal factorized U-Net for denoising while LDM (Blattmann et al., 2023) implement video diffusion models in the latent space. Recently, con- trollable video generation (Yin et al., 2023; Li et al., 2023; Chen et al., 2023b; Teng et al., 2023) guided by optical flow fields facilitates dynamic interactions between humans and generated content.
Text-to-Image Editing T21 editing is the task of editing the visual appearance of a source image based on textual prompts. Many recent methods (Avrahami al., 2022; Couairon et al., 2022; Zhang & Agrawala, 2023) work on pre-trained diffusion models. SDEdit (Meng et al., 2021) adds noise to the input image and performs denoising through the specific prior. Pix2pix-Zero (Parmar al., (Hertz et al., 2022) manipulates
2023) performs cross-attentio guidance while the cross-attention layers directly. PNP-Diffusion (Tumanyan et al., 2023) saves diffusion features during reconstruction and injects these features during T2I editing. While video editing can benefit from these creative image methods, relying on them exclusively can lead to inconsistent output.
Text-to- Vixdeo Editing Gen-1 (Esser et al., 2023) demonstrates structure and content-driven video editing model while Text2Livv (Bar-Tal et al., 2022) uses a layered video representation. However, training these models is very time-consuming. Recent works attempt to extend pre-trained image diffusion models into a T2V editing model. Tune-A-Vided (Wu et al., 2022) extends a latent diffusion model the spatio-temporal domain and fine-tunes it with source videos, but still has difficulties in modeling complex motion. Text2 Video-Zeor (Khachatryan et al., 2023) and Con- Vioeo (Zhang et al., 2023) use ControlNet (Zhang & Agrawala, 2023) to help editing. They can preserve the per-frame structure but relatively lack control of visual consistency. FateZero (Qi et al., 2023) introduces an attention blending block to enhance shape-aware editing while the editing words have to be specified. To improve consistency, TokenFlow (Geyer al., 2023) enforces linear com- binations between diffusion features based on source correspondences. However, the pre-defined combination weights are not adapted to all videos, resulting in high- figh-freeuency flickering.
Different from the aforementioned methods, we propose a novel flow-guided attention, which im- plicitly uses optical flow to guide attention modules during the diffusion process. Our framework can improve the overall visual consistency for T2V editing and can also be seamlessly integrated into existing video editing frameworks without any training or fine-tuning.
METHODOLOGY
3.1 PRELIMINARIES
Latent Diffusion Models Latent diffusion models operate in the latent space with an auto-encoder and demonstrate superior performance in text-to-image generation. In the forward process, Gaussian
3

---

Published as a conference paper at ICLR 2024
Eg d
FLATTEN Integration
DOIM
Projectonn
latent Teature
source video
Progction
Irgpetoo Sumpinn
optical flw
lotent
Eg i
Denoising
Conv
attention
temporal
ourput video
Flow-guided attertion
Test cross-attention
Editing Prompt: "A cat, Van Gogh style."
Figure 3: Overview of our framework. We inflate the existing U-Net architecture along the temporal axis and combine flow-guided attention (FLATTEN) with dense spatio-temporal attention to avoid introducing any new parameters. The outcome of dense spatio-temporal attention H is further used for FLATTEN. The keys and values for FLATTEN are gathered from H based on the patch trajectories sampled from the optical flow. The weights of the U-Net cæ are frozen.
noise is added to the latent input Zg. The density of 2, given 2 can be formulated as:
q N v 3,
(I)
where 3, is the variance schedule for the timestep t. The number of timesteps used to train the diffusion model is denoted by T.. The backward process uses a trained U-Net eg for denoising:
pa (= x))= (((2-11 T, t), (= T,t)),
where T indicates the textual prompt. Jig and Eg are computed by the denoising model ep-
DDIM Inversion DDIM can convert a random noise to deterministic during sampling (Song et al., 2020a; Dhariwal & Nichol, 2021). Based on the assumption that the ODE process can be reversed in the small-step limit, the deterministic DDIM inversion can be formulated as:
az -1 c0(2)),
V
+ Vo++l
(3)
at
04++
where Or denotes (1 B:). DDIM inversion employed to invert the input 2n into ø7, which can be used for reconstruction and further editing tasks.
3.2 OVERALL FRAMEWORK
Our framework aims to edit the source video ) according to an editing textual prompt t and output visually consistent video. To this end, we expand the U-Net architecture of a T2I diffusion model along the temporal axis inspired by previous works (Wu et al., 2022; Khachatryan et al., 2023; Zhang et al., 2023). Furthermore, facilitate consistent T2V editing, (FLATTEN) into the U-Net blocks without introducing new parameters. To retain the high-fidelity of the generated video, we employ DDIM inversion in the latent space with re-designed U-Net to estimate the latent noise 27 from the source video. We use empty text for DDIM inversion without the need to define a caption for the source video. Lastly, we generate an edited video using the DDIM process with inputs from the latent noise žy and the target prompt т. Our framework as illustrated in Figure is training-fres thus comfortably reducing additional computation.
incorporate flow-guided attention
4

---

Published as a conference paper at ICLR 2024
U-Net Inflation The original U-Net architecture employed in image- based diffusion model comprises a stack of 2D convolutional residual blocks, spatial attention blocks, and cross-attention blocks that incorporate textual prompt embeddings. To adapt the T2I model to the T2V editing task, we inflate the convolutional residual blocks and the spatial attention blocks. Similar to previous works (Ho et al., 2022b; Wu et al., 2022), the 3 x 3 convolution kernels in the convolutional residual blocks are converted to 1 1x3x3 kernels by adding a pseudo temporal channel. In addition, the spatial attention replaced with dense spatio-temporal attention paradigm, In contrast to the spatial self- attention strategy applied to the patches in a single frame, we adopt all patch embeddings across the entire video as the queries (Q), keys (K), and values (V). This dense spatio-temporal attention can provide a comprehensive perspective throughout the video. Note that the parameters of the linear projection layers and the feed-forward networks in the new dense spatio-temporal attention blocks are inherited from those in the original spatial attention blocks.
FLATTEN Integration To further improve the visual consistency of the output frames, we in- tegrate our proposed flow-guided attention in the extended U-Net blocks. We combine FLATTEN with dense spatio-temporal attention since both attention mechanisms are designed to aggregate visual context. Given the latent video features, we first perform dense spatio-temporal attention. Specific linear projection layers are employed to convert the patch embeddings of the latent features into the queries, keys, and values, respectively. The results of dense spatio-temporal attention are denoted as H. To avoid introducing newly trainable parameters and preserve the feature distribu- tion, we do not apply new linear transformations to recompute the queries, keys, and values. We directly use H the input of flow-guided attention. Note that positional encoding is introduced. When a patch embedding serves as a query, the corresponding keys and the values for FLATTEN are gathered from the output of dense spatio-tempore attention H based on the patch trajectories sam- pled from optical flow. More details are demonstrated in Section 3.3. After performing flow- sow-guided attention, the output is forwarded the feed-forward network from the dense spatio-temporal at- tention block. We activate FLATTEN not only during DDIM sampling but also when performing DDIM inversion since using FLATTEN DDIM inversion allows a more efficient inversion by introducing additional temporal dependencies. More details are discussed in Appendix A.
We also implement the feature injection following the image editing method (Tumanyan et al., 2023). For efficiency, we do not reconstruct the source video but inject the features from DDIM inversion during sampling. With these adaptations, our framework establishes and enhances the connections between frames, thus contributing to high-quality and highly consistent edited videos.
3.3 FLOW-GUIDED ATTENTION
Optical Flow Estimation Given two consecutive RGB frames from the source video, we use RAFT (Teed & Deng, 2020) to estimate optical flow. The optical flow between two frames denotes a dense pixel displacement field (fz: fu). The coordinates of each pixel (æy, yk) in the k-th frame can be projected to its corresponding coordinates in the (k + 1)-th frame based on the displacement field. The new coordinates in the (k- + 1)-th frame can be formulated as:
(II+1, 11++) = (Ix + (a yk), yk + fa (Tk, 91))
(4)
In order to implicitly use optical flow to guide the attention modules, we downsample the displace- ment fields of all frame pairs to the resolution of the latent space.
Patch Trajectory Sampling We sample the patch trajectories in the latent space based on the downsampled fields (fz, We start iterating from the patches the first frame. For patch with coordinates (No. Wo) on the first frame, its coordinates on all subsequent frames can be derived from the displacement field. The coordinates are linked, and the trajectory sequence can be presented as:
(5) where K denotes the frame number of the source video. For a latent space with the size H W. there is ideally trajectory set denoted as {traji, traj2. ..., trajN) ). where N = HW. However, certain patches disappear over time, and new patches appear in the video. For each new patch that appears in the video, new trajectory is created. As result, the size of the trajectory set N is generally larger than HW. To simplify the implementation of flow- flow-guided attention, when an occlusion happens, randomly select trajectory continue sampling and stop the other conflicting trajectories. This strategy ensures that each patch in the video is uniquely assigned to a single trajectory, and there is no case where a patch on multiple trajectories.
traj= {(r0, 3o), (I1, 91), (z2, U2). ... , (*K. uK)},
5

---

Published as a conference paper at ICLR 2024
Fatsh
quary
key
VIELM
Multisheed Actention
Figure 4: Illustration of FLATTEN. We use RAFT to estimate the optical flow of the source video and downsample them to the resolution of the latent space. The trajectories of the patches in the latent space sampled based the displacement field. For each query, we gather the patch embeddings on the same trajectory from the latent feature as the corresponding key and value. The multi-head attention is then performed, and the patch embeddings are updated.
Attention Process Flow-guided attention is performed on the sampled patch trajectories. The overview of FLATTEN is illustrated Figure 4. We gather the embeddings of the patches on the same trajectory from the latent feature The patch embeddings on trajectory traj can be presented as:
Ztraj (=(o,,y))) (21,, y1), ( (. [x),
(6)
where z =<<x, yh indicates the patch embedding the coordinates (IXx, ykk in the k-th frame. We per- form multi-head attention with the patch embeddings on the same trajectory. For query z(Tx, ykk ) the corresponding keys and values are the other patch embeddings on the same trajectory traj. No additional position encoding is introduced. Our flow-guided attention can be formulated as follows:
Q = =(TN, Ikk), K V Ztroj {2(IK, vn)),
Attn(Q. K, V) = Softmax QKT
V
where Vd is scaling factor. The latent features updated by flow-guided attention to eliminate the negative effects from feature aggregation of irrelevant patches dense spatio-temporal attention. Importantly, ensure that each patch embedding the latent feature is uniquely assigned single trajectory during patch trajectory sampling. This assignment resolves conflicts and allows for a comprehensive update of all patch embeddings.
We utilize optical flow to connect the patches in different frames and sample the patch trajectories. Our flow-guided attention facilitates the information exchange between patches the same trajec- tory, thus improving visual consistency in video editing. We integrate FLATTEN into our framework and implement text-to-video editing without any additional training. Furthermore, FLATTEN can also be easily integrated into any diffusion-based T2V editing method, as shown in Section 4.4.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets We evaluate our text-to-video editing framework with 53 videos sourced from LOVEU- TGVE'. 16 of these videos are from DAVIS (Perazzi al., 2016), and we denote this subset as TGVE-D. The other 37 videos are from Videvo, which are denoted as TGVE-V. The resolution of the videos re-scaled 512 512. Each video consists of 32 frames labeled with ground-truth caption and 4 creative textual prompts for editing.
Evaluation Metries As per standard (Wu et al., 2022; Qi et al., 2023; Ceylan et al., 2023; Geyer al., 2023), we use the following automatic evaluation metries: For textual alignment, we use CLIP (Radford et al., 2021) to measure the average cosine similarity between the edited frames and the textual prompt, denoted as CLIP-T. To evaluate visual consistency, we adopt the flow warping error Ewarp (Lai et al., 2018), which warps the edited video frames according to the estimated optical
Prttps:/aati goog w.com//veww// wwwuuvpr23/traat

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results on TGVE-D and TGVE-V.
TGVE-D
TOVE-V
Method
CLIP-Ft
PiekSeore
CLIPTT'
Evarn
Swa f
CLIP-F
PickScore
CLIP-T
Exerp
S.Rt
Time-A. Videa
11.58
27.31
29.21
9.35
20.20
15.58
Text2Videu Zero
92.39
20.32
27.86
22.07
12.62
96.84
20.43
26.53
11.55
22.97
Control
27.72
40.70
20.36
25.92
ZateZero
92.58
20.45
27.06
5.79
46.74
96.64
20.09
25.72
5.10
50.43
Flow
92.45
20.93
26.91
5.36
50.21
96.72
20.61
25:57
3.15
81.17
FLAITEN (ours)
92.49
20.95
28.05
4.92
57.01
96.75
20.63
26.70
3.16
84,49
flow of the source video and measures the pixel-level difference. Using these metrics independently cannot comprehensively represent editing performance. For instance, Ewarp reports 0 errors when the edited video is exactly the source video. Therefore, we propose Sedit as our main evaluation metric, which combines CLIP-T and Euarp as unified score. Specifically, the editing score is calculated as Sedit = CL.P-T//Ewarp Following the previous work (Wu et al., 2022), we also adopt CLIP-F and PickScore, which computes the average cosine similarity between all frames in a video and the estimated alignment with human preferences, respectively. For brevity, the numbers of CLIP-F/CLIP-T/Euee shown in this paper are scaled up by 100/100/1000.
Implementation Details We inflate pre-trained text-to-image diffusion model and integrate FLATTEN into the U-Net to implement T2V editing without any training or fine-tuning. To es- timate the optical flow of the source videos, we utilize RAFT (Teed & Deng, 2020). We find that applying flow-guided attention in DDIM inversion can also improve latent noise estimation by in- troducing additional temporal dependencies. Therefore, we use flow-guided attention both in DDIM sampling and inversion. More details are shown in Appendix A. We implement 100 timesteps for DDIM inversion and 50 timesteps for DDIM sampling. Following the image editing method (Tu- manyan et al., 2023), the diffusion features are saved during DDIM inversion and are further injected during sampling. To efficiently perform the dense spatio-temporal attention in the modified U-Net, we use xFormers (Lefaudeux et al., 2022), which can reduce GPU memory consumption.
4.2 QUANTITATIVE COMPARISON
We compare our approach with publicly available text-to-video editing methods: Tune-A- Video (Wu et al., 2022), FateZero (Qi et al., 2023), Text2Video-Zero (Khachatryan et al., 2023), ControlVideo (Zhang et al., 2023), and TokenFlow (Geyer et al., 2023). In these methods, Tune-A- Video requires fine-tuning the source videos. Both Tune-A-Video and FateZero need the additional caption of the source video, while our model does not. Vixeo-ideo-Zer and Control trolNet (Zhang & Agrawala, 2023) to preserve the structural information. Edge maps are used as the condition in our experiments, which have better performance than depth maps. TokenFlow linearly combines the diffusion features based on the correspondences of the source video features.
use Con-
Table shows the quantitative comparisons of TGVE-D and TGVE-V. Our approach outperforms other compared methods in terms of CLIP-T, PickScore, and editing score Sadit on both datasets. In terms of the warping error Ewarp. our method is slightly 0.1 x 10 lower than TokenFlow. While considering textual faithfulness, our CLIP-T score is significantly higher. As a result, our method has higher editing score overall. Text2 weakly in terms visual consistency. Although FateZero has the highest CLIP-F on TGVE-D, its output video is sometimes very similar to the source video due to the hyperparameter setting issue. Our approach demonstrates superior performance on all evaluation metrics.
has high CLIP-F and CLIP-T, but performs
4.3 QUALITATIVE RESULTS
The qualitative comparison is presented in Figure 5. The source video at the top is from TGVE- D. and the source video at the bottom is from TGVE-V. Tune-A-Vide generates videos with high quality per frame, but it struggles to preserve the source structure, e.g., the wrong number of trucks. FateZero sometimes cannot edit the visual appearance based on the prompt, and the output video is almost identical to the source, as shown in the top example. Both Text2Video-Zerma and Control rely on pre-existing features (e.g., edge maps) provided by ControlNet. If the source condition features are of low quality, for example, due to motion blur, this leads to an overall decrease in video editing quality. TokenFlow samples keyframes and performs a linear combination of features to keep visual consistency. However, the pre-defined combination weights may not be appropriate for all videos. In the example at the bottom, a white intermittently appears and disappears in the
7

---

Published as a conference paper at ICLR 2024
Source video
Tune-A-Video
FateZero
Text2Video-Zero
ControlVideo
TokenFlow
FLATTEN (ours)
Editing Prompt:
Wooden trucks drive racetrack."
Tune-A-Video
FateZero
Text2Video-Zero
ControlVideo
(ours)
Editing Prompt: "A ship sails on the sea during sunset, 2D vector art."
Figure 5: Qualitative comparison between advanced T2V editing approaches and our method. The first column shows the source frames from TGVE-D (top) and TGVE-V (bottom), while the other columns present the corresponding frames edited by different methods. The complete videos are provided in the supplementary material.
frames edited by TokenFlow. In contrast, method generate consistent videos based the prompt with flow-guided attention. More qualitative results are shown in Appendix B.
4.4 PLUG-AND-PLAY FLATTEN
FLATTEN can be seamlessly integrated into other diffusion-based T2V editing methods. To verify its compatibility, we incorporate FLATTEN into the Net blocks of ControlVideo (Zhang et al., 2023). The visual consistency of the videos edited by ControlVideo with FLATTEN is significantly improved, as shown in Figure 6. The fish (cyan box) in the bottom frame edited by the original ControlVideo disappears while using FLATTEN ensures a consistent visual appear- ance. We evaluate the ControlVideo with FLATTEN on TGVE-D. After integrating FLATTEN, the warping error Ewarp decreases remarkably from 6.81 to 4.78,
ControlVideo
"Several goldfish swim pond."
while CLIP-T slightly decreases from 27.72 to 26.97. The editing score Sedst is improved from 40.70 to 56.42, which shows that FLATTEN can improve visual consistency for other T2V editing methods.
Figure 6: FLATTEN also improve vi- sual consistency for other methods.
4.5 ABLATION STUDY
To verify the contributions of different modules to the overall performance, we systematically deac- tivated specific modules our framework. Initially, we ablate both dense spatio-temporal attention (DSTA) and flow-guided attention (FLATTEN) from
framework. The dense spatio-temporal attention replaced by the original spatial attention in the pre-trained image model. This is viewed baseline model (Base). As shown in Figure 7, the edited structure is sometimes distorted. We individually activate DSTA and FLATTEN. They both can reason about temporal dependen- cies and enhance structural preservation and visual consistency. As further step, combine
8

---

Published as a conference paper at ICLR 2024
Source video with the editing prompt A colored wolf."
Base+FLATTEN
Base
Base-DSTA-FLATTEN (I)
Base+DSTA
FLAATEN (II)
Figure 7: Qualitative results on the effectiveness of flow-guided attention (FLATTEN) and dense spatio-tempore attention (DSTA). We also explore two combinations of FLATTEN and DSTA. To easily compare visual consistency, we zoom in the area of nose in different frames. In the lower right frames, both the structure as well as the colorization is temporally consistent.
Table 2: Ablation results for dense spatio-temporal atten- tion (DSTA), flow-guided attention (FLATTEN), and their combinations TGVE-D.
Table 3: User study of different T2V editing methods. The numbers indicate the average user preference rating (%).
Method
CLIP-T
Error
S.de 7
Consisiency
Base
28.36
13,40
21.16
Tune-A Vidioo
18:43
7.42
8.18
Base DSTA
27.97
6.65
42.06
Texi2Vioo-Zaao
11.01
4.49
4.21
ControlVideo
Base + FLATTEN
28.02
6.27
44.69
FateZero
8.09
13.26
17:76
FLATTEN
Taken Flow
18.65
26.74
24.30
Base DSTA FLATTEN
28.05
4.92
$7.01
FLAUTEN
DSTA and FLATTEN in two distinct ways and explore their effectiveness: (I) The output of dense peatio-temporal attention is forwarded the linear projection layers to recompute the queries, keys, and values for FLATTEN; (II) The output of DSTA is directly used as queries, keys, and values for FLATTEN. We find that the first combination sometimes results in blurring, which reduces the editing quality. The second combination performs better and is adopted as the final solution. The quantitative results for the ablation study on TGVE-D are presented Table 2.
4.6
USER STUDY
study since automatic metrics cannot fully represent human perception. We collect 180 edited videos and divide them into 30 groups. Each group consists of 6 videos edited by different methods with the source video and prompt. We asked 16 participants vote on their preference from the following perspectives: (1) semantic alignment (2) visual consistency, and (3) motion and structure preservation. The average user preference rating shown Table 3. Our method achieves higher user preference in all perspectives. More details are shown in Appendix C.
We conduct
5 CONCLUSION
We propose FLATTEN, novel flow-guided attention to improve the visual consistency for text-to- video editing, and present a training-free framework that achieves the new state-of-the-art perfor- mance the existing T2V editing benchmarks. Furthermore, FLATTEN can also be seamlessly integrated into any other cettusioo-basee T2V editing methods to improve their visual consistency. We conduct comprehensive experiments validate the effectiveness of our method and benchmark the task of text-to- video editing. Our approach demonstrates superior performance, especially in maintaining the visual consistency for edited videos.

---

Published as a conference paper at ICLR 2024
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXi::2211.01322 2022.
Omer Bal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In European conference on computer vision, pp. 707-723. Springer, 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhor,, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575 2023.
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. arXiv preprint arXiv:2303. 126888, 2023.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprint 12.04527, 2023a.
Tsai-Shien Chen, Chieh Hubert Lin, Hung- Yu Tseng, Tsung-Yr Lin, and Ming- Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304. 14404, 2023b.
Yuren Cong, Jinhui Yi, Bodo Rosenbahn, and Michael Ying Yang. Ssgvs: Semantic scene graph- to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2554 2564, June 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthicu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210.1 142,7 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 079400087448-00 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 198222--988 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12883 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint 2xxv:2305.104111 2023.
Michal Geyer, Omer Bar-Tal Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arxiv:2307. 10373, 2023.
10

---

Published as a conference paper at ICLR 2024
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. coompt-to-proman image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 68511, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023003 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022b.
Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9772-9781, 2021.
Minguk Kang. Jun-Yay Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 4401-4410, 2019.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13449 2023.
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In Proceedings of the European conference on com- puter vision (ECCV), pp. 170-185, 2018.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: context-aware controllable video synthesis. Advances in Neural Information Processing Systems, 34: 34:14042-14055, 2021.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https: /ttp:://gtthin /ttpttttttttt//////// com/ facebookresearch http//ooorrrrrrsooo 2022.
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. arXiv preprint arXiv:2309.07906 2023.
Zhengxiong Luo. Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 12209-10218, 2023.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Fol- low your pose: gusdeg text-to-video generation using pose-free free videos. arXiv preprint arXiv:2304.01 186, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.00 01077, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.

---

Published as a conference paper at ICLR 2024
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang. Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp- 1-11, 2023.
Federico Perazzi, Jordi Tusee,,set, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Workine-Hornune A benchmark dataset and evaluation methodology for video object segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 724-732, 2016.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint wrXiv:2303.09533 2023.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.15169, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning. pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):3, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479 -6494,, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rX1v:2011.134456, 2020b.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pp. 402-419. Springer, 2020.
Yao Teng, Enze Xie, Yue Wu, Haoyu Han, Zhenguo Li, and Xihui Liu. Drag-a-video: Non-rigid video editing with point-based interaction. arXiv preprint arXiv:2312.02936, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to tange-to-image translation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:22 1565, 2022.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308 20..800088 2023.
12

---

Published as a conference paper at ICLR 2024
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10459-10469, 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.005433 2023.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:230..13077, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11015 2022.
13

---

Published as a conference paper at ICLR 2024
A DDIM INVERSION WITH FLATTEN
Nlowguuided attention (FLATTEN) can also improve the DDIM inversion process, which critical in our T2V editing framework. We have validated the effectiveness of FLATTEN for the editing task in the ablation study (see Table 2). To further demonstrate that FLATTEN can contribute to high- quality latent noise estimation,
perform DDIM inversion the source videos and reconstruct them using the U-Net with and without FLATTEN, respectively. When activating FLATTEN during DDIM inversion, more details in the source video can be restored, such the eyes of the goldfish in Figure 8. Quantitatively, using FLATTEN results in higher scores for reconstruction metrics, with PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index measure) reaching the values of 33.89dB and 0.9159, respectively. In contrast, PSNR and SSIM of the reconstruction without FLATTEN drop to 32.74dB and 0.8974. The quantitative results are shown in Table 4.
Table 4: The results of DDIM inversion and reconstruction with and without FLATTEN.
Method
PSNR f
SSIM f
w/o FLATTEN
32.74dB
0.8974
w/ FLATTEN
33.89dB
0.9159
DDIM Inversion and Reconstruction without FLATTEN
DDIM Inversion and
Source video
Reconstruction with FLATTEN
Figure 8: Using FLATTEN during DDIM inversion helps to improve the quality of the estimated latent noise. This is reflected in video reconstruction. The fish eyes and other details in the third column are successfully reconstructed, while in the second column, some details are missing.
B ADDITIONAL QUALITATIVE RESULTS
The additional qualitative results are shown in Figure 9 and Figure 11. With flow-guided attention, training-fre free framework enables high-quality and highly consistent T2V editing.
To further demonstrate the visual consistency of videos generated by our approach, we provide the additional qualitative comparisons, which are shown in Figure 10. The videos produced by FLATTEN exhibit superior quality, characterized by a remarkable level of visual consistency and semantic alignment.
C USER STUDY DETAILS
We randomly sampled 30 source videos from TGVE-D and TGVE-V then edit them with 6 text- to-video editing approaches, including Tune-A-Video (Wu et al., 2022), FateZero (Qi et al., 2023), Texi2Video-Zero (Khachatryan et al., 2023), Control (Zhang et al., 2023), ControlNet (Zhang & Agrawala, 2023), TokenFlow (Geyer al., 2023) and our FLATTEN. For each group, we asked 16 participants vote their preference for 6 edited videos from the following perspectives:
Semantic Alignment: The edited videos should match the given editing prompt.
Visual Consistency: The adjacent frames in the edited vidcos should be smooth.
Motion and Structure Preservation: The motion/structure of the edited videos should align with the source video.
14

---

Published as a conference paper at ICLR 2024
Source video
"A detailed woolen toy cat." "
Source video
"Cartoon Style."
Source video
"Pointillism painting, detailed."
Figure 9: Additional qualitative results. material.
The complete videos are provided in the supplementary
15

---

Published as a conference paper at ICLR 2024
Source video
Tune-A-Video
FateZero
Text2Video-Zero
ControlVideo
TokenFlow
FLATTEN (ours)
Editing Prompt:
"Ski lift chairlifts moving up and down with desert background..
Source video
Tune-A-Video
FateZero
Text2Video-Zen
ControlVideo
TokenFlow
FLATTEN (ours)
Editing Prompt: "Cows graze together a yellow meadow.. "
Figure 10: Qualitative comparison between advanced text-to-vides editing approaches and FLAT-
TEN.
16

---

Published as a conference paper at ICLR 2024
Source video
""A husky."
""A metal sculpture."
"A dirty tiger."
"A cute pig." "
Figure 11: Our approach can output highly consistent videos conditional on different textual prompts.
An example of our user study interface is shown in Figure 12.
D LIMITATIONS
Our approach is designed for highly consistent text-to-video editing utilizing optical flow from the source video. Therefore, our approach excels in style transfer, coloring, and texture editing but is relatively limited in dramatic structure editing. A failure case is demonstrated in Figure 13. The shape of sharks is completely different from quadrotor drones. The model changes the original sharks into "mechanical sharks", but not drones.
TRAJECTORY VISUALIZATION
The flow estimator, Raft (Teed & Deng, 2020), has demonstrated its superior performance in many applications, being able accurately predict the flow field of dynamic videos. To demonstrate the
17

---

Published as a conference paper at ICLR 2024
Perspective: Visual consistency
Given Prompt: Cows graze together in a yellow meadow.
O 2
O
Figure 12: An example of our user study interface. Given source video with an editing prompt, users should select their preferred video from 6 videos edited by different T2V editing methods from different perspectives (e.g., visual consistency).
18

---

Published as a conference paper at ICLR 2024
Source video
Edited video
Two quadrotor drones swim in the blue ocean on a coral reef."
Figure 13: Our approach is relatively limited in dramatic structure editing, egg., turning sharks into drones.
Figure 14: Visualization of the patch trajectories. The trajectories computed based the downsampled flow fields (64 x 64) and the patches on the trajectories are marked with red dots.
robustness of the flow field estimation, we sample several predicted trajectories for video examples with large motion and visualize the trajectories in Figure 14. RAFT is robust even for videos with large and abrupt motions. Note that our approach does not rely on any specific flow estimation module. The trajectory prediction could be more precise with better flow estimation models in the future.
F ROBUSTNESS TO FLOwS
One notable advantage of our method is the integration of the flow field into the attention mecha- nism, significantly enhancing adaptability and robustness. To further demonstrate the robustness of FLATTEN the pre-computed optical flows, we add random Gaussian noise to the pre-computed flow field and use the corrupted flow field for video editing. The qualitative comparison is shown in Figure 15. The corrupted flow field results in few artifacts in the edited video (3rd row). However, the editing result still better than the output of the baseline model without using optical flow as guidance.
19

---

Published as a conference paper at ICLR 2024
Moreover, replace the optical flow from RAFT in flow-guided attention with the flow estimated by another flow prediction model, GMA (Jiang et al., 2021). The comparison is shown in Figure 16. There is obvious difference between the output videos and it shows that method is robust to small differences patch trajectories.
No Flow
Raft Flow
Raft Flow+Noise
Figure 15: Video editing results from the baseline model (1st row), FLATTEN with the Raft flow (2nd row), and FLATTEN with the noised flow (3rd row).
Raft
GMA
Figure 16: Comparison between using the optical flow from Raft (left) and GMA (right).
20

---

Published as a conference paper at ICLR 2024
G RUNTIME EVALUATION
To compare the computational cost of different text-to-video editing models, we measure the runtime required to edit single video (with 32 frames) by the different models. The runtime of the different models at different stages on a single A100 GPU is shown in Table 5. Our model has relatively short runtime in the sampling stage and there is scope for further improvement.
Table 5: Runtime evaluation of different T2V editing models.
Method
Finetuning
DDIM Inversion 3min52s
Sampling
Lune-A-Video (Wu et al., 2022)
15s
3min34s
Texe2 Zero (Khachatryan
al., 2023)
3min17s
ControlVides (Zhang et al., 2023)
4min36s
FateZero (Qi al., 2023)
4min56s
4min49s
TokenFlow (Geyer et al., 2023)
3min41s
3min29s
FLATTEN (ours)
3min52s
3min45s
21


Reviews:
[{"review_id": "kqi1UYWfGu", "reviewer": "Reviewer_UKpY", "summary": "Summary: The paper focuses on text guided video editing. Previous methods to tackle this problem extend the text-to-image U-net to the temporal dimension to implement spatiotemporal attention where patches from different frames attend to one another. The paper argues that such methods introduce irrelevant information, since they allow all patches in the video to attend to one another where in fact many of these spatiotemporal patch to spatiotemporal patch connections might be irrelevant. To address this problem, the paper suggests using optical flow to guide the attention. Specifically, a pre-trained optical flow network is used to estimate the flow field and tracks of patches along flow trajectories are aggregated to enforce only patches on the same trajectory to attend to one another in a second step of MHSA. This results in more visual-consistent videos as the paper demonstrate both qualitatively and qualitatively \n\nMethod: First, the \"standard\" the text to image U-net architecture is inflated to account for the temporal dimension and the image patch spatial self attention mechanism is replaced with spatiotemporal self-attention with all patches in the video used as tokens for Q,K,V. Secondly, a pre-trained optical flow network is employed to compute the flow field along the frames of the video. Tracks of patches (in the latent space) are aggregated using the downsampled flow field. Next, self attention is performed between patch-embeddings on the same track. Specifically, the queries are taken from the original dense spatiotemporal MHSA, but for every query associated with a specific patch - the keys and values in MHSA are only the ones which are associated with patches on the same track. Note that this method does not require re-training as it only refines the existing embedded spatiotemporal patch embedded tokens with additional information by applying MHSA again but with restrictions on which patches can attend to one another (where this \"restriction\" is derived from the flow field).\n\nExperiments: the paper compares the proposed method against 5 publicly available text-to-video editing methods on two standard benchmarks. The proposed method performs favourably both in terms of visual quality/alignment metrics  and visual consistency metrics. The paper also presents quantitative results as well as a user-study demonstrating the effectiveness of the method, particularly in the aspect of visual-consistency when motion is introduced.", "questions": "As I understand, the method is designed to improve visual consistency, particularly with respect to motion. The method relies on optical flow to \"enhance\" the embedded tokens with motion information derived from the flow field. As the pipeline relies on flow field computation and errors introduce in that step may affect the results. Something that is missing in the paper in my opinion is some discussion/experiments/results on how robust is the method to mistakes in the flow field computation. Specifically: \n\n1. How well the proposed method can handle large motion (large displacement in the flow field) or abrupt motion? are there any examples you can provide? \n2. How well the proposed method can handle videos in which both global motion (camera movement) and local motion (object movement) are present? are there any examples you can provide? \n3. Are there any situations where the method can do \"more harm than good\"? I mean, cases where the errors in the flow-field computation can cause the method to produce worse results than the baseline? how often do they occur? \n4. Are there any examples that you can provide in which the flow field is far from accurate? In those cases, are the results worse than the baseline, meaning the method did \"more harm than good\"?\n5. How does the results change with respect to accuracy of the flow field? For example, by taking a specific video and flow field results from several models where some perform dramatically worse than others or by gradually corrupting the flow field and measuring the affect on the results?\n\nI find that the qualitative results provided the supplemental video are extremely helpful (particularly the \"racing trucks\" example in which the results of other methods are provided). I would be grateful if the authors would be able to provide more examples that address the questions above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The proposed method is sound and original. The framework is very simple, does not require further training and can be easily plugged to various existing architectures. The paper is well written and the effectiveness of the method is demonstrated relatively well.", "weaknesses": "In my opinion, a drawback of the method is the heavy reliance on pre-computed flow field using a pre-trained network that is used as a black box. Thus, errors in this step can negatively affect the results of the proposed pipeline. However, the paper does not address this issue and there are no results to measure the robustness of the method. See question in the section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61Mo1KlEEa", "reviewer": "Reviewer_1FeZ", "summary": "To improve the visual consistency for text-to-video editing， FLATTEN is proposed to enforce the patches on the same flow path across different frames to attend to each other in the attention module. Experiment results on existing text-to-video editing benchmarks show that the proposed method achieves the new state-of-the-art performance.", "questions": "See Weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed Flow-guided attention is intuitive and makes sense.", "weaknesses": "1. The method is only suitable for scenarios where every pixel in the original video aligns spatially with the generated video. For misaligned areas, the optical flow trajectory of the original video is not appliable for the motion of the generated video, leading to incorrect key and query identifications. For instance, in the example of transforming a cat to a tiger in Figure 1, the tiger's face is larger than the cat and thus there exist pixels that belong to the tiger's face and belong to the background in the cat example. For the original video with the cat, these pixels belong to the background with an optical flow near zero. However, for the tiger, it's part of the face and should rotate with the head, requiring an optical flow describing a leftward movement. This sets too high a requirement for editing scenarios.\n\n2. If optical flow tracking is accurate enough, why not simply select a keyframe and then directly copy pixels following the same optical flow path to other frames? This approach seems more accurate than aligning through attention. For example, bilinear interpolation combined with optical flow is often used to predict the next frame in videos.\n\n3. In the provided MP4, there is only one visual comparison, which is too limited. It's suggested not to cherry-pick comparisons so that the effectiveness of the method can be judged intuitively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FHixBwB65R", "reviewer": "Reviewer_F5Lm", "summary": "This paper proposes text-guided video editing systems that considers optical flow to preserve temporal consitency\nIn detail the temporal attention is guided by the paths estimated from optical flow.", "questions": "My questions are above", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "[+] The idea makes sense that involves optical flow into the diffusion model for holding temporal consistency \n[+] Performances are enhanced compared to previous editing systems", "weaknesses": "[-] Optical flow can be effective when a single objective appears. However, we can easily come up with other cases including occlusion, objects' appearing or disappearing, or else. Therefore the method seems sensitive to the input video, which can ruin the attention or even worse than previous temporal attention methods. Can you explain why the method should be better than previous temporal attention?\n\n[-] The performances are better than many previous works. What are the samples that this method validates? I am quite curious about the videos that they evaluated.\n\n[-] Are there any qualitative or quantitative results about the trajectory patches in the aforementioned cases in question [1]? I want to see if the trajectory is truely following the flow of objectives in the video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IuyTFVkuyl", "reviewer": "Reviewer_Hpug", "summary": "The paper investigates the text-to-video editing task. To improve the temporal consistency, they introduce dense spatio-temporal attention and flow-guided attention to acquire information from the whole video. The proposed method achieves good performance on the test videos.", "questions": "The paper is good, but I still have some concerns as in the weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed flow-guided attention is novel and effective.\n\n2. The proposed flow-guided attention can be applied to other base models to further improve the temporal consistency.\n\n3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The comparison with previous works is very limited. For the video editing task, multiple images should be shown for qualitative comparison in the main paper (Figure 5), which is important to verify the temporal consistency. Furthermore, even in the supplementary material, only one example is compared with previous works, which is not convincing. I compared the ``wolf’’ example with the TokenFlow example on its website. The results of this paper are not good to me.\n\n2. From my perspective, the text information for editing is only acquired by cross attention with editing prompts. Other editing techniques like prompt-to-prompt are not used. This might cause the inaccurate editing of the background. For example, in the first example of Figure 5, the grass also turns yellow, while TokenFlow and FateZero can better keep the background.\n\n3. DSTA conducts cross attention across all 32 frames, which takes many computational resources. It is better to compare the computational cost and inference time with other methods.\n\n4. There are also some works based on optical flow trajectory for video generation [ref-1,ref-2,ref-3], which should also be discussed in the related work.\n\n[ref-1] Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.\n\n[ref-2] Generative image dynamics.\n\n[ref-3] Motion-Conditioned Diffusion Model for Controllable Video Synthesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)