COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing
Authors: Jiangshan Wang
Yue Ma
Jiayi Guo
Yicheng Xiao
Gao Huang
Xiu Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model; Video editing
Primary area: diffusion_based_models
TL;DR: We propose to leverage the inherent correspondence information of diffusion feature to achieve high-quality and consistent video editing.

Abstract: Video editing is an emerging task, in which most current methods adopt the pre-trained text-to-image (T2I) diffusion model to edit the source video in a zero-shot manner. Despite extensive efforts, maintaining the temporal consistency of edited videos remains challenging due to the lack of temporal constraints in the regular T2I diffusion model. To address this issue, we propose COrrespondence-guided Video Editing (COVE), leveraging the inherent diffusion feature correspondence to achieve high-quality and consistent video editing. Specifically, we propose an efficient sliding-window-based strategy to calculate the similarity among tokens in the diffusion features of source videos, identifying the tokens with high correspondence across frames. During the inversion and denoising process, we sample the tokens in noisy latent based on the correspondence and then perform self-attention within them. To save the usage of GPU memory and accelerate the editing process, we further introduce the temporal-dimensional token merging strategy, which can effectively reduce the redundancy. COVE can be seamlessly integrated into the pre-trained T2I diffusion model without the need for extra training or optimization. Extensive experiment results demonstrate that COVE achieves the start-of-the-art performance in various video editing scenarios, outperforming existing methods both quantitatively and qualitatively. The source code will be released.

COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing
Jiangshan Wang' Yue Ma Jiayi Guo" Yicheng Xiao', Gao Huang Xiu Li
Tsinghua University, "HKUST
https://coov- gitroo..tttuu.ooo
Source Video
Source Video
A man in blue clothes
Snow leopard
A Barbie Doll with blonde hair
The Lion King and his children
Sand sculptures by the sea
Cartoon style
Figure 1: We propose C'rrespondence-guidee Video Editing (COVE), which leverages the corre- spondence information of the diffusion feature achieve consistent and high-quality video editing. Our method is capable of generating high-quality edited videos with various kinds of prompts (style, category, background, etc.) while effectively preserving temporal consistency in generated videos.
Abstract
Video editing is an emerging task, in which most current methods adopt the pre- trained text-to-image (T2I) diffusion model to edit the source video zero-shot manner. Despite extensive efforts, maintaining the temporal consistency of edited videos remains challenging due the lack of temporal constraints in the regular T2I diffusion model. To address this issue, we propose COrrespondence-guidde Video Editing (COVE), leveraging the inherent diffusion feature correspondence to achieve high-quality and consistent video editing. Specifically, propose an efficient wodhiingwwiooww-- kens in the diffusion features of source videos, identifying the tokens with high correspondence across frames. During the inversion and denoising process, we
strategy to calculate the similarity among to-
"Equal contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sample the tokens in noisy latent based on the correspondence and then perform self-attentic within them. To save GPU memory usage and accelerate the editing process, we further introduce the remporal-dimensionn which can effectively reduce redundancy. COVE can be seamlessly integrated into the pre-trained T21 diffusion model without the need for extra training optimization. Extensive experiment results demonstrate that COVE achieves the start-of-the-arl performance in various video editing scenarios, outperforming ex- isting methods both quantitatively and qualitatively. The code will be release at https:/ /gi thub. www.coo//aaaaa coom/wanjjaa wawssaaannnannnaaaa waggsann022// COVE
token merging strategy,
Introduction
Diffusion models [27, 63, 65] have shown exceptional performance in image generation [57]. thereby inspiring their application in the field of image editing [6, 25, 7, 53, 67, 24]. These approaches typically leverage a pre-trained Text-to-Image (T2I) stable diffusion model [57]. using DDIM [64] inversion to transform source images into noise, which is then progressively denoised under the guidance of a prompt to generate the edited image.
Despite satisfactory performance in image editing, achieving high-quality video editing remains challenging. Specifically, unlike the well-established open-source T2I stable diffusion models [57], comparable T2V diffusion models are not as mature due to the difficulty of modeling complicated temporal motions, and training T2V model from scratch demands substantial computational resources [26, 29, 62]. Consequently, there is a growing focus on adapting the pre-trained T2I diffusion for video editing [16, 33, 11, 72, 73, 54]. In this case, maintaining temporal consistency in edited videos is one of the biggest challenges, which requires the generated frames to stylistically coherent and exhibit smooth temporal transitions, rather than appearing as a series of independent images. Numerous methods have been working on this topic while still facing various limitations, such as the inability to ensure fine-grained temporal consistency (leading to flickering [33, 54] or blurring [16] in generated videos), requiring additional components [30, 72, 11, 73] or needing extra training optimization [73, 69, 41], etc.
In this work, our goal is to achieve highly con- sistent video editing by leveraging the intra- frame correspondence relationship among to- kens, which is intuitively closely related to the temporal consistency of videos: If correspond- ing tokens across frames exhibit high similar- ity, the resulting video will thus demonstrate high temporal consistency. Taking vidco of man as an example, if the token representing his nose has high similarity across frames, his nose will be unlikely to deform or flicker through- out the video. However, how obtain accurate correspondence information among tokens is still largely under-explored in existing works, although the intrinsic characteristic of the video editing task (i.e., the source video and edited video are expected to share similar motion and semantic layout) determines that it naturally exists in the source video. Some previous methods [11. 73] leverage pre-trained optical-flow model to obtain the flowing trajectory of each token across frames, which can be seen as a kind of coarse correspondence information. Despite the self-attention among tokens in the same trajectory can enhance the temporal consistency of the edited video, it still encounters two primary limitations: Firstly, these methods heavily rely on a highly accurate pre-trained optical-flor floow model to obtain the correspondence relationship of tokens, which is not available in many scenarios [32]. Secondly, supposing we have access to an extremely accurate optical-flow model, it is still only able to obtain the coarse one-to-one correspondence among tokens in different frames (Figure 2a). which would lead to the loss of information because one token is highly likely to correspond to multiple tokens in other frames in most cases (Figure 2b).
Based pretrained Optical-
flow model
Optinal-aax
One-to-one correspondence
Frame
Frame 1 1
(a). Previous methods
Based on inherent diffusion
feature correspondence
One-to-many mo-o correspondence
Frame
Frame i +
COVE
Figure 2: Comparison between COVE (our method) and previous methods[11, 73].

---

Addressing these problems, we notice that the inherent diffusion features naturally contain precise correspondence information. For instance, it easy find the corresponding points between two images by extracting their diffusion features and calculating the cosine similarity between tokens [66]. However, until now none of the existing works have successfully utilized this characteristic in more complicated and challenging tasks such as video editing. In this paper, we propose COVE, which is the first work unleashing the potential of inherent diffusion feature correspondence to significantly enhance the quality and temporal consistency in video editing. Given source video, we first extract the diffusion feature of each frame. Then for each token in the diffusion feature, we obtain its corresponding tokens in other frames based on their similarity. Within this process, we propose heindoow-baee
sliding-window window-basee -baed approach to ensure computational efficiency. In our method, for each token, it is only required to calculate the similarity between it and the tokens in the next frame located within a small window, identifying the tokens with the top K (K > 1) highest similarity. After the correspondence calculation process, for each token, the coordinates of its K corresponding tokens in each other frame can be obtained. During the inversion and denoising process, we sample the tokens in noisy latents based on the obtained coordinates. To reduce the redundancy and accelerate the editing process, token merging is applied in the temporal dimension, which is followed by self-attention. Our method can be seamlessly integrated into the off-the-shelf T21 diffusion model without extra training or optimization. Extensive experiments demonstrate that COVE significantly improves both the quality and the temporal consistency of generated videos, outperforming wide range of existing methods and achieving state-of sate-of-the-arre fi-thh--rt art results.
2 Related Works
2.1 Diffusion-bassdd Image and Video Generation.
Diffusion Models [27, 63, 65] have recently showcased impressive results in image generation, which generates the image through gradual denoising from the standard Gaussian noise[12, 13, 52, 21, 57, 64, 22]. A large number of efforts diffusion models [28, 34, 60) has enabled it to be applied to numerous scenarios [3, 15, 35, 39, 44, 49, 50, 14, 58, 9, 23, 47, 42, 18]. With the aid of large-scale pretraining [55, 61], text-to-image diffusion models exhibit remarkable progress in generating diverse and high-quality images [51, 71, 56, 57, 59, 19, 48, 18J. ControlNet [75] enables users to provide structure layout information for precise generation. Naturally, diffusion models have found application in video synthesis, often by integrating temporal layers into image-based DMs [4, 26, 29, 70, 10]. Despite successes in unconditional video generation [29, 74, 45], texteoo-video diffusion models lag behind their image counterparts.
2.2 Text-to-Video Editing.
There are increasing works adopting the pre-trained text-to-image diffusion model the video editing task [43, 68, 69, 46, 20]. where keeping the temporal consistency in the generated video is the most challenging. Recently, large number of works focusing on zero-shot video editing has been proposed. FateZero [54] proposes to use attention blending to achieve high-quality edited videos while struggling to edit long videos. TokenFlow [16] reduces the effects of flickering through the linear combinations between diffusion features, while the smoothing strategy can cause blurring in the generated video. RAVE [33] proposes the randomized noise shuffling method, suffering the problem of fine details flickering. There are also large number of methods that enhance the temporal consistency with the aid of pre-trained optical-flow models [73, 72, 11, 30]. Although the effectiveness of them, all of them severely rely on pre-trained optical-flow model. Recent works [66] illustrate that the diffusion feature contains rich correspondence information. Although VideoSwap [17] adopts this characteristic by tracking the key points across frames, it still needs users to provide the key points as the extra addition manually.
3 Method
In this section, we will introduce COVE detail, which can be seamlessly integrated into the pre-trained T21 diffusion model for high-quality and consistent video editing without the need for training or optimization (Figure 3). Specifically, given a source video, we first extract the diffusion feature of each frame using the pre-trained T2I diffusion model. Then, we calculate the one-to-mant many
3

---

Source Video
Diffusion feature F
Correspondecce
Source Video
Inversion x T
(a). Sliding-winddow-bbssed Correspondence Calculation
"Cartoon Style"
Multi-head Self Attention
Mergp
Corresondence basedd
b
z
V
Edited Video
Denoisingx
token sampling
(b). Adopting the Carespondeneeinnttortii
(c). Overall Pipeline of Video-edditing
inflation
Convolation
Corresponaance
Self-attention Unet)
Spatial-temporal
Frozen 2D Unet
Frozzem 3D Unst
block
guided atlention
attention (in 3D Unet)
Text cooss-atention
Figure 3: The overview of COVE. (a). Given source video, we extract the diffusion feature of
each frame using the pre-trained T2I model and calculate the correspondence among tokens (detailed in Figure 4). (b). During the video editing process, we sample the tokens noisy latent based on correspondence and apply self-attention among them. (c). The correspondence-guiee attention can be scamlessly integrated into the T2I diffusion model for consistent and high-quality video editing.
correspondence of each token across frames based on cosine similarity (Figure 3a). To reduce resource consumption during correspondence calculation, we further introduce an efficient sliding- window-based strategy (Figure 4). During each timestep of inversion and denoising in video editing, tokens in the noisy latent are sampled based on correspondence and then merged. Through the self-attention among merged tokens (Figure 3b), the quality and temporal consistency of edited videos are significantly enhanced.
3.1 Preliminary
Diffusion Models. DDPM [27] is the latent generative model trained to reconstruct fixed forward Markov chain Z1,..., Tr. Given the data distribution To ~ g( the Markov transition (( (x+-1) is defined as a Gaussian distribution with a variance schedule 3r E (0, 1).
((<;/ ( = N(x;; /
To generate the Markov chain To. ... IT. DDPM leverages the reverse process with a prior distribu- tion p(æT) -x(r; II) and Gaussian transitions. A neural network Eg trained predict noises, ensuring that the reverse process is close to the forward process.
po( = (m T, t). Eo(z,,
where 7 indicates the textual prompt. M4 and Eg are predicted by the denoising model Since the diffusion and denoising process in the pixel space is computationally extensive, latent diffusion [57] proposed to address this issue by performing these processes in the latent space of VAE [37].
DDIM Inversion. DDIM can convert random noise deterministic 2g during sampling [64, 13].
The inversion process in deterministic DDIM can be formulated as follows:
a1+1 at
F141
Dz + Var+1
co c0(x))
a++1
at
where oa denotes IT
B.). The inversion process of DDIM utilized transform the input 2o into 27, facilitating subsequent tasks such as reconstruction and editing.
3.2 Correspondence Acquisition
As discussed in Section 1, intra-frame correspondence is crucial for the quality and temporal consis- tency edited videos while remaining largely under-explored existing works. this section, we introduce method for obtaining correspondence relationships among tokens across frames.

---

F
F
Fi+1
Fi+z
F(43
A dijusting the center of the window to
P(ihww in Ft
Pl--
other tokens
oohhe tokens
The sliding window in Fi41 (centered wn(),,w))
Similarit Matrix
The sliding window in Fiva (osntered (/
Similanty Matrix
Figure 4: Sliding-window- strategy for correspondence calculation. represents the token represents the obtained corresponded tokens in other frames.
wo... and
Diffusion Feature Extraction. Given a source video V with N frames, VAE [37] is employed on each frame to extract the latent features Z = {.1.... zN ). where Z E R xxxxxx xd Here, H and W denote the height and width of the latent feature and denotes the dimension of each token. For each frame of Z, we add noise of specific timestep and feed the noisy frame Z' = (=f,--. ... =xx into the pre-trained T2I model fo respectively. The diffusion feature (i.e., the intermediate feature from the U-Net decoder) is extracted through single step of denoising [66]:
= {Fi} = (fo(z)),, (1,...,NN, ,N.,
where &NNxxxxx denoting the normalized diffusion feature of each frame.
One-to-many Correspondence Calculation. For each token within the diffusion feature F, its corresponding tokens in other frames are identified based on the cosine similarity. Without loss of generality, we could consider a specific token w(i.,...;) in the ith frame F; with the coordinate [h,, wy). Unlike previous methods where only one corresponding token of P(i.h.,w can be identified in each frame (Figure 2a), our method can obtain the one-to-many correspondences simply by selecting tokens with the top K highest similarity in each frame. We record their coordinates, which are used for sampling the tokens for self-attention in the subsequent inversion and denoising process. To implement this process, the most straightforward method is through a direct matrix multiplication of the normalized diffusion feature F.
F=F.F"
(5)
where S E ([W*HxW)x(NXXHXI) represents the cosine similarity between each token and all tokens in the diffusion feature of the video.
The similarity between w and N x The coordinates of the corresponding tokens in the jth frame (j selecting the tokens with the top K similarities in the jth frame.
tokens in the feature is given by Sli, h4, wi, 1, F, :] are then obtained by
h w top-k-an argmaax (t) h4 w. j.
,,)))) *),
(6)
Here the top-k-argmax(() denotes the operation to find coordinates of the top K biggest values in a matrix, where E {1,--. }.. [h wz) represents the coordinates of the token in jth frame
which has highest similarity with pi }. A similar process can be conducted for each token of F, thereby obtaining their correspondences among frames.
Sliding-window Strategy. Although the one-to one-to-many correspondence among tokens can be effec- tively obtained through the above process, requires excessive computational resources because (N x H x W) is always huge number, especially in long videos. As result, the computational complexity of this process is extremely high, which can be represented as O(N2 x H² W d). At the same time, multiplication between these two huge matrices consumes substantial amount
5

---

of GPU memory in practice. These limitations severely limit its applicability in many real-world scenarios, such as on mobile devices.
To address the above problem, we further propose the sliding- wwwdww--ddwwwsseee basedd strategy as an alternative, which not only effectively obtains the one-to-many correspondences but also significantly reduces the computational overhead (Figure 4). Firstly, for the token p it is only necessary to calculate its similarity with the tokens in the next frame Fi+1 instead of all frames, i.e.,
S F. F.FF i+11
S; E # denotes the similarity between the tokens in ith frame and those in (i+1 11th frame. The overall similarity matrix is S {S.]. i E 21, .N 1}, where S E R(N-1)xHxXXXXXX Then, we obtain the K corresponded tokens of mn in F,+1 through S,.
h w top-k-argmax (2+ wxt (S. w,,x", 1²)),
(8)
For tokens in (i + 2)th frame, instead of considering w,,, we identify the tokens in (i + 2)th frame which have the top K largest similarity with the token P(i+1.,4 through the S Similarly, we obtain the corresponding token in other future or previous frames.
h 2 uft2 top-k-argnan argmax (a* (a*,y* 3
) (S
[h]+1 w x* y²1).
(9)
Through the above process, the overall complexity is reduced to O((N - 1) x H² x W x d). Furthermore, it is noteworthy that frames in video exhibit temporal continuity, implying that the spatial positions of corresponding tokens are unlikely to change significantly between consecutive frames. Consequently, for the token pii. it is enough to only calculate the similarity within small window of length / in the adjacent frame, where is much smaller than H and W.
F Fi+1(h; 1/2: h; + 1/2, w; 1/2 wt + 1/2,:]-
(10)
F E RixxXx represents the tokens in F+ within the sliding window. We calculate the cosine similarity between P(i.h.,..)) and the tokens in F selecting tokens with top K highest similarity within F.: This approach further reduces the computational complexity to O(NN 1) x H x W x 12 x d) and the GPU memory consumption is also significantly reduced in practice. Additionally, it worth noting that calculating correspondence information from the source video is only conducted once before the inversion and denoising process of video editing. Compared with the subsequent editing process, this process only takes negligible time.
3.3 Correspondence-guidded Video Editing.
In this section, we explain how to apply the correspondence information to the video editing process (Figure 3c). In the inversion and denoising process of video editing. we sample the corresponding tokens from the noisy latent for each token based on the coordinates obtained in Section 3.2. For the token o to the set of corresponding tokens in other frames at a timestep t is:
Corr = L wy J.j E {1,-..,. 1,++,, 1,... NN, E {I....,KK)
(1))
We merge these tokens following [5], which can accelerate the editing process and reduce GPU memory usage without compromising the quality of editing results:
Corr Merge (oorr.
(12)
Then, the self-attention is conducted on the merged tokens,
Q (n:)) K V Corr, Attention(4 Q, K, V) SoftMax
(13)
Q KT
V.
(14)
Vdz
where ddz is the scale factor. The above process of correspondence-guide in Figure 3b. Following the previous methods [73, 11], we also retain the spatial-tempore attention [69] in the U-Net. In spatial-temporal attention, considering a query token, all tokens in the video serve as keys and values, regardless of their relevance to the query. This correspondence-agnen self-attention is not enough to maintain temporal consistency, introducing irrelevant information into each token, and thus causing serious flickering effects [11, 16]. Our correspondenc--guidde attention can significantly alleviate the problems of spatial-temporal attention, increasing the similarity of corresponding tokens and thus enhancing the temporal consistency of the edited video.
attention is illustrated

---

Source Video
Source Video
Source Video
Ship in the river, evening glow
A car driving in sunset
A plushy toy of teddy bear
A driving in milky way
Van Gogh style
A cute raccoon
Source Video
Source Video
Source Video
Origami style
Milk pour out of the pot
sorrow woman
An ego flying in blue sky
Cartoon style
Happy woman with sunglasses
vidroevideo
In winter
Source Video
Traditional Chinese house
autumn
Wide sea and snow mountain
Figure 5: Qualitative results of COVE. COVE can effectively handle various types of prompts, generating high-quality videos. For both global editing (e.g., style transferring and background editing) and local editing (e.g.. modifying the appearance of the subject), COVE demonstrates outstanding performance. Results best-viewed zoomed-in.
Experiment
4.1 Experimental Setup
In the experiment, we adopt Stable Diffusion (SD) 2.1 from the official Huggingface repository for COVE, employing 100 steps of DDIM inversion and 50 steps of denoising. To extract the diffusion feature, the noise of the specific timestep & = 261 is added to each frame of the source video following [66]. The feature is then extracted from the intermediate layer of the 2D Unet decoder during single step of denoising. The window size is set to 9 for correspondence calculation, and & is set to 3 for surespondeece-guidee attention. The merge ratio for token merging is 50%. both qualitative and quantitative evaluation, we select 23 videos from social media platforms such as TikTok and other publicly available sources [1, 2]. Among these 23 videos, videos have length of 10 frames, 15 videos have length of 20 frames, and 5 videos have a length of 32 frames. The experiments

---

Source video
Courcevides
Source video
An old woman with white hair
Cartoon style
A polar bear
FabZZeo
RAVE
FRESCO
TokenFlow
COVE (Uug)) FLATTEN
HIIDIL
Figure 6: Qualitative comparison of COVE and various state-of-the-as methods. Our method outperforms previous methods across a wide range of source videos and editing prompts, demonstrat- ing superior visual quality and temporal consistency. Results are best-viewed zoomed-in.
conducted on single RTX 3090 GPU for our method unless otherwise specified. We compare COVE with bascline methods: FateZero [54], TokenFlow [16], FLATTEN [11]. FRESCO [73] and RAVE [33]. For all of these baseline methods, we follow the default settings from their official Github repositories. The more detailed experimental settings of our method are provided in Appendix A.
4.2 Qualitative Results
We evaluate COVE on various videos under different types of prompts including both global and local editing (Figure 5). Global editing mainly involves background editing and style transferring. For background editing, COVE can modify the background while keeping the subject of the video unchanged (e.g. Third row, first column. "a car driving
milky way"). For style transfer. COVE can effectively modify the global style of the source video according to the prompt (e.g. Third row, second column. "Van Gogh style"). Our prompts for local editing include changing the subject of the video to another one (e.g. Third row, third column. "A cute raccoon") and making local edits to the subject (e.g. fifth row, third column. "A sorrow woman""). For all of these editing tasks, COVE demonstrates outstanding performance, generating frames with high visual quality while successfully preserving temporal consistency. We also compare COVE with a wide range of state-of-the-art video editing methods (Figure 6). The experimental results illustrate that COVE effectively edits the video with high quality, significantly outperforming the previous methods.
4.3 Quantitative Results
For quantitative comparison, we follow the metrics proposed in VBench [31], including Subject Consistency, Motion Smoothness, Aesthetic Quality, and Imaging Quality. Among them, Subject Consistency assesses whether the subject (c.g., person) remains consistent throughout the whole

---

video by calculating the similarity of DINO [8] feature across frames. Motion Smoothness utilizes the motion priors of the video frame interpolation model [40] to evaluate the smoothness of the motion in the generated video. Aesthetic Quality uses the LAION aesthetic predictor [38] to assess the artistic and beauty value perceived by humans on each frame. Imaging Quality evaluates the degree of distortion in the generated frames (e.g., blurring, flickering) through the MUSIQ [36] image quality predictor. Each video undergoes editing with global prompts (such as style transferring, background editing, etc.) and 2 local prompts (such as editing the appearance of the subject in the video), generating a total of 115 text-video pairs. For each metric, we report the average score of these 115 videos. We further conducted user study with 45 participants following [73]. Participants required to choose the most preferable results among these methods. The result is shown Table 1. Among various methods, COVE achieves outstanding performance in both qualitative metrics and user studies, further demonstrating its superiority.
video
Subject Consistency
Mation Smoothhess
Aesthetic Quality
Imaging Quality
User
Study
FateZero [54]
0.9622
0.9547
0,6258
0.6951
7.4%
TokenFlow [16]
0,9513
0.9803
0.6904
0.7354
13.0%
FLATTEN
0,9617
0.9622
14.8%
FRESCO [73]
0.9358
0,9737
0.6582
0,6331
9.2%
Crochet Style
RAVE [33]
0.9518
0.9733
0.6369
0.7355
11.1%
COVE (Ours)
0
0,9892
0.7122
0.7441
44.5%
O/M
Table 1: Quantitative comparison among COVE and a wide range of state-of-the-art video editing meth- ods. The evaluation metrics[31] can effectively reflect the temporal consistency and frame quality of gener- ated videos. COVE illustrates superior performance in both keeping the temporal consistency and generating frames with high quality in edited videos.
K=l
K.
K= K=3 3
Subject Consistency
Motion Smoothness
Aesthetic Quality
Imaging Quality
w/o
0.9431
0.9049
0.6913
0.7132
K
0.9637
0.9817
0,6979
0.7148
0.9731
0.9892
0.7122
0.7441
0.9745
0.9886
0.7167
0.7429
Table 2:
Ablation study on the value of K in correspondence-guieel attention. w / means with- out correspondence-guider attention in Unet. When K 3 the quality of the video is the best.
Figure 7: Ablation study about the correspondence-guideed attention and the value of K. w o means do not apply purespondencc-guidee attention.
4.4 Ablation Study
We conduct ablation study to illustrate the effectiveness of the Correspondence-guidee attention and the number of tokens selected in each frame (i.e., the value of K). The experimental results (Table 2 and Figure 7) illustrate that without
ceisspondenceguideen attention, the edited video exhibits obvious temporal inconsistency and flickering effects (which is marked in yellow and orange boxes in Figure 7), thus severely impairing the visual quality. As K increases from 1 to 3, the generated video contains more fine-grained details, exhibiting better visual quality. However, further increasing K to 5 does not significantly improve the video quality. We also illustrate the effectiveness of temporal dimensional token merging. By merging the tokens with high correspondence across frames, the editing process becomes more efficient (Table 3) while there no significant decrease in the quality of the edited video (Figure 8). The ablation of the sliding-window size is shown in Appendix If the window size is too small, the actual corresponding token may not be included within the window, resulting in suboptimal correspondence and poor editing results. On the other hand, a too-large window size is not necessary for identifying the corresponding tokens, which would lead to high computational complexity and excessive memory usage. The experiment results illustrate that = 9 suitable to strike balance. Additionally, we also visualize the correspondence obtained by COVE, which is shown in Appendix C.

---

Source video
Correspondence
Token Merging
Memory Usage
Speed
Guided Attention
X
2.2 min
9 GB
X
2.7 min
14 GB
2.4 min
11 GB
Husky
Table 3: Ablation Study on the effeet of temporal dimensional token merging. Temporal dimensional token merging can speed up the editing process and save GPU memory usage while hardly impairing the quality of the generated video. The experiment is con- ducted on single RTX3090 GPU with 10-frame source video. k is set to 3.
Figure 8: Token merging would not im- pair the quality of edited video.
Conclusion
In this paper, we propose COVE, which is the first to explore how to employ inherent diffusion feature correspondence in video editing to enhance editing quality and temporal consistency. Through the proposed efficient biding-window-based strategy, the one-to-many correspondence relationship among tokens across frames is obtained, During the inversion and denoising process, self-attention is performed within the corresponding tokens enhance temporal consistency. Additionally, apply token merging in the temporal dimension to improve the efficiency of the editing process. Both quantitative and qualitative experimental results demonstrate the effectiveness of method, which outperforms a wide range of previous methods, achieving state-of-the- art editing quality.
also
Linitaions. The limitation of our method is discussed in Appendix G.
Acknowledgments and Disclosure of Funding
This work was supported by the STI Major Projects under Grant 20212D0201404,
References
[1] Pexels. attps://wwwt httpp:///www.pxxel...o wow.:/lss.oom..moommmmmmmm accessed: 2023-11-16
[2] Pixabay. https: com/, accessed: 2023-11-16
[3] Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of ral images. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18208-18218 (2022)
[4] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 22563-22575 (2023) 3
[5] Bolya, D., Fu, C.Y., Dai, X., Zhang. P., Feichtenhofer, C., Hoffman,, J.: Token merging: Your vit but faster. arXiv preprint arXiv:2210.0946) (2022) 6
[6] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 18392-18402 (2023)*
[7] Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., Zheng, Y.: Masactrl: Tuning-free mutual self- attention control for consistent image synthesis and editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22560-22570 (2023) 2
[8] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference computer vision. pp. 9650-9660 (2021)
10

---

[9] Chen, Q. Ma, Y., Wang, H.. Yuan. J., Zhao, W. Tian, Q. Wang, H., Min, S., Chen, Q. Liu, W.: Follow-your-cane canvas: Higher-resolution video outpainting with extensive content generation. arXiv preprint arXiv:2409.00 ..0050 (2024) 3
[10] Chen, X., Wang. Y., Zhang, L., Zhuang. S., Ma, X., Yu, J., Wang, Y., Lin, D., Qiao, Y., Liu, Z.: Seine: Short-to-long video diffusion model for generative transition and prediction. In: The Twelfth International Conference Learning Representations (2023) 3
[11] Cong. Y., Xu, M., Simon, C., Chen, S., Ren, J., Xie, Y., Perez-Rua, J.M., Rosenhahn, B., Xiang, T., He, S.: Flatten: optical Now-guided attention for consistent text-to-video editing, arXiv preprint arXiv:2310.05922 (2023) 2, 3, 6, 8,
[12] Croitoru, F.A., Hondru, V., Ionescu, R.T., Shah, M.: Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[13] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34,
(2021) 3,
[14] Fang. C., He, C., Xiao, F., Zhang, Y., Tang. L., Zhang, Y., Li, K., Li, X.: Real-world image dehazing with coherence-based label generator and cooperative unfolding network. arXiv preprint arXiv:2406.07966 (2024) 3
[15] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint urXiv:2208.016
(2022) 3
[16] Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T:: Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307. 10373 (2023) 2, 6, 8,
[17] Gu, Y., Zhou, Y., Wu, B., Yu, L.,, Liu, J.W., Zhao, R., Wu, J.Z., Zhang, DJJ., Shou, M.Z., Tang, K.: Videoswap: Customized video subject swapping with interactive semantic point correspondence. arXiv preprint arXiv:2312.0208 (2023) 3
[18) Guo, H., Dai, T., Ouyang, 7.,, Zhang, T., Zha, Y., Chen, B., Xia, S.t.: Refir: Grounding large restoration models with retrieval augmentation. arXiv preprint arXiv:2410.06 (2024) 3
[19] Guo, J., Du, C., Wang, J., Huang, H., Wan, P., Huang, G.: Assessing single image in reference- guided image synthesis. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 753-761 (2022) 3
[20] Guo, J., Manukyan, H., Yang. C., Wang. C., Khachatryan, L., Navasardyan, Song, S., Shi, H., Huang, G.: Faceclip: Facial image-to-video translation via a brief text description. IEEE Transactions on Circuits and Systems for Video Technology (2023) 3
[21] Guo, J., Wang, C., WU, Y., Zhang, E., Wang, K., Xu, X., Song, S., Shi, H., Huang, G.: Zero- shot generative model adaptation via image-specific prompt learning. In: Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition. pp. 11494-11503 (2023) 3
[22] Guo, J., Xu, X., Pu, Y., Ni, Z., Wang. C., Vasu, M., Song, S.. Huang, G., Shi, H.: Smooth diffusion: Crafting smooth latent spaces in diffusion models. arXiv preprint arXiv:2312.04410 (2023) 3
[23] Guo, J., Zhao, J.. Ge, C., Du C.. Ni. Z., Song, S., Shi, H., Huang, G.: Everything to the synthetic: Diffusion-driven test-time adaptation via synthetic-domain alignment. arXiv preprint arXiv:2406.0429. (2024)* 3
[24] Guo, Q. Lin, T.: Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. arXiv preprint arXiv:2312. 101 (2023)*
[25] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 (2022)
2
11

---

[26] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B.. Norouzi, M., Fleet, DJJ., et al.: Imagen video: High definition video generation with diffusion models, arXiv preprint arXiv:2210.0220 (2022) 2, 3
[27] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840-6851 (2020) 3,4
[28] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1250 (2022)
3
[29] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. Advances in Neural Information Processing Systems 35, 8633-8646 (2022) 2,
[30] Hu, Z., Xu, D.: Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet. arXiv preprint arXiv:2307. 14073 (2023) 3
[31] Huang, Z., He, Y., Yu, Zhang, F., Si, C., Jiang. Y., Zhang. Y., Wu, T., Jin, Q. Chanpaisit, N., al.: Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311. Siiv:2311.179982 (2023) 8,
[32] Jonschkowski, R., Stone, A., Barron, J.T., Gordon, A., Konolige, K., Angelova, A.: What matters in unsupervised optical flow. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 557-572. Springer (2020) 2
[33] Kara, O., Kurtkaya, B., Yesiltepe, H., Rehg, J.M., Yanardag. P.: Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. arXiv preprint MrXiv:2312.04524 (2023) 8.9
[34] Karras, T., Aittala, M., Aila, T., Laine, S.: Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems 35, 26565-26577 (2022)* 3
[35] Kawar, B., Elad, M., Ermon, S., Song, J.: Denoising diffusion restoration models. Advances in
Neural Information Processing Systems 35, 23593-23606 (2022)
[36] Ke, J., Wang, Q., Wang, Y., Milanfar, P., Yang, F:: Musiq: Multi-scale image quality transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5148-511 5157 (2021) 9
[37] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv: 131.611312.6144 (2013) 4,5
[38] LAION-AI: (2022) 9
[39] Li, H., Yang. Y., Chang, M., Chen, S., Feng, H., Xu, Z., Li, Q. Chen, Y.: Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing 479, 47-59 (2022) 3
[40] Li, Z., Zhu, Z.L., Han, L.H., Hou, Q., Guo, C.L., Cheng, M.M.: Amt: All-pairs multi-field transforms for efficient frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9801-9810 (2023) 9
[41] Liew, J.H., Yan, H., Zhang, J., Xu, Z., Feng, J.: Magicedit: High-fidelity and temporally coherent video editing. arXiv preprint arXiv:2308. 12223388.144999 14749 (2023)
[42] Lin, Y., Han, H., Gong, C., Xu, Z., Zhang, Y., Li, X.: Consistent One image to highly consistent 3d asset using case-aware diffusion priors. arXiv preprint arXiv:2309.17261 (2023)
[43] Liu. S., Zhang, Y., Li, W., Lin, Z., Jia, J.: Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.dm 1367062303 (2023) 3
[44] Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L.: Repaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CV conference on computer vision and pattern recognition. pp. 11461-11471 (2022)* 3
12

---

[45] Ma, Y., Cun, X., He, Y., Qi. C., Wang, X., Shan, Y., Li, X., Chen, Q.: Magicstick: Controllable video editing via control handle transformations. arXiv preprint arXiv:2312.03047 (2023) 3
[46] Ma, Y., He, Y. Cun, X., Wang, X., Chen, S., Li, X., Chen, Q.: Follow your pose: Pose-guided text-to-video generation using pose-free videos. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 4117-4125 (2024)
[47] Ma, Y., He, Y., Wang. H., Wang, A., Qi. C., Cai, C., Li, X., Li, Z., Shum, H.Y., Liu, W. et al.: Follow-your-cliek: Open-domain regional image animation via short prompts. arXiv preprint arXiv:2403.088 008802688888888 (2024) 3
[48] Ma, Y., Liu, H., Wang, H., Pan, H., He, Y., Yuan, J., Zeng, A., Cai, C.,, Shum, H.Y., Liu, W., al.: Follow-your-emoji: Fine-controllabee and expressive freestyle portrait animation. arXiv preprint arXiv:2406.01900 (2024) 3
[49] Meng, C.. He, Y.. Song, Y., Song, J.. Wu, J., Zhu, J.Y., Ermon, S.: Sdedit: Guided image synthesis and editing with stochastic differential equations, arXiv preprint arXiv:2108.0100 01073 (2021) 3
[50] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen Or, D.: Null-text inversion for editing real images using guided diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6038-6047 (2023) 3
[51] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1 10741 (2021) 3
[52] Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International conference on machine learning. pp. 8162-8171 8171. PMLR (2021) 3
[53] Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11 (2023) 2
[54] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15932-15942 (2023) 2, 3, 8,
[55] Radford, A., Kim, J.W Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.. Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
[56] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06 125 1(2), 3 (2022) 3
[57] Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusson models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684-10000 10695 (2022) 3,
[58) Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 22500-22515 (2023)
[59] Saharia, C., Chan, W., Saxena, S., Li, I.., Whang, J., Denton, E.I., Ghasemipour, K., Gon- tijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems 35, 36479-36466 (2022)* 3
[60] Salimans. T., Ho. J.: Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 (2022)* 3
[61] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T. Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances Neural Information Processing Systems 35, 25278-25209 (2022) 3
13

---

[62] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al.: Make-a-video; Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14 14792 (2022) 2
[63] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics.. In: International conference on machine learning. pp- 2256-223 2265. PMLR (2015) 2,3
[64] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint
arXiv:2010.02502 (2020) 2, 3,4
[65] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole. B.: Score-based generative modeling through stochastic differential equations. arXiv preprint urXiv:2011.1345 (2020) 3,
[66] Tang, I.., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems 36, 1363-1389 (2023) 3, 5,
15
[67] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Blu--and-play diffusion features for text-driven image-to-image translation. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930 (2023) 2
[68] Wang, W., Jiang, Y., Xie, K., Liu, Z.., Chen, H., Cao, Y., Wang, X., Shen, C.: Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.175:1 (2023)
[69] Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y.. Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: auneea-video: One-shot tuning image diffusion models for text-to-video generation. In: Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7623-7633 (2023) 2,
[70] Xiang, J., Huang, R., Zhang, J., Li, G., Han, X., Wei, Y.: Versvideo: Leveraging enhanced tem- poral diffusion models for versatile video generation. In: The Twelfth International Conference on Learning Representations (2023)
[71] Xue, J., Wang, H., Tian, Q., Ma, Y., Wang. A., Zhao, Z., Min, S., Zhao, W., Zhang, K., Shum, H.Y., al.: Follow-your-pe v2: Multiple-condition guided character image animation for stable pose control. Xiviv e-prints pp. arXiv-2406 (2024)* 3
[72] Yang, S., Zhou, Y., Liu, Z., Loy, C.C.: Rerender video: Zero-shot text-guided video-to-video translation. In: SIGGRAPH Asia 2023 Conference Papers, pp. 1-11 (2023) 2, 3
[73] Yang, S., Zhou, Y., Liu, 7.,, Loy, C.C.: Fresco: Spatial-temporal correspondence for zero-shot video translation. arXiv preprint arXiv:2403.12962 (2024) 2, 3, 6, 8,
[74] Yu, S., Sohn, K., Kim, S.. Shin, J.: Video probabilistic diffusion models in projected latent space. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18456-18466 (2023) 3
[75] Zhang. L., Rao, A., Agrawala, M.: Adding conditional control text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3836-3847 (2023) 3
14

---

Appendix
A Detailed Experimental Settings
In the experiment, the size of all source videos is 512 x 512. We adopt Stable Diffusion (SD) 2.1 from the official Huggingface repository for our method. To extract the diffusion feature, following [66], the noise of the timestep & = 261 is added to each frame of the source video. The noisy frames of video are fed into the U-net, the feature extracted from the intermediate layer of the 2D Unet decoder. The height and weight of the diffusion feature is 64. Following previous works, at the first 40 timesteps, the diffusion features are saved during DDIM inversion and are further injected during denoising. For Spatial-tempont attention, we use the xFormers to reduce memory consumption, while it is not used in correspondence guided attention.
B Ablation Study on the Window Size
To illustrate the influence of window size I, we conduct the experiment on a video with 20 frames on a single A100 GPU. During the correspondence calculation process, we calculate the theoretical computational complexity, which is the total number of multiplications and additions required. We also record actual GPU memory consumed under different window sizes, the result is shown in Appendix B. With our sliding window strategy, the computational complexity and the GPU memory in the correspondence calculation process are significantly reduced. The visualization result is shown in Fig. 9. If the window size is too small, the motion in the video cannot be tracked, causing unsatisfying results. We choose 9 for the experiments other sections, which can achieve a balance between the memory consumed and the quality of the edited video.
Window Size
9
15
w/o
Computational Complexity (x 100) GPU Memory (GB)
0.448
4.03
11.2
241
11
14
18
32
Table Ablation Study on the window size w/o means that the medding-window strategy is not applied. The sliding window strategy can significantly reduce the use of computational complexity and GPU memory.
Source Video
Van Gogh style
3
=
w/O
Figure 9: Ablation Study on the window size 7.
15

---

C Visualisation of the Correspondence
We visualize the correspondence calculated by our sliding-winden window-based method illustrate its effectiveness (Fig. 10). To be specific, we calculate the correspondence based on the 64 x 64 diffusion feature, which extracted at the final layer of the U-net decoder. The result illustrates that our method can effectively identify the corresponding tokens.
"A raccoon" Corr Source
1 frume
6 frume
11 frume 16 Trame 21 frame 26th frume 31 frame 36P Frame 41th frume 46 frume
51 frume 55 frame
Figure 10:
Visualization of the correspondence in long videos. Given long video, we first obtain the correspondence information (K 3) through the sliding-windee strategy. Then, considering point in the first frame (the red point in the first image of the second row), we visualize the correspondence (respectively marked yellow, green, and blue) each frame.
D Accuracy of Correspondance
The correspondence acquired through the diffusion feature is accurate and robust. As there is existing video dataset with the annotated keypoints on each frame, to further evaluate its accuracy quantitatively, collect videos with 30 frames and 5 videos with 60 frames and manually label some keypoints on each frame. Then we report the percentage of correct keypoints (PCK).
Specifically, for each video, given the first frame with the keypoints, corresponding keypoints on other frames through the diffusion feature. Then we evaluate the distance between the predicted points and the ground truth. The predicted point is considered to be correct if lies in small neighborhood of the ground truth. Finally, the total number of correctly predicted points divided by the total number of predicted points the value of PCK. The result in Appendix D illustrates that the diffusion feature can accurately find the correct position in most cases for video editing.
obtain the predicted
Method
PCK
Optical- flow Correspondence
0.87
Diffusion feature Correspondence
0.92
Table Accuracy of Correspondance.
E
Effectiveness of correspondence guided attention during inversion
The quality of noise obtained by inversion can significantly affect the final quality of editing. The Correspondence-Guiin Guided Attention (CGA) during inversion can increase the quality and temporal consistency of the obtained noise, which can further help enhance the quality and consistency of edited videos. The ablation of it is shown in Fig. 11
F
Broader Impacts
Our work enables high-quality video editing, which is in high demand across various social media platforms, especially short video websites like Tok Using
method, people can easily create
16

---

Source
"A traditional Chinese House"
"In winter"
Without Con
With (Ours)
Without Car
With Carr (Ours)
Figure 11: Ablation Study about correspondence in inversion. Here Without Corr means not
applying the correspondence-guidee attention during inversion, which suffers blurring and flickering. With Corr means the correspondence-guss guiiedd attention is applied in both inversion and denoising stages, illustrating satisfying performance.
high-quality and creative videos, significantly reducing production costs. However, there a potential for misuse, such replacing the characters in videos with celebrities, which may infringe upon the celebrities' image rights. Therefore, it also necessary to improve relevant laws and regulations to ensure the legal use of our method.
G Limitations
Despite achieving outstanding results, our methods still encounter several limitations. First, although the correspondence calculation process is efficient through the proposed sliding window strategy, the implementation of correspondence -uidedd attention is still not efficient enough, leading to the extra usage of GPU memory and time (Table 3). This problem expected to be alleviated largely through the use of xFormers. We will work on it in the future.
Second, further exploration is required to optimize the application of the obtained correspondence information. In this study, we utilize the correspondence information to sample tokens during the inversion and denoising processes and do the self-attention. However, believe that there may be more effective alternatives to self-attention that could further unleash the potential of the correspondence information.
H More Qualitative Results
We provide more qualitative results of our method to illustrate its effectiveness, which is shown in Fig. 13 and Fig. 12.
17

---

Source Video
A red car in snowy winter
A Wooden car
FAR
A car driving in dry desert
A car, oil painting style
A car driving in sunset
Style of Comic Book
Style of Van Gogh
Figure 12: Qualitative results of our methods. Our method can effectively handle various kinds of prompts, generating high-quality videos. Results are best viewed in zoomed-in.
18

---

Source Video
An African woman in grey clothes
A woman dressed like Black widow
A marble sculpture
Cartoon style
Figure 13: Qualitative results of our methods. Our method can effectively handle various kinds of prompts, generating sigh-qualty videos. Results are best viewed in zoomed-in.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our works.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We prove that the self-attention can enhance the similarity among tokens, which included in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the experimental details and submit the code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We submit the code the supplementary results while not creating a public GitHub repo.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce hepollcc/giieee Codeeubbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment details are specified.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The qualitative results are important in the field generation. The error bar is relatively not necessary.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have described the resources required to perform our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ww./eerrppp..e/ cc cc/ppblic////cccccccccocccaclltt www.:iiiiethiiiuuiiillleoeleeeee ines?
Answer: [Yes]
Justification: Our work conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited and the license and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "orLuxuvkmV", "reviewer": "Reviewer_sLJK", "summary": "The paper proposes using the correspondence features already exist in diffusion models to find matched tokens between different frames in a video for consistent video editing. The motivation is that video editing models using optical flow to find matched features can exhibit one-to-many matching issue which limits the quality of video editing consistency when there are large motions. To overcome expensive compute, the paper proposes using \"sliding window\" (w.r.t frames) by only finding correspondences in subsequent frames. Reasonable qualitative and quantitative results are shown in the experiments section.", "questions": "See weakness", "limitations": "No", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method intuitively make sense\n- The flow of the paper is relatively easy to follow\n- The results shown are reasonable", "weaknesses": "- Optical flow is also essentially just finding matched patches/features between different frames. The proposed way of finding one-to-many correspondences between frames can also be applied uusing optical flow (with some small modifications of existing optical flow algorithms). I do not see any evidence from the paper that using the features in a diffusion model is better than using the features extracted by an optical flow algorithm (explicitly or inexplicitly depending on different optical flow methods)\n\n- the above should be an important ablation study that is currently missing from the paper\n\n- the sliding window algorithm, though simple and achieving compute reduction, can also lead to errors accumulating across frames when editing long videos\n\n- the novelty is a bit limited. My argument is the following:\n    1. Features inexplicitly extracted in Stable Diffusion can be used for feature correspondence is well-known to the community, e.g. in Tang et al. NeurIPS 2023\n    2. Finding correspondences for video editing is widely studied and shown to be effective, including but not limited to the several references this paper already cited\n    3. The main contribution then seems to be proposing using SD's features to find correspondences instead of using optical flow. However, the one-to-many problem and many-to-one problem is essentially a thresholding problem when finding correspondences and the receptive field size used to find matched patches, and I cannot see why the features extracted by SD is superior than the features extracted by a state-of-the-art optical flow model. This ablation study is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer sLJK,\n\nThanks for your time and thoughtful review. We appreciate your recognition of the satisfying experimental results and clear writing of the paper. We provide our feedback as follows.\n# Optical flow model for one-to-many correspondences\n> Optical flow is $\\cdots$ by an optical flow algorithm.\n\nThanks for your valuable suggestions. Modifying the optical flow model [1,2] from one-to-one to one-to-many correspondence may seem simple and intuitive. Unfortunately, we would like to point out that it is still nontrivial in practice due to the inherent characteristics of optical flow models. \n- For the $i$th frame $I_i$ (with the shape of [H,W]) in a video, the optical flow model ***directly predicts*** an optical field $F$ with the shape of [H,W,2], representing the offset of each pixel along $x$ and $y$ axis in $I_{i+1}$. The prediction of $F$ is a ***regression problem*** during training, where L1 loss is applied between $F$ and the ground truth. In other words, for each pixel, optical flow models ***directly predict*** the position of the ***single*** corresponding pixel in the next frame, rather than firstly calculate confidence for each pixel in the next frame and then select the pixel with the highest confidence. \n- Based on the analysis above, the off-the-shelf optical flow model cannot obtain the one-to-many correspondence. Furthermore, there is also ***no existing dataset with the annotations of one-to-many correspondence*** to support the training of a model with modified structures from scratch . The construction of dataset and the design of new model structure are still unexplored by existing works, which is out of the scope of our paper. \n\nAs a result, current optical flow models fail to obtain one-to-many correspondence. Instead, we provide the ablation study between COVE with $K=1$ (i.e., one-to-one correspondence) and flow-based baseline (Figure 13 and Table 5). The results illustrate that even with one-to-one correspondence, COVE still achieves comparable or even better result compared with the flow-based baseline. Furthermore, one of the main advantages of COVE is that it can effectively obtain one-to-many correspondence. With the increase of $K$, the quality of edited videos is better, achieving SOTA performance. We will update the details in the paper.\n\n**Table 5.** Quantitative comparison between OFC(optical-flow correspondence) and DFC(diffusion feature correspondence). The detailed experiment settings follow original paper.\n|Method|SC|MS|IQ|AQ|\n|-|-|-|-|-|\n|OFC|0.9617|0.9622|0.7155|0.6544|\n|**DFC ($K=1$)**|0.9637|0.9817|0.7148|0.6979|\n|**DFC ($K=3$)**|**0.9731**|**0.9892**|**0.7441**|**0.7122**|\n# Accuracy of the sliding window\n> The sliding window $\\cdots$ long videos.\n\nThanks for your advice. In our method, the position of the sliding window is dynamically adjusted rather than fixed in each frame (Figure 4), which can ensure the accuracy of correspondence in long video. We also add the experiment about the editing quality of 10 short videos (20 frames) and their longer counterpart (60 frames). The results (Table 6 and Figure 14) illustrate that COVE is also effective for longer videos. We also provide the quantitative results about accuracy of correspondence in Table 7 (in feedback for Reviewer LETZ).\n\n**Table 6.** Quantitative comparison between the editing quality of 20 frames and 60 frames.\n|Video length|SC|MS|IQ|AQ|\n|-|-|-|-|-|\n|20|0.9689|0.9887|0.7447|0.7135|\n|60|0.9685|0.9882|0.7441|0.7140|\n\nWhat's more, the window size is a hyper-parameter. We may enlarge the window size if there are extremely sharp and fast motions (although such videos are very rare in real world). Even with a larger window size, our method still greatly reduces complexity.\n# Novelty\nWe believe COVE elegantly addresses several challenging limitations in previous SOTA methods, potentially offering valuable insights for further research.\n> 1.Features inexplicitly $\\cdots$ to the community.\n\nExisting work has primarily focused on correspondence between two images. However, it remains to be explored whether this correspondence can maintain consistent stability and accuracy over multiple frames and longer time scales in videos. Furthermore, whether the correspondence can aid in video editing tasks also lacks exploration. COVE elegantly proposes using correspondence to guide attention during inversion and denoising procedures. Achieving outstanding results (Figure 15 in the uploaded PDF), our work illustrates that ***using diffusion features to enhance quality and consistency*** is an insightful topic in the field of video editing.\n\n> 2.Finding correspondences $\\cdots$ be effective.\n\nAlthough some previous methods also explore the correspondence for video editing, they heavily depend on the availability of a pretrained optical flow model with high accuracy, which is often not feasible in practice. Until now, acquiring correspondence information without optical flow models is still a challenging problem for video editing. Addressing this, COVE successfully illustrates that the ***optical flow model is not indispensable for finding correspondence in video editing***, and achieves ***even better*** results compared to flow-based methods.\n\n> 3.The main contribution $\\cdots$ is missing.\n\nAs stated above, in the optical flow model, the position of corresponding token is directly predicted through regression rather than firstly calculate the similarity in the feature and then select the token with highest similarity. As a result, the one-to-many and one-to-one problem ***is not a simple thresholding problem in optical flow models***. COVE elegantly addresses this limitation, obtaining one-to-many correspondence simply and effectively. So, we believe that our work is an insightful and interesting contribution to the community.\n# References\n[1] RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, ECCV 2020 Best Paper\n\n[2] GMFlow: Learning optical flow via global matching, CVPR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W3BukVGX95", "reviewer": "Reviewer_LETZ", "summary": "This paper focused on improving the temporal consistency of video editing. They propose to leverage the inherent diffusion feature correspondence with a sliding-window based strategy. With this design, the tokens in noisy latents can be sampled based on the “one-to-many” correspondence. The experiments demonstrate superior performance than other methods.", "questions": "o\tThe reviewer appreciates the qualitative examples of correspondences of diffusion features in appendix. Is there any quantitative metric to evaluate the correspondences of diffusion features?\n\no\tHow to define the temporal length for token merging? Does it remain the same for all experiment videos?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper proposed a new diffusion feature correspondences guided video editing method. In addition, they introduce the token merging and sliding window for higher efficiency.\n\n+ The method demonstrates superior performance than previous methods with extensive ablations to evaluate the effectiveness. \n\n+ The paper is well-written with clear motivations.", "weaknesses": "- The proposed method highly relies on the correspondences of diffusion features. However, such correspondences may be difficult to obtain for videos with large content motions. The reviewer suggests a more detailed discussion about how to ensure the accuracy of the correspondences, as well as what are the potential limitations of inaccurate correspondences. In addition, it would be better to quantitatively evaluate the correctness of the correspondences of diffusion features.\n\n- The paper mainly discussed and compared their method with optical flow-guided video editing methods. However, such correspondence-based idea is also related to deformation field based methods such as CoDeF [1] or neural atlas based methods [2], where they can ensure the accuracy of pixel / point level accuracy by evaluating the video reconstruction accuracy. The authors are suggested to compare or discuss their method with such approaches.\n\n- The related works seem to be a bit short and less extensive. There have been many video editing works that target at improving temporal / subject consistency, efficiency, long video editing, etc. The reviewer suggests to include a more extensive related works and discuss the differences of the proposed work with them. \n\n[1] Ouyang, Hao, et al. \"Codef: Content deformation fields for temporally consistent video processing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Chai, Wenhao, et al. \"Stablevideo: Text-driven consistency-aware diffusion video editing.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LETZ,\n\nThank you for your comprehensive and detailed review of our paper and the recognition of our work's clarity and effectiveness. We provide our feedback as follows.\n# Accuracy of correspondences\n> The proposed method highly relies on the correspondences of diffusion features $\\cdots$. In addition, it would be better to quantitatively evaluate the correctness of the correspondences of diffusion features. \n\nAs illustrated in Figure 10 in appendix and Figure 14 in the uploaded PDF, the correspondence acquired through the diffusion feature is accurate and robust. As there is no existing video dataset with the annotated keypoints on each frame, to further evaluate its accuracy quantitatively, we collect 5 videos with 30 frames and 5 videos with 60 frames and manually label some keypoints on each frame. Then we report the percentage of correct keypoints (PCK) following the prior work [1,2] (Table 7). \n\nSpecifically, for each video, given the first frame with the keypoints, we obtain the predicted corresponding keypoints on other frames through the diffusion feature. Then we evaluate the distance between the predicted points and the ground truth. The predicted point is considered to be correct if it lies in a small neighborhood of the ground truth. Finally, the total number of correctly predicted points divided by the total number of predicted points is the value of PCK. The result in Table 7 illustrates that the diffusion feature can accurately find the correct position in most cases for video editing. \n\n**Table 7.** Quantitative comparison of the accuracy between diffusion feature correspondence (DFC) and optical flow correspondence (OFC). \n|Method|PCK|\n|-|-|\n|OFC|0.87|\n|**DFC**|**0.92**|\n\nThe potential limitation of inaccurate correspondences is that suboptimal results could be obtained. For instance, as discussed in Section 4.4, using a too-small window size leads to inaccurate correspondences, resulting in poor editing outcomes. By adjusting the window size, this problem can be effectively solved.\n\n# Comparasion with baselines\n> The paper mainly discussed and compared their method with optical flow-guided video editing methods $\\cdots$ . The authors are suggested to compare or discuss their method with such approaches.\n\nThanks for your advice. The deformation field-based methods [3] factorize the video into a 2D content canonical field and a 3D temporal deformation field. For each video, it requires complicated training of implicit deformable models to obtain a 2D and a 3D hash table. What's more, it still needs the optical flow model for preprocessing the video sequences. \n\nNeural atlas-based methods [4] leverage the layered representations to propagate the appearance information among frames. However, two diffusion models are employed to respectively process the background and foreground, which increases resource consumption. Furthermore, they also need extra training. \n\n**In contrast**, COVE is a training-free method that effectively obtains highly accurate correspondence simply by calculating the similarity between features. We also compare their performance (Table 8 and Figure 15 in the uploaded PDF). We are willing to add the results to the final revision of the paper.\n\n**Table 8.** Quantitative comparison among CoDeF [3], StableVideo [4] and COVE. We use them to respectively edit 10 20-frame videos which are widely used in the field of video editing and report the quantitative results.\n|Method|SC|MS|AQ|IQ|\n|-|-|-|-|-|\n|CoDeF [3]|0.9369|0.9619|0.6847|0.7134|\n|StableVideo [4]|0.9215|0.9771|0.6781|0.7273|\n|**Ours**|**0.9689**|**0.9887**|**0.7135**|**0.7447**|\n\n# Add more related works\n> The related works seem to be a bit short and less extensive $\\cdots$. The reviewer suggests to include more extensive related works and discuss the differences of the proposed work with them. \n\nThanks for your advice, we will discuss more related works in our paper, including but not limited to the CoDeF [3] and StableVideo [4].\n\n# Temporal length for token merging\n> How to define the temporal length for token merging? Does it remain the same for all experiment videos?\n\nIn our experiment, the temporal length for token merging is simply set to the length of the video.\n\n# References\n[1]. Cats: Cost aggregation transformers for visual correspondence, NeurIPS 2021\n\n[2]. Transformatcher: Match-to-match attention for semantic correspondence, CVPR 2022\n\n[3]. Codef: Content deformation fields for temporally consistent video processing, CVPR 2024.\n\n[4]. Stablevideo: Text-driven consistency-aware diffusion video editing, ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qMD78VLkNf", "reviewer": "Reviewer_Ht4d", "summary": "In this paper, the authors tackle the problem of video editing using Text-to-Image diffusion models. To achieve this, the authors make use of strong diffusion model’s feature correspondence abilities. The authors propose a sliding-window based strategy to track features of source video based on correspondences. Here, the goal is to overcome the computational complexity involved with finding similarity across all patches of the video. This is achieved by computing similarity scores within a sliding window. After this, a set of corresponding tokens are collected and merged using an existing work (Token Merging paper, ICLR 2023). Experimental analysis shows that the proposed method achieves state-of-the-art performance across many tasks.", "questions": "- How does applying \\tilde{Corr} help during inversion?", "limitations": "The paper addresses the limitations of the work in Appendix E.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Model does not require any training or even inference time optimization. The method computes correspondences in an efficient manner and track it effectively. \n\nS2. The ideas proposed in the paper are simple and well presented overall. \n \nS3. Results look impressive - the method is able to achieve state-of-the-art in comparison to the works in the area of video editing with T2I models.", "weaknesses": "W1. The paper clearly states that the base model is Stable diffusion on L208 and the model does not require any training including inflation (which I think is a major strength).  Stable diffusion model has only self-attention and cross-attention layers. It is not clear how “temporal layers” come into picture on L200. Further, it is not clear what is the 3D-Unet in context of Fig. 3.\n\nW2. The proposed method tends to alter the regions of the image that do not correspond to the text prompt. In Fig. 1, the results shows that the proposed alters the background significantly. \n\nW3. Results show that the proposed method is unable to alter the shape of the object, since the correspondence merging limits the shape changing ability drastically. \n\nW4. The core novelty of exploiting the sliding window to use a lower field of comparison (as. In Eq. 10) is somewhat straightforward and limited given the obvious computational constraints. However, in conjunction with other ideas such as Token Merging [ref 1] and correspondences [ref 2], the work is novel. \n\nW5. There have been works in the video editing space which exploit correspondences for video editing [ref 3,4]. It will be helpful to differentiate the proposed method with them. However, I would like to acknowledge that these works could have been parallel to this submission.\n\nReferences\n\n[ref 1] Token Merging: Your ViT But Faster, ICLR 2023\n\n[ref 2] Emergent Correspondence from Image Diffusion, NeurIPS 2023\n\n[ref 3] Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR \n\n[ref 4] GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models, CVPRw", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Ht4d,\n\nThanks for your comprehensive review and insightful comments on our paper. We appreciate that you recognize the advantages of training-free and impressive results of our method. The response to your concerns is shown below. \n# Temporal layers and 3D Unet\n> The paper clearly states that the base model is Stable diffusion $\\cdots$. Further, it is not clear what is the 3D-Unet in context of Fig. 3.\n\nThe temporal layer is widely applied in video diffusion models, which mainly consist of spatio-temporal attention [1]. In spatio-temporal attention, considering a query Q in one frame, the key K and value V are from both the current frame and other frames. This operation aims to capture spatio-temporal consistency of the video. \n\nTo convert a 2D Unet into a 3D Unet, current methods [1] usually inflate the 2D convolution layers to pseudo-3D convolution layers, with 3x3 kernels being replaced by 1x3x3 kernels. And the self-attention in 2D Unet is replaced by the aforementioned spatio-temporal attention. We will add a detailed introduction to 3D Unet in the appendix of our paper.\n\n# Editing unrelated regions\n> The proposed method tends to alter $\\cdots$ alters the background significantly.\n\n\nWe would like to point out that maintaining the background unchanged is not the primary focus of our work. In fact, most current SOTA video editing methods (such as [1,2,3,4,5,6]) which ***take texts as the only condition*** also cannot maintain the background. To solve this problem, additional conditions such as a mask for regional editing need to be added, which we believe is an interesting topic for future work. \n\nOn the other hand, our paper mainly focuses on maintaining temporal consistency and enhancing the quality of generated videos. The experimental results in the paper illustrate that we have effectively achieved this goal. In addition, we also notice that in some cases COVE can maintain the background unchanged (such as Figure 16 in uploaded PDF). \n\n# Shape editing\n> Results show that the proposed method is unable to alter the shape of the object, since the correspondence merging limits the shape changing ability drastically.\n\nCOVE also illustrates the satisfying capability of making small modifications to the shape of the object such as changing a jeep into a sport car or a bus (Figure 15 in the uploaded PDF). We believe this is due to the correspondence can accurately reflect the motion information in the source video. This motion information remains unchanged between the source video and edited video even though the users want to make modifications to the object's shape. As a result, COVE not only significantly alleviates the problem of temporal inconsistency but also shows satisfying performance for altering the shape of objects.\n\n# Novelty\n> The core novelty of exploiting the sliding window to use $\\cdots$ the work is novel. \n\nThanks for acknowledging the novelty of our work! As stated in the paper, our proposed sliding-window method is based on the continuity of videos, which is intuitive and effective. Significantly reducing the computational complexity while achieving outstanding performance, we believe that our work is insightful for the community of video editing.\n\n# Comparasion with baselines\n> There have been works in the video $\\cdots$ could have been parallel to this submission.\n\nThanks for your advice. [6] proposes using a pretrained T2V model for motion transferring. [7] proposes target and shape-aware InvEdit masks for shape editing. However, they need extra training [7] or optimization [6]. **In contrast**, COVE is a training-free framework that can leverage popular pre-trained T2I model (such as Stable Diffusion) to achieve impressive performance. We compare our method with [6] (Figure 15 and Table 9). *The source code of [7] is not released*, we are willing to compare our method with it after the source code is released.\n\n**Table 9.** Quantitative comparison between DMT [6] and our method. We use them to respectively edit 10 20-frame videos which are widely used in the field of video editing and report the quantitative results.\n|Method|SC|MS|AQ|IQ|\n|-|-|-|-|-|\n|DMT [6]|0.9573 |0.9874 | 0.7003|0.7396 |\n|**Ours**|**0.9689**|**0.9887**|**0.7135**|**0.7447**|\n\n# Effectiveness of correspondence guided attention during inversion\n> How does applying $\\tilde{Corr}$ help during inversion?\n\nThe quality of noise obtained by inversion can significantly affect the final quality of editing [8, 9]. The **C**orrespondence-**G**uided **A**ttention (CGA) during inversion can increase the quality and temporal consistency of the obtained noise ($z^T$ in Figure 3), which can further help to enhance the quality and consistency of edited videos. We add the ablation study of it (Figure 16 and Table 10), which will also be added to our main paper.\n\n**Table 10.** Ablation Study of **C**orrespondence-**G**uided **A**ttention (i.e., applying $\\tilde{Corr}$) during inversion.\n||SC|MS|AQ|IQ|\n|-|-|-|-|-|\n|No CGA during inversion|0.9413|0.9740|0.7098|0.7422|\n|**CGA in both inversion and denoising**|**0.9689**|**0.9887**|**0.7135**|**0.7447**|\n\n# References\n\n[1] Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation, ICCV2023\n\n[2] FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing, ICLR 2024\n\n[3] RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models, CVPR 2024\n\n[4] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation, CVPR 2024\n\n[5] Codef: Content deformation fields for temporally consistent video processing, CVPR 2024\n\n[6] Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR 2024\n\n[7] GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models, CVPRw\n\n[8] Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models, CVPR 2024\n\n[9] Pix2video: Video editing using image diffusion, ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p4k2gw0kAz", "reviewer": "Reviewer_nziD", "summary": "The paper introduces COrrespondence-guided Video Editing (COVE), a method to improve video editing with pretrained text-to-image (T2I) diffusion models. It addresses the challenge of maintaining temporal consistency by using diffusion feature correspondence. COVE identifies and samples highly corresponding tokens across frames, applying self-attention to them during editing. It also reduces GPU memory usage with a temporal-dimensional token merging strategy. COVE integrates seamlessly into existing T2I models without extra training and shows superior performance in various scenarios.", "questions": "How do you make sure that $k=3$ is suitable for all videos? As suggested in Fig. 2, the subject may suffer from severe morph across video frames when there is a significant motion change.", "limitations": "See weakness above.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is reasonable and easy to follow.\n2. The presentation is clear and easy to understand.\n3. The Sliding-window Strategy seems useful in video editing methods.", "weaknesses": "1. Experimental performance improvement seems limited. Most of the editing results present a similar type involving style/color change, which is easy to implement in the previous methods from my experience. According to Fig. 6, the performance compared to earlier methods does not show high superiority, which is not convincing to support \"...COVE achieves the start-of-the-art performance...\". \n\n2. I noticed that [1] also uses DIFT features to guide video editing, especially object replacement. Could you please compare COVE with it? For example, replace the \"jeep\" with a \"sports car\" or \"bus\" in Fig. 11. I would like to see the performance when there is a large motion/shape change.\n\n[1] VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nziD,\n\nThank you for your time and thoughtful feedback on COVE. We are pleased that you recognize the effectiveness of the sliding window strategy. We provide our feedback as follows.\n# Experimental performance\n> Experimental performance improvement seems limited. Most of the editing results present a similar type involving style/color change, which is easy to implement in the previous methods from my experience. According to Fig. 6, the performance compared to earlier methods does not show high superiority, which is not convincing to support \"...COVE achieves the start-of-the-art performance...\".\n\nThe baseline methods mentioned in our paper are among the ***most representative and effective*** approaches in recent years, having received widespread recognition and attention. In the experiment, we present ***both quantitative and qualitative results*** to illustrate COVE's superior performance. As shown in Table 1, COVE outperforms these baseline methods on several widely used quantitative metrics. The result of user study also illustrates that edited video quality aligns with human subjective perception. For qualitative comparison, COVE successfully alleviates the problems of baselines such as blurring, temporal inconsistency, inconsistency with the prompt, etc (Figure 6). The ablation study (Table 2 and Figure 7) also illustrates that COVE can significantly enhance the temporal consistency of edited videos. We also ***provide more qualitative results*** (Figure 15 in the uploaded PDF), illustrating the ability of COVE for shape editing. \n\n# Comparasion with Baselines\n> I noticed that [1] also uses DIFT features to guide video editing, especially object replacement. Could you please compare COVE with it? For example, replace the \"jeep\" with a \"sports car\" or \"bus\" in Fig. 11. I would like to see the performance when there is a large motion/shape change.\n\nVideoSwap proposes to leverage the semantic point correspondence for aligning motion trajectories for high-quality video editing. However, it requires users to manually provide keypoints as the additional condition for each video, which is a relatively troublesome burden for users. Furthermore, the method needs extra training, increasing the cost of video editing. \n\n**In contrast**, COVE is a training-free framework that only requires users to provide the text prompt. Unfortunately, *the code of VideoSwap is not publicly available*, we have filled in the form provided by the authors to inquire about the source code. We are willing to compare our method with it after obtaining its source code. \n\nWe also provide more experiment results as you suggested, such as changing the jeep into sports car and bus. The results are shown in Figure 15 in the uploaded PDF, illustrating the satisfying performance of our method for shape editing. We are willing to add the results to the final revision of our paper.\n\n# The value of $K$\n\n> How do you make sure that K=3 is suitable for all videos? As suggested in Fig. 2, the subject may suffer from severe morph across video frames when there is a significant motion change.\n\nAs illustrated in Table 2, for the videos in our experiments, $K=3$ and $K=5$ achieve similar performance. As a result, we choose $K=3$ for a better trade-off between quality and resource consumed. What's more, $K$ is a hyperparameter in our method, which can be adjusted for each video to achieve high-quality editing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Overall reply for all reviewers

We thank all reviewers for their constructive comments. We have responded to each of the concerns below. In the following response, Figure 1-12 and Table 1-4 are in our main paper, ***Figure 13-16 are in the uploaded PDF of the rebuttal, and Table 5-10 are in our response***. For quantitative comparison of the video quality, we still report the four popular metrics used in the paper, i.e., Subject Consistency (SC), Motion Smoothness (MS), Aesthetic Quality (AQ), and Imaging Quality (IQ).

Meta-review: The paper received ratings of $(7, 6, 5, 4)$ after the rebuttal and discussions. **Reviewer Ht4d** and **Reviewer LETZ**, who gave ratings of $7$ and $6$, acknowledged that the authors' responses addressed their concerns and thus expressed their support. The remaining two reviewers, while having borderline opinions, did not raise further issues after the rebuttal. Specifically, the rebuttal provided clarifications with additional figures and tables to address **Reviewer sLJK**'s questions about the differences with optical flow models, the robustness of using sliding windows, and the novelty of the work.

In light of these considerations, the area chair recommends accepting this paper. The authors are encouraged to incorporate the discussions and additional figures and tables in the final version. Furthermore, it is imperative that the authors address the ethical concerns raised by the Ethics Reviewers regarding potential misuse and the user study involving human subjects.
Decision: Accept (poster)