COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing

Authors: jiangshan  wang
yue  ma
jiayi  guo
yicheng  xiao
gao  huang
xiu  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 04:40:00

Abstract: Video editing is an emerging task, in which most current methods adopt the pre-trained text-to-image (T2I) diffusion model to edit the source video in a zero-shot manner. Despite extensive efforts, maintaining the temporal consistency of edited videos remains challenging due to the lack of temporal constraints in the regular T2I diffusion model. To address this issue, we propose COrrespondence-guided Video Editing (COVE), leveraging the inherent diffusion feature correspondence to achieve high-quality and consistent video editing. Specifically, we propose an efficient sliding-window-based strategy to calculate the similarity among tokens in the diffusion features of source videos, identifying the tokens with high correspondence across frames. During the inversion and denoising process, we sample the tokens in noisy latent based on the correspondence and then perform self-attention within them. To save the usage of GPU memory and accelerate the editing process, we further introduce the temporal-dimensional token merging strategy, which can effectively reduce the redundancy. COVE can be seamlessly integrated into the pre-trained T2I diffusion model without the need for extra training or optimization. Extensive experiment results demonstrate that COVE achieves the start-of-the-art performance in various video editing scenarios, outperforming existing methods both quantitatively and qualitatively. The source code will be released.

Main paper: 
 introduction: 
 Diffusion models [27, 63, 65] have shown exceptional performance in image generation [57] , thereby inspiring their application in the field of image editing [6, 25, 7, 53, 67, 24] . These approaches typically leverage a pre-trained Text-to-Image (T2I) stable diffusion model [57] , using DDIM [64] inversion to transform source images into noise, which is then progressively denoised under the guidance of a prompt to generate the edited image.Despite satisfactory performance in image editing, achieving high-quality video editing remains challenging. Specifically, unlike the well-established open-source T2I stable diffusion models [57] , comparable T2V diffusion models are not as mature due to the difficulty of modeling complicated temporal motions, and training a T2V model from scratch demands substantial computational resources [26, 29, 62] . Consequently, there is a growing focus on adapting the pre-trained T2I diffusion for video editing [16, 33, 11, 72, 73, 54] . In this case, maintaining temporal consistency in edited videos is one of the biggest challenges, which requires the generated frames to be stylistically coherent and exhibit smooth temporal transitions, rather than appearing as a series of independent images. Numerous methods have been working on this topic while still facing various limitations, such as the inability to ensure fine-grained temporal consistency (leading to flickering [33, 54] or blurring [16] in generated videos), requiring additional components [30, 72, 11, 73] or needing extra training or optimization [73, 69, 41] , etc.Frame 𝑖 Frame 𝑖 + 1Frame 𝑖 Frame 𝑖 + 1Figure 2 : Comparison between COVE (our method) and previous methods [11, 73] .In this work, our goal is to achieve highly consistent video editing by leveraging the intraframe correspondence relationship among tokens, which is intuitively closely related to the temporal consistency of videos: If corresponding tokens across frames exhibit high similarity, the resulting video will thus demonstrate high temporal consistency. Taking a video of a man as an example, if the token representing his nose has high similarity across frames, his nose will be unlikely to deform or flicker throughout the video. However, how to obtain accurate correspondence information among tokens is still largely under-explored in existing works, although the intrinsic characteristic of the video editing task (i.e., the source video and edited video are expected to share similar motion and semantic layout) determines that it naturally exists in the source video. Some previous methods [11, 73] leverage a pre-trained optical-flow model to obtain the flowing trajectory of each token across frames, which can be seen as a kind of coarse correspondence information. Despite the self-attention among tokens in the same trajectory can enhance the temporal consistency of the edited video, it still encounters two primary limitations: Firstly, these methods heavily rely on a highly accurate pre-trained optical-flow model to obtain the correspondence relationship of tokens, which is not available in many scenarios [32] . Secondly, supposing we have access to an extremely accurate optical-flow model, it is still only able to obtain the coarse one-to-one correspondence among tokens in different frames (Figure 2a ), which would lead to the loss of information because one token is highly likely to correspond to multiple tokens in other frames in most cases (Figure 2b ).Addressing these problems, we notice that the inherent diffusion features naturally contain precise correspondence information. For instance, it is easy to find the corresponding points between two images by extracting their diffusion features and calculating the cosine similarity between tokens [66] . However, until now none of the existing works have successfully utilized this characteristic in more complicated and challenging tasks such as video editing. In this paper, we propose COVE, which is the first work unleashing the potential of inherent diffusion feature correspondence to significantly enhance the quality and temporal consistency in video editing. Given a source video, we first extract the diffusion feature of each frame. Then for each token in the diffusion feature, we obtain its corresponding tokens in other frames based on their similarity. Within this process, we propose a sliding-window-based approach to ensure computational efficiency. In our sliding-window-based method, for each token, it is only required to calculate the similarity between it and the tokens in the next frame located within a small window, identifying the tokens with the top K (K > 1) highest similarity. After the correspondence calculation process, for each token, the coordinates of its K corresponding tokens in each other frame can be obtained. During the inversion and denoising process, we sample the tokens in noisy latents based on the obtained coordinates. To reduce the redundancy and accelerate the editing process, token merging is applied in the temporal dimension, which is followed by self-attention. Diffusion Models [27, 63, 65] have recently showcased impressive results in image generation, which generates the image through gradual denoising from the standard Gaussian noise [12, 13, 52, 21, 57, 64, 22] . A large number of efforts on diffusion models [28, 34, 60] has enabled it to be applied to numerous scenarios [3, 15, 35, 39, 44, 49, 50, 14, 58, 9, 23, 47, 42, 18] . With the aid of large-scale pretraining [55, 61] , text-to-image diffusion models exhibit remarkable progress in generating diverse and high-quality images [51, 71, 56, 57, 59, 19, 48, 18] . ControlNet [75] enables users to provide structure or layout information for precise generation. Naturally, diffusion models have found application in video synthesis, often by integrating temporal layers into image-based DMs [4, 26, 29, 70, 10] . Despite successes in unconditional video generation [29, 74, 45 ], text-to-video diffusion models lag behind their image counterparts. 
text-to-video editing.: 
 There are increasing works adopting the pre-trained text-to-image diffusion model to the video editing task [43, 68, 69, 46, 20] , where keeping the temporal consistency in the generated video is the most challenging. Recently, a large number of works focusing on zero-shot video editing has been proposed. FateZero [54] proposes to use attention blending to achieve high-quality edited videos while struggling to edit long videos. TokenFlow [16] reduces the effects of flickering through the linear combinations between diffusion features, while the smoothing strategy can cause blurring in the generated video. RAVE [33] proposes the randomized noise shuffling method, suffering the problem of fine details flickering. There are also a large number of methods that enhance the temporal consistency with the aid of pre-trained optical-flow models [73, 72, 11, 30] . Although the effectiveness of them, all of them severely rely on a pre-trained optical-flow model. Recent works [66] illustrate that the diffusion feature contains rich correspondence information. Although VideoSwap [17] adopts this characteristic by tracking the key points across frames, it still needs users to provide the key points as the extra addition manually. 
method: 
 In this section, we will introduce COVE in detail, which can be seamlessly integrated into the pre-trained T2I diffusion model for high-quality and consistent video editing without the need for training or optimization (Figure 3 ). Specifically, given a source video, we first extract the diffusion feature of each frame using the pre-trained T2I diffusion model. Then, we calculate the one-to-many correspondence of each token across frames based on cosine similarity (Figure 3a ). To reduce resource consumption during correspondence calculation, we further introduce an efficient slidingwindow-based strategy (Figure 4 ). During each timestep of inversion and denoising in video editing, the tokens in the noisy latent are sampled based on the correspondence and then merged. Through the self-attention among merged tokens (Figure 3b ), the quality and temporal consistency of edited videos are significantly enhanced.Diffusion Models. DDPM [27] is the latent generative model trained to reconstruct a fixed forward Markov chain x 1 , . . . , x T . Given the data distribution x 0 ∼ q(x 0 ), the Markov transition q(x t |x t-1 ) is defined as a Gaussian distribution with a variance schedule β t ∈ (0, 1).EQUATIONTo generate the Markov chain x 0 , • • • , x T , DDPM leverages the reverse process with a prior distribution p(x T ) = N (x T ; 0, I) and Gaussian transitions. A neural network ϵ θ is trained to predict noises, ensuring that the reverse process is close to the forward process.EQUATIONwhere τ indicates the textual prompt. µ θ and Σ θ are predicted by the denoising model ϵ θ . Since the diffusion and denoising process in the pixel space is computationally extensive, latent diffusion [57] is proposed to address this issue by performing these processes in the latent space of a VAE [37] .DDIM Inversion. DDIM can convert random noise to a deterministic x 0 during sampling [64, 13] .The inversion process in deterministic DDIM can be formulated as follows:EQUATIONwhere α t denotes t i=1 (1 -β i ). The inversion process of DDIM is utilized to transform the input x 0 into x T , facilitating subsequent tasks such as reconstruction and editing.As discussed in Section 1, intra-frame correspondence is crucial for the quality and temporal consistency of edited videos while remaining largely under-explored in existing works. In this section, we introduce our method for obtaining correspondence relationships among tokens across frames. Diffusion Feature Extraction. Given a source video V with N frames, a VAE [37] is employed on each frame to extract the latent featuresZ = {z 1 , • • • , z N }, where Z ∈ R N ×H×W ×d .Here, H and W denote the height and width of the latent feature and d denotes the dimension of each token. For each frame of Z, we add noise of a specific timestep t and feed the noisy frame Z t = {z t 1 , • • • , z t N } into the pre-trained T2I model f θ respectively. The diffusion feature (i.e., the intermediate feature from the U-Net decoder) is extracted through a single step of denoising [66] :EQUATIONwhere F ∈ R N ×H×W ×d , denoting the normalized diffusion feature of each frame.One-to-many Correspondence Calculation. For each token within the diffusion feature F , its corresponding tokens in other frames are identified based on the cosine similarity. Without loss of generality, we could consider a specific token p {i,hi,wi} in the ith frame F i with the coordinate [h i , w i ]. Unlike previous methods where only one corresponding token of p {i,hi,wi} can be identified in each frame (Figure 2a ), our method can obtain the one-to-many correspondences simply by selecting tokens with the top K highest similarity in each frame. We record their coordinates, which are used for sampling the tokens for self-attention in the subsequent inversion and denoising process.To implement this process, the most straightforward method is through a direct matrix multiplication of the normalized diffusion feature F .EQUATIONwhere S ∈ R (N ×H×W )×(N ×H×W ) represents the cosine similarity between each token and all tokens in the diffusion feature of the video.The similarity between p {i,hi,wi} and all N ×H×W tokens in the feature is given by S[i, h i , w i , :, :, :].The coordinates of the corresponding tokens in the jth frame (j ∈ {1, • • • , N }) are then obtained by selecting the tokens with the top K similarities in the jth frame.EQUATIONHere the top-k-argmax(•) denotes the operation to find coordinates of the top K biggest values in a matrix, wherek ∈ {1, • • • , K}. [h k j , w k j ]represents the coordinates of the token in jth frame which has highest similarity with p {i,hi,wi} . A similar process can be conducted for each token of F , thereby obtaining their correspondences among frames.Sliding-window Strategy. Although the one-to-many correspondence among tokens can be effectively obtained through the above process, it requires excessive computational resources because (N × H × W ) is always a huge number, especially in long videos. As a result, the computational complexity of this process is extremely high, which can be represented asO(N 2 × H 2 × W 2 × d).At the same time, multiplication between these two huge matrices consumes a substantial amount of GPU memory in practice. These limitations severely limit its applicability in many real-world scenarios, such as on mobile devices.To address the above problem, we further propose the sliding-window-based strategy as an alternative, which not only effectively obtains the one-to-many correspondences but also significantly reduces the computational overhead (Figure 4 ). Firstly, for the token p {i,hi,wi} , it is only necessary to calculate its similarity with the tokens in the next frame F i+1 instead of in all frames, i.e.,EQUATIONS i ∈ R H×W ×H×W denotes the similarity between the tokens in ith frame and those in (i+1)th frame. The overall similarity matrix isS = {S i }, i ∈ {1, 2, • • • , N -1}, where S ∈ R (N -1)×H×W ×H×W .Then, we obtain the K corresponded tokens of p {i,hi,wi} in F i+1 through S i ,EQUATIONFor tokens in (i + 2)th frame, instead of considering p {i,hi,wi} , we identify the tokens in (i + 2)th frame which have the top K largest similarity with the token p {i+1,h 1 i+1 ,w 1 i+1 } through the S i+1 . Similarly, we can obtain the corresponding token in other future or previous frames.EQUATIONThrough the above process, the overall complexity is reduced to O((N -1) × H 2 × W 2 × d).Furthermore, it is noteworthy that frames in a video exhibit temporal continuity, implying that the spatial positions of corresponding tokens are unlikely to change significantly between consecutive frames. Consequently, for the token p {i,hi,wi} , it is enough to only calculate the similarity within a small window of length l in the adjacent frame, where l is much smaller than H and W ,EQUATIONF w i+1 ∈ R l×l×d represents the tokens in F i+1 within the sliding window. We calculate the cosine similarity between p {i,hi,wi} and the tokens in F w i+1 , selecting tokens with top K highest similarity within F w i+1 . This approach further reduces the computational complexity to O((N -1) × H × W × l 2 × d) and the GPU memory consumption is also significantly reduced in practice. Additionally, it is worth noting that calculating correspondence information from the source video is only conducted once before the inversion and denoising process of video editing. Compared with the subsequent editing process, this process only takes negligible time.In this section, we explain how to apply the correspondence information to the video editing process (Figure 3c ). In the inversion and denoising process of video editing, we sample the corresponding tokens from the noisy latent for each token based on the coordinates obtained in Section 3.2. For the token z t i,hi,wi , the set of corresponding tokens in other frames at a timestep t is:EQUATIONWe merge these tokens following [5] , which can accelerate the editing process and reduce GPU memory usage without compromising the quality of editing results:EQUATION) Then, the self-attention is conducted on the merged tokens,EQUATIONEQUATIONwhere √ d k is the scale factor. The above process of correspondence-guided attention is illustrated in Figure 3b . Following the previous methods [73, 11] , we also retain the spatial-temporal attention [69] in the U-Net. In spatial-temporal attention, considering a query token, all tokens in the video serve as keys and values, regardless of their relevance to the query. This correspondence-agnostic self-attention is not enough to maintain temporal consistency, introducing irrelevant information into each token, and thus causing serious flickering effects [11, 16] . Our correspondence-guided attention can significantly alleviate the problems of spatial-temporal attention, increasing the similarity of corresponding tokens and thus enhancing the temporal consistency of the edited video. In winterTraditional Chinese house In autumn Wide sea and snow mountain 4 Experiment 
experimental setup: 
 In the experiment, we adopt Stable Diffusion (SD) 2.1 from the official Huggingface repository for COVE, employing 100 steps of DDIM inversion and 50 steps of denoising. To extract the diffusion feature, the noise of the specific timestep t = 261 is added to each frame of the source video following [66] . The feature is then extracted from the intermediate layer of the 2D Unet decoder during a single step of denoising. The window size l is set to 9 for correspondence calculation, and k is set to 3 for correspondence-guided attention. The merge ratio for token merging is 50%. For both qualitative and quantitative evaluation, we select 23 videos from social media platforms such as TikTok and other publicly available sources [1, 2] . Among these 23 conducted on a single RTX 3090 GPU for our method unless otherwise specified. We compare COVE with 5 baseline methods: FateZero [54] , TokenFlow [16] , FLATTEN [11] , FRESCO [73] and RAVE [33] . For all of these baseline methods, we follow the default settings from their official Github repositories. The more detailed experimental settings of our method are provided in Appendix A.We evaluate COVE on various videos under different types of prompts including both global and local editing (Figure 5 ). Global editing mainly involves background editing and style transferring.For background editing, COVE can modify the background while keeping the subject of the video unchanged (e.g. Third row, first column. "a car driving in milky way"). For style transfer, COVE can effectively modify the global style of the source video according to the prompt (e.g. Third row, second column. "Van Gogh style"). Our prompts for local editing include changing the subject of the video to another one (e.g. Third row, third column. "A cute raccoon") and making local edits to the subject (e.g. fifth row, third column. "A sorrow woman"). For all of these editing tasks, COVE demonstrates outstanding performance, generating frames with high visual quality while successfully preserving temporal consistency. We also compare COVE with a wide range of state-of-the-art video editing methods (Figure 6 ). The experimental results illustrate that COVE effectively edits the video with high quality, significantly outperforming the previous methods.For quantitative comparison, we follow the metrics proposed in VBench [31] , including Subject Consistency, Motion Smoothness, Aesthetic Quality, and Imaging Quality. Among them, Subject Consistency assesses whether the subject (e.g., a person) remains consistent throughout the whole video by calculating the similarity of DINO [8] feature across frames. Motion Smoothness utilizes the motion priors of the video frame interpolation model [40] to evaluate the smoothness of the motion in the generated video. Aesthetic Quality uses the LAION aesthetic predictor [38] to assess the artistic and beauty value perceived by humans on each frame. Imaging Quality evaluates the degree of distortion in the generated frames (e.g., blurring, flickering) through the MUSIQ [36] image quality predictor. Each video undergoes editing with 3 global prompts (such as style transferring, background editing, etc.) and 2 local prompts (such as editing the appearance of the subject in the video), generating a total of 115 text-video pairs. For each metric, we report the average score of these 115 videos. We further conducted a user study with 45 participants following [73] . Participants are required to choose the most preferable results among these methods. The result is shown in Table 1 . Among various methods, COVE achieves outstanding performance in both qualitative metrics and user studies, further demonstrating its superiority. We conduct an ablation study to illustrate the effectiveness of the Correspondence-guided attention and the number of tokens selected in each frame (i.e., the value of K). The experimental results (Table 2 and Figure 7 ) illustrate that without correspondence-guided attention, the edited video exhibits obvious temporal inconsistency and flickering effects (which is marked in yellow and orange boxes in Figure 7 ), thus severely impairing the visual quality. As K increases from 1 to 3, the generated video contains more fine-grained details, exhibiting better visual quality. However, further increasing K to 5 does not significantly improve the video quality. We also illustrate the effectiveness of temporal dimensional token merging. By merging the tokens with high correspondence across frames, the editing process becomes more efficient (Table 3 ) while there is no significant decrease in the quality of the edited video (Figure 8 ). The ablation of the sliding-window size l is shown in Appendix B. If the window size is too small, the actual corresponding token may not be included within the window, resulting in suboptimal correspondence and poor editing results. On the other hand, a too-large window size is not necessary for identifying the corresponding tokens, which would lead to high computational complexity and excessive memory usage. The experiment results illustrate that l = 9 is suitable to strike a balance. Additionally, we also visualize the correspondence obtained by COVE, which is shown in Appendix C. Table 3 : Ablation Study on the effect of temporal dimensional token merging. Temporal dimensional token merging can speed up the editing process and save GPU memory usage while hardly impairing the quality of the generated video. The experiment is conducted on a single RTX3090 GPU with a 10-frame source video. k is set to 3.  
conclusion: 
 In this paper, we propose COVE, which is the first to explore how to employ inherent diffusion feature correspondence in video editing to enhance editing quality and temporal consistency. Through the proposed efficient sliding-window-based strategy, the one-to-many correspondence relationship among tokens across frames is obtained. During the inversion and denoising process, self-attention is performed within the corresponding tokens to enhance temporal consistency. Additionally, we also apply token merging in the temporal dimension to improve the efficiency of the editing process. Both quantitative and qualitative experimental results demonstrate the effectiveness of our method, which outperforms a wide range of previous methods, achieving state-of-the-art editing quality.Limitaions. The limitation of our method is discussed in Appendix G. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use explicitly are mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We visualize the correspondence calculated by our sliding-window-based method to illustrate its effectiveness (Fig. 10 ). To be specific, we calculate the correspondence based on the 64 × 64 diffusion feature, which is extracted at the final layer of the U-net decoder. The result illustrates that our method can effectively identify the corresponding tokens. The correspondence acquired through the diffusion feature is accurate and robust. As there is no existing video dataset with the annotated keypoints on each frame, to further evaluate its accuracy quantitatively, we collect 5 videos with 30 frames and 5 videos with 60 frames and manually label some keypoints on each frame. Then we report the percentage of correct keypoints (PCK). Specifically, for each video, given the first frame with the keypoints, we obtain the predicted corresponding keypoints on other frames through the diffusion feature. Then we evaluate the distance between the predicted points and the ground truth. The predicted point is considered to be correct if it lies in a small neighborhood of the ground truth. Finally, the total number of correctly predicted points divided by the total number of predicted points is the value of PCK. The result in Appendix D illustrates that the diffusion feature can accurately find the correct position in most cases for video editing.Optical-flow Correspondence 0.87 Diffusion feature Correspondence 0.92 The quality of noise obtained by inversion can significantly affect the final quality of editing. The Correspondence-Guided Attention (CGA) during inversion can increase the quality and temporal consistency of the obtained noise, which can further help to enhance the quality and consistency of edited videos. The ablation of it is shown in Fig. 11 F Broader Impacts Our work enables high-quality video editing, which is in high demand across various social media platforms, especially short video websites like TikTok. Using our method, people can easily create With Corr means the correspondence-guided attention is applied in both inversion and denoising stages, illustrating satisfying performance.high-quality and creative videos, significantly reducing production costs. However, there is a potential for misuse, such as replacing the characters in videos with celebrities, which may infringe upon the celebrities' image rights. Therefore, it is also necessary to improve relevant laws and regulations to ensure the legal use of our method.Despite achieving outstanding results, our methods still encounter several limitations. First, although the correspondence calculation process is efficient through the proposed sliding window strategy, the implementation of correspondence-guided attention is still not efficient enough, leading to the extra usage of GPU memory and time (Table 3 ). This problem is expected to be alleviated largely through the use of xFormers. We will work on it in the future.Second, further exploration is required to optimize the application of the obtained correspondence information. In this study, we utilize the correspondence information to sample tokens during the inversion and denoising processes and do the self-attention. However, we believe that there may be more effective alternatives to self-attention that could further unleash the potential of the correspondence information.We provide more qualitative results of our method to illustrate its effectiveness, which is shown in Fig. 13 and Fig. 12 . Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper received ratings of $(7, 6, 5, 4)$ after the rebuttal and discussions. **Reviewer Ht4d** and **Reviewer LETZ**, who gave ratings of $7$ and $6$, acknowledged that the authors' responses addressed their concerns and thus expressed their support. The remaining two reviewers, while having borderline opinions, did not raise further issues after the rebuttal. Specifically, the rebuttal provided clarifications with additional figures and tables to address **Reviewer sLJK**'s questions about the differences with optical flow models, the robustness of using sliding windows, and the novelty of the work.

In light of these considerations, the area chair recommends accepting this paper. The authors are encouraged to incorporate the discussions and additional figures and tables in the final version. Furthermore, it is imperative that the authors address the ethical concerns raised by the Ethics Reviewers regarding potential misuse and the user study involving human subjects.

Reviews: review: paper_summary: paper_summary: The paper proposes using the correspondence features already exist in diffusion models to find matched tokens between different frames in a video for consistent video editing. The motivation is that video editing models using optical flow to find matched features can exhibit one-to-many matching issue which limits the quality of video editing consistency when there are large motions. To overcome expensive compute, the paper proposes using "sliding window" (w.r.t frames) by only finding correspondences in subsequent frames. Reasonable qualitative and quantitative results are shown in the experiments section.
strength_weakness: strength_weakness: - Optical flow is also essentially just finding matched patches/features between different frames. The proposed way of finding one-to-many correspondences between frames can also be applied uusing optical flow (with some small modifications of existing optical flow algorithms). I do not see any evidence from the paper that using the features in a diffusion model is better than using the features extracted by an optical flow algorithm (explicitly or inexplicitly depending on different optical flow methods)

- the above should be an important ablation study that is currently missing from the paper

- the sliding window algorithm, though simple and achieving compute reduction, can also lead to errors accumulating across frames when editing long videos

- the novelty is a bit limited. My argument is the following:
    1. Features inexplicitly extracted in Stable Diffusion can be used for feature correspondence is well-known to the community, e.g. in Tang et al. NeurIPS 2023
    2. Finding correspondences for video editing is widely studied and shown to be effective, including but not limited to the several references this paper already cited
    3. The main contribution then seems to be proposing using SD's features to find correspondences instead of using optical flow. However, the one-to-many problem and many-to-one problem is essentially a thresholding problem when finding correspondences and the receptive field size used to find matched patches, and I cannot see why the features extracted by SD is superior than the features extracted by a state-of-the-art optical flow model. This ablation study is missing
questions: questions: See weakness
limitations: limitations: No
paper_summary: paper_summary: This paper focused on improving the temporal consistency of video editing. They propose to leverage the inherent diffusion feature correspondence with a sliding-window based strategy. With this design, the tokens in noisy latents can be sampled based on the “one-to-many” correspondence. The experiments demonstrate superior performance than other methods.
strength_weakness: strength_weakness: - The proposed method highly relies on the correspondences of diffusion features. However, such correspondences may be difficult to obtain for videos with large content motions. The reviewer suggests a more detailed discussion about how to ensure the accuracy of the correspondences, as well as what are the potential limitations of inaccurate correspondences. In addition, it would be better to quantitatively evaluate the correctness of the correspondences of diffusion features.

- The paper mainly discussed and compared their method with optical flow-guided video editing methods. However, such correspondence-based idea is also related to deformation field based methods such as CoDeF [1] or neural atlas based methods [2], where they can ensure the accuracy of pixel / point level accuracy by evaluating the video reconstruction accuracy. The authors are suggested to compare or discuss their method with such approaches.

- The related works seem to be a bit short and less extensive. There have been many video editing works that target at improving temporal / subject consistency, efficiency, long video editing, etc. The reviewer suggests to include a more extensive related works and discuss the differences of the proposed work with them. 

[1] Ouyang, Hao, et al. "Codef: Content deformation fields for temporally consistent video processing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Chai, Wenhao, et al. "Stablevideo: Text-driven consistency-aware diffusion video editing." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
questions: questions: o	The reviewer appreciates the qualitative examples of correspondences of diffusion features in appendix. Is there any quantitative metric to evaluate the correspondences of diffusion features?

o	How to define the temporal length for token merging? Does it remain the same for all experiment videos?
limitations: limitations: Yes.
paper_summary: paper_summary: In this paper, the authors tackle the problem of video editing using Text-to-Image diffusion models. To achieve this, the authors make use of strong diffusion model’s feature correspondence abilities. The authors propose a sliding-window based strategy to track features of source video based on correspondences. Here, the goal is to overcome the computational complexity involved with finding similarity across all patches of the video. This is achieved by computing similarity scores within a sliding window. After this, a set of corresponding tokens are collected and merged using an existing work (Token Merging paper, ICLR 2023). Experimental analysis shows that the proposed method achieves state-of-the-art performance across many tasks.
strength_weakness: strength_weakness: W1. The paper clearly states that the base model is Stable diffusion on L208 and the model does not require any training including inflation (which I think is a major strength).  Stable diffusion model has only self-attention and cross-attention layers. It is not clear how “temporal layers” come into picture on L200. Further, it is not clear what is the 3D-Unet in context of Fig. 3.

W2. The proposed method tends to alter the regions of the image that do not correspond to the text prompt. In Fig. 1, the results shows that the proposed alters the background significantly. 

W3. Results show that the proposed method is unable to alter the shape of the object, since the correspondence merging limits the shape changing ability drastically. 

W4. The core novelty of exploiting the sliding window to use a lower field of comparison (as. In Eq. 10) is somewhat straightforward and limited given the obvious computational constraints. However, in conjunction with other ideas such as Token Merging [ref 1] and correspondences [ref 2], the work is novel. 

W5. There have been works in the video editing space which exploit correspondences for video editing [ref 3,4]. It will be helpful to differentiate the proposed method with them. However, I would like to acknowledge that these works could have been parallel to this submission.

References

[ref 1] Token Merging: Your ViT But Faster, ICLR 2023

[ref 2] Emergent Correspondence from Image Diffusion, NeurIPS 2023

[ref 3] Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer, CVPR 

[ref 4] GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models, CVPRw
questions: questions: - How does applying \tilde{Corr} help during inversion?
limitations: limitations: The paper addresses the limitations of the work in Appendix E.
paper_summary: paper_summary: The paper introduces COrrespondence-guided Video Editing (COVE), a method to improve video editing with pretrained text-to-image (T2I) diffusion models. It addresses the challenge of maintaining temporal consistency by using diffusion feature correspondence. COVE identifies and samples highly corresponding tokens across frames, applying self-attention to them during editing. It also reduces GPU memory usage with a temporal-dimensional token merging strategy. COVE integrates seamlessly into existing T2I models without extra training and shows superior performance in various scenarios.
strength_weakness: strength_weakness: 1. Experimental performance improvement seems limited. Most of the editing results present a similar type involving style/color change, which is easy to implement in the previous methods from my experience. According to Fig. 6, the performance compared to earlier methods does not show high superiority, which is not convincing to support "...COVE achieves the start-of-the-art performance...". 

2. I noticed that [1] also uses DIFT features to guide video editing, especially object replacement. Could you please compare COVE with it? For example, replace the "jeep" with a "sports car" or "bus" in Fig. 11. I would like to see the performance when there is a large motion/shape change.

[1] VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence
questions: questions: How do you make sure that $k=3$ is suitable for all videos? As suggested in Fig. 2, the subject may suffer from severe morph across video frames when there is a significant motion change.
limitations: limitations: See weakness above.

Comments: title: Response by Reviewer LETZ
comment: Dear reviewer nziD,

Thanks for your valuable suggestions on our paper!

We have responsed to each of your concerns in our rebuttal, including further explanation on the performance of our method, more 
qualitative results for shape editing (Figure 15 in the uploaded PDF) and the chosen value $K$. 

If you have any unsolved issues, please let us know. We are more than happy to answer any further questions you may have!

Authors
Dear reviewer sLJK,

Thanks for your valuable advice on our paper. 

We have responsed to each of your concerns in our rebuttal, including the comparasion between optical-flow correspondence and diffusion feature correspondence (Table 5, Table 7 and ***Figure 13 in the uploaded PDF***), accuracy of sliding window method on long videos (Table 6 and ***Figure 14 in the uploaded PDF***), and the novelty of our method. We also point out that using the optical flow model to obtain the one-to-many correspondence is nontrivial. 

If you have any unsolved questions, please let us know. We are more than happy to answer any further questions you may have!

Authors
The rebuttal addresses the concerns. I highly encourage authors to include this discussion in the paper. Hoping that authors will include these discussions in the paper, I raise my score to an "accept".
Dear Authors,

Thanks for your hard work and detailed analysis. The rebuttal has resolved all my concerns. The quantitative experiments of diffusion feature correspondences on rebuttal is helpful to demonstrate the effectiveness of leveraging diffusion features for video editing. The authors are suggested to include these results in the paper. Therefore, I decide to raise my score from 5 to 6. 

Best regards,

Reviewer LETZ
Dear all reviewers,

Thanks for your time and effort in reviewing our paper! 

We are pleased that you acknowledge the strength of our paper such as its clear motivation and superior performance. **We have responsed to each of your concerns in our rebuttal. If you have any unresolved issues, please let us know.** We are more than happy to answer any further questions you may have!

Thank you once again for your consideration.
Dear ethics reviewer oZr4,

Thanks for your suggestions! 

We have discussed the broader impacts of our method in appendix. One possible solutions to mitigating the risks you mentioned (such as abuse in private life) is through conditionally releasing the code. Every user needs to register in an application system before obtaining the source code of our paper. Then if there is any illegal use, we can locate the person who misuse our code and take actions to mitigate the negative impact in a short time.

We will further discuss more potential misuse cases and possible strategies to mitigate them in the final revision of our paper as you suggested.
Dear ethics reviewer Q3sx,

Thanks for your valuable advice! 

In our user study, participants were recruited through advertisements targeting individuals interested in video quality assessment. We aimed for a diverse pool, ensuring a mix of age, gender, and technical expertise. All participants were provided with a detailed informed consent form outlining the study's purpose and procedures, and potential risks. They were required to read and sign the form before participating in the study. We emphasized their right to withdraw from the study at any time without penalty. Participants were compensated for their time with a small monetary payment.

We will provide the more detailed information about our user study in the final revision of our paper as you suggested.