UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

Authors: Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, Jiang Bian
Primary area: generative models
Keywords: Diffusion Model, Video Editing

Abstract:
Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, we present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversion-then-generation framework.
To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, we introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers. Extensive experiments demonstrate that UniEdit covers video motion editing and various appearance editing scenarios, and surpasses the state-of-the-art methods. Our code will be publicly available.

Decision: reject
Reviewer scores: 5, 8, 5, 1

Reviews and discussion:
<thread>
creation date:
1730611131959

thread_type:
official_review

<thread_object>
creator: reviewer_exku

<content>
summary:
Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, They present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversionthen-generation framework. However, this paper seems not latest version. Some new sota approaches are not included and had a comparison.

strengths:
In this work, they present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversionthen-generation framework. To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, they introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers.

weaknesses:
- Novelty.  Using ReferenceNet is not a novel approach in video generation. What is the difference between yours and previous work.
- Overclaim.  "UniEdit represents a pioneering leap in text-guided, tuning-free video motion editing."  Why this work is a pioneering leap? 
- Lack of comparison. Many new works, such as Revideo, COVE , accepted by NeurIPS 2024, are all not included in this paper. 
- Long video editing,  All results are only 16 frames，which is too short.
- Motion Editing. It seems that all motion are similar. How to editing it ?
- Inconsistency.  Such as cat -> dog. The consistency of the head turn is problematic and exhibits sudden changes.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687093662

thread_type:
official_review

<thread_object>
creator: reviewer_vcxp

<content>
summary:
The authors propose UniEdit, addressing the task of video motion and appearance editing using pre-trained video diffusion models (VDMs) in a united and tuning-free manner. Specifically, they apply the common "inversion-and-generation" with two innovative auxiliary branches, i.e., the reconstruction branch for content preservation and the motion-reference branch for motion editing, in addition to the main editing path. The appearance editing, motion editing and structure preservation are realized by swapping attention features (QKV) in specific types and depth of attention layers, and denoising steps as well. UniEdit is compared with previous zero-shot/tuning-free methods and outperforms them both quantitatively and qualitatively.

strengths:
- The paper demonstrates good clarity in writing, making it easy for readers to follow the flow of ideas.
- The paper is well-motivated, offering various types of video editing while preserving non-target properties using a unified framework.
- The design of attention replacement for appearance, structure, and motion editing is technically sound, leveraging the insights and functionality of spatial and temporal self-attention, respectively.
- The paper and the auxiliary material show impressive editing results produced with their framework with substantial improvement on existing baselines, especially in temporal coherence.

weaknesses:
- The authors claim in L217 that "better tackle source video with large dynamics". However, most visual examples are with small motion magnitudes. Does the model struggle with appearance and motion editing in videos with larger dynamics? Temporal inconsistencies are noticeable in videos with large motion (see Fig G.1).
- The authors claim in L312 that "UniEdit is not limited to specific video diffusion models", which is inaccurate. The proposed framework depends on the separate spatial and temporal attention design of video diffusion models, which is incompatible with full 3D attention DiT-based video generators. And 3D full attention is increasingly being proven to be more effective [1].
- This work still follows the common "inversion-and-genertation" paradigm, potentially performing better on generated videos but struggling with out-of-domain real videos.
- In Fig.4 (bottom-right case), motion-editing results show background changes, such as alterations in the lawn.
- The user study involved an insufficient number of participants (only 10), making it difficult to support the conclusions effectively with robust experimental results.

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer, Yang et al., 2024

Minor concerns:
- [Clarity] Consider including a note below the table or in the caption indicating that higher values are better.

questions:
- [Ablation study] Why does the *content preservation* improve *textual alignment* a lot in Table 2?
- I believe different methods have their own "success rate" in generation. Is there a way to measure this rate for fair comparison, beyond just showcasing several visual examples? Additionally, how to know if this tuning-free method really works or not (to what degree), compared to potential methods with large-scale training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703461580

thread_type:
official_review

<thread_object>
creator: reviewer_u6ds

<content>
summary:
UniEdit presents a novel approach to text-guided video editing, utilizing a tuning-free framework that leverages the capabilities of pre-trained text-to-video (T2V) diffusion models. The key strength of UniEdit lies in its ability to perform both motion and appearance editing within a unified architecture. This is a significant advancement compared to prior methods, which often focus on either motion or appearance editing, but not both. The core idea behind UniEdit is to invert the source video into the latent space of a pre-trained T2V model, and then guide the generation process using the target text prompt and auxiliary branches that control motion and appearance.

strengths:
- **Unified Framework for Motion and Appearance Editing**: UniEdit effectively addresses a key challenge in video editing by providing a single framework that can handle both motion and appearance modifications. This is achieved by leveraging the inherent properties of spatial and temporal self-attention layers in the T2V model. The motion-reference branch, conditioned on the target prompt, injects motion features into the temporal self-attention layers, enabling text-guided motion changes. The reconstruction branch focuses on preserving source content by injecting features into the spatial self-attention layers.

- **Zero-Shot Editing Capability**: UniEdit operates in a tuning-free manner, meaning it doesn't require fine-tuning the base T2V model for each new video or editing task. This makes it highly efficient and user-friendly, as it avoids the computational cost and complexity associated with model training.

- **Emphasis on Temporal Consistency:** Maintaining temporal consistency is crucial for high-quality video editing. UniEdit tackles this challenge by injecting features from the reconstruction branch into the spatial self-attention layers, which helps preserve the content and structure of the source video. The use of mask-guided coordination further enhances the consistency of unedited areas.

weaknesses:
1. The problem is challenging, and I believe this paper only partially addresses it. The editing capabilities need to be more clearly defined. For example, are all text prompts representing motion control or appearance control feasible? The scope requires further clarification. The inherent ambiguity of language and the complexity of video editing tasks make it practically impossible to guarantee success for every possible textual input.

**Defining Boundaries and Constraints:** To enhance clarity regarding its scope, the UniEdit paper should clearly state:

- **Types of Edits Supported:** Specify the categories of motion and appearance edits that UniEdit is designed to handle effectively.

- **Limitations in Prompt Complexity:** Provide guidelines or examples of text prompts that might pose challenges for UniEdit, particularly those involving complex scenes or multiple simultaneous edits.

- **Known Failure Cases:** Discuss specific scenarios where UniEdit is known to perform poorly, offering insights into the reasons for these limitations.

2. UniEdit's core techniques are indeed inspired by existing approaches:
   - **Auxiliary Branches:** The concept of using auxiliary branches for specific tasks, such as reconstruction or motion guidance, is not entirely new.
   - **Feature Injection via Self-Attention:** Injecting features into specific layers of a neural network, especially within attention mechanisms, is a common practice in various image and video editing tasks.
   - **Temporal and Spatial Self-Attention:** The observation that temporal self-attention layers encode inter-frame dependencies, while spatial self-attention layers capture intra-frame information, is a widely recognized principle reported in other works like I2VEdit (https://i2vedit.github.io/) for diffusion-based methods and SIFA (https://github.com/FuchenUSTC/SIFA) for other video editing tasks.

   While UniEdit's core techniques are not entirely novel, **the integration and adaptation of these mechanisms for tuning-free video motion editing demonstrate a degree of innovation.**

3. The improvements, while statistically significant, do not represent a leap in performance but rather a refinement of existing techniques. For instance, UniEdit's performance on Frame Quality and Temporal Quality metrics surpasses competing methods like TokenFlow and Rerender, but the margins are relatively modest (e.g., a 1-2% improvement in some metrics).

4. As the authors have pointed out, UniEdit has a **performance ceiling** when performing both motion and appearance editing simultaneously and struggles with complex scenes.

5. **Missing Critical Baseline:** ReVideo (https://mc-e.github.io/project/ReVideo/) should be included as a baseline. ReVideo's focus on remaking videos with control over both motion and content aligns well with UniEdit's capabilities. A direct comparison would reveal their strengths and weaknesses in tasks such as object replacement, style transfer, and motion modification.

   Including additional works for comparison will further strengthen this paper. 

  In addition, a discussion with the following related works is beneficial:
   - **AnyV2V:** Its compatibility with various image editing methods and emphasis on simplicity and consistency present an interesting comparison point for UniEdit.
   - **I2VEdit:** Comparing with I2VEdit would reveal the advantages and disadvantages of using an edited first frame versus direct text guidance for video editing. This comparison could be particularly insightful for tasks involving local edits, global style transfer, identity manipulation, and subject customization. The temporal consistency of I2VEdit appears to be quite robust.
   - **FLATTEN:** Comparing FLATTEN to UniEdit would showcase the differences between explicitly incorporating optical flow information and UniEdit's approach to motion injection.

6. **Metric Coverage:** The evaluation relies heavily on CLIP scores and user preferences. While these metrics are valuable, the absence of more comprehensive metrics (e.g., temporal coherence measures beyond flickering, user engagement metrics) limits the robustness of the evaluation.
   - **Flow Warping Error (Ewarp):** This metric can be used to assess visual consistency by comparing the edited video frames warped according to the estimated optical flow of the source video.
   - **Frame Variance Metric for Motion Detection (FVMD):** This may provide a more objective measure of how well the model preserves motion consistency across frames.

7. The evaluation dataset could include BalanceCC if possible.

8. **Sensitivity Analysis** can be more detailed:
   - **Sensitivity to Injection Steps:** The quantitative analysis clearly shows that the choice of injection steps influences the quality and consistency of the output.
   - **Sensitivity to Blend Layers and Steps:** The qualitative exploration indicates a trade-off between stylization and fidelity controlled by the blend layers and steps. This suggests a need for adjustment based on the desired outcome.

9. The related works are primarily categorized by broad topics (e.g., video generation, video editing) rather than by the specific innovations relevant to UniEdit (e.g., tuning-free frameworks, self-attention manipulation). This top-down organization makes it harder to discern how UniEdit differentiates itself from existing methods.

questions:
Please refer to the weaknesses section for my questions. I may revise my rating based on the authors' rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731306298

thread_type:
official_review

<thread_object>
creator: reviewer_ikzp

<content>
summary:
The paper proposes a method to simultaneously edit motion and appearance through text guidance. Unlike previous approaches that required fine-tuning a pretrained generator model for the target video, the proposed UniEdit method is tuning-free. It utilizes the temporal self-attention layer within the generator to naturally express motion. Additionally, the spatial self-attention layer preserves the regions that are not being edited. Furthermore, the method ensures that the spatial structure of the video remains intact during appearance editing.

strengths:
The method can be applied to various Text-to-Video models, clearly stating the intended goal of simultaneously editing the content's motion and appearance.
The paper presents results applied to multiple Text-to-Video models. Experiments using various prompts demonstrate that this method is not limited to specific prompts.
The video quality available on the project page is excellent, and the experimental results are easy to understand.
Videos generated using UniEdit outperformed existing methods in terms of CLIP score, Frame Quality, and Temporal Quality.

weaknesses:
The user study involved only 10 participants, which may lead to highly subjective results. Including subjective results in the main performance table could confuse readers, and it would be beneficial to either evaluate with more participants or clearly indicate near the table that the results may be subjective due to the limited number of participants.
The subjective evaluation from only 10 participants being included in the quantitative results could cause confusion among readers.

questions:
Please check if it is reasonable to have conducted the experiment with 10 subjects.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732449470726

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision

comment:
We thank all the reviewers for your time and thoughtful feedback. We revised our manuscript and submitted a new version for review. We also added the video results corresponding to the rebuttal to our anonymous project page (https://uni-edit.github.io/UniEdit/#sectionRebuttal) for your reference.

### Project Page

1. Figure A: Including enhanced results of UniEdit with mask-guided coordination on challenging scenarios (Reviewer u6DS).
2. Figure B: Adding comparison with ReVideo [1] (Reviewer u6DS, Reviewer eXku).
3. Figure C: Adding enhanced results of UniEdit with improved base T2V generation model (Reviewer vcxP, Reviewer eXku).
4. Figure D: Adding enhanced results of UniEdit with mask-guided coordination (Reviewer vcxP).

### Main Paper

1. We update the user study results on more participants (Reviewer iKZP, Reviewer vcxP).
2. We reorganize and revise the related works section to make it clearer and add a detailed discussion of recent works (Reviewer u6DS).
3. We fix some typos.

### Appendix

1. We include Fréchet video motion distance(FVMD) [2] to evaluate the motion consistency in Tab. 7 (Reviewer u6DS).


[1] Mou, Chong, et al. "ReVideo: Remake a Video with Motion and Content Control." *arXiv preprint arXiv:2405.13865* (2024).

[2] Liu, Jiahe, et al. "Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos." *First Workshop on Controllable Video Generation@ ICML24*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732560560693

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for Your Review

comment:
Dear Reviewers,

Thanks again for the constructive comments and the time you dedicate to the paper! Many improvements have been made according to your suggestions and other reviewers' comments. We also updated the manuscript and summarized the changes below. We hope that our responses and the revision will address your concerns. 

Since the discussion is about to close, we would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues. We are glad to follow up with your further comments.

Thanks a lot again, and with sincerest best wishes

Submission 5877 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a tuning-free framework for video motion and appearance editing with a pre-trained text-to-video base model. A primary objective of the editing process is to change the motion while preserving source video content. This goal is accomplished by adding auxiliary branches for motion reference and reconstruction, which produce text-guided motion and source features. These features are then integrated into the temporal and spatial self-attention layers to control inter-frame and intra-frame dependencies.

The paper initially received mixed and somewhat negative ratings: 6, 6, 5, and 3. After the rebuttal, the ratings became more varied with scores of 8, 5, 5, and 1. Despite the authors' efforts to address the reviewers' comments and provide additional results, three of the reviewers continued to express concerns regarding the novelty and technical contributions of the work. The area chair reviewed the paper and found the reviewers' arguments somewhat convincing. Consequently, the area chair did not identify strong reasons to overrule the reviewers' decision and regretfully recommended rejecting the paper.