UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

## Abstract

Abstract Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, we present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversion-then-generation framework.
To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, we introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers.
Extensive experiments demonstrate that UniEdit covers video motion editing and various appearance editing scenarios, and surpasses the state-of-the-art methods. Our code will be publicly available.

## 1 Introduction

The advent of pre-trained diffusion-based [; ] text-to-image generators [; ; ] has revolutionized the fields of design and filmmaking, opening new vistas for creative expression. These advancements, underpinned by seminal works in text-to-image synthesis, have paved the way for innovative text-guided editing techniques for both images [; ; ; ] and videos [; ; ; ; ; ]. Such techniques not only enhance creative workflows but also promise to redefine content creation within these industries.

Video editing, in contrast to image editing, introduces the intricate challenge of ensuring frame-wise consistency. Efforts to address this challenge have led to the development of methods that leverage shared features and structures with the source video [; ; ; ; ; ; ; ; ] through an inversion-then-generation pipeline [; ], exemplified by Pix2Video’s approach [] to consistent appearance editing across frames. To transfer the edited appearance from the anchor frame to the remaining frames consistently, it employs a pre-trained image generator and extends the self-attention layers to cross-frame attention to generate each remaining frame. Despite these advancements in performing video appearance editing (e.g., stylization, object appearance replacement, etc.), these methodologies fall short in editing video motion (e.g., replacing the movement of playing guitar with waving), hampered by a lack of motion priors and limited control over inter-frame dependencies, underscoring a critical gap in video editing capabilities.

Previous attempts [; ] at video motion editing through fine-tuning a pre-trained generator on the given source video and then editing motion through text guidance. Although effective, they necessitate a delicate balance between the generative prowess of the model and the preservation of the source video’s content. This compromise often leads to restricted motion diversity and unwanted content variations, indicating a pressing need for a more robust solution.

In response, our work aims to explore a tuning-free framework that adeptly navigates the complexities of editing both the motion and appearance of videos. To achieve this, we identify three technical challenges: 1) it is non-trivial to incorporate the text-guided motion into the source content, as directly applying video appearance editing [; ] or image editing [] schemes leads to undesirable results (as shown in Fig. 5); 2) non-edited content preservation of the source video; 3) inheriting the spatial structure of the source video during appearance editing.

Our solution, UniEdit, harnesses the power of a pre-trained text-to-video generator (e.g., LaVie []) within an inversion-then-generation framework [], tailored to overcome the identified challenges. Particularly, we introduce three key innovations: 1) To inject text-guided motion into the source content, we highlight the insight that the temporal self-attention layers of the generator encode the inter-frame dependency. Acting in this way, we introduce an auxiliary motion-reference branch to generate text-guided motion features, which are then injected into the main editing path via temporal self-attention layers. 2) To preserve the non-edited content of the source video, motivated by the image editing technique [], we follow the insight that the spatial self-attention layers of the generator encode the intra-frame dependency. Therefore, we introduce an auxiliary reconstruction branch, and inject the features obtained from the spatial self-attention layers of the reconstruction branch into the main editing path. 3) To retain the spatial structure during the appearance editing, we replace the spatial attention maps of the main editing path with those in the reconstruction branch. These innovations collectively enable UniEdit to redefine the landscape of video editing.

To our best knowledge, UniEdit represents a pioneering leap in text-guided, tuning-free video motion editing. In addition, its unified architecture not only facilitates a wide array of video appearance editing tasks, as shown in Fig. 1, but also empowers image-to-video generators for zero-shot text-image-to-video generation. Through comprehensive experimentation, we demonstrate UniEdit’s superior performance relative to existing state-of-the-art methods, highlighting its potential to significantly advance the field of video editing.

## 2 Related Works

### 2.1 Video Generation

Researchers have achieved video generation with generative adversarial networks [; ; ], language models [; ], or diffusion models [; ; ; ; ; ; ; ; ; ]. To make the generation more controllable, recent endeavors have also incorporated additional structure guidance (e.g., depth map) [; ; ; ; ; ], or conducted customized generation [; ; ; ; ; ]. These models have generally learned real-world video distribution from large-scale data, and achieved promising results on text-to-video or image-to-video generation. Based on their success, we leverage the learned prior in the pre-trained model to achieve tuning-free video motion and appearance editing.

### 2.2 Video Editing

Video editing aims to produce a new video that is aligned with the provided editing instructions (e.g., text) while maintaining the other characteristics of the source video. It can be categorized into appearance and motion editing.

For appearance editing [; ; ; ; ], like turn a video into the style of Van Gogh, the main challenge is to achieve temporal-consistent generation across different frames. Early attempts [; ; ; ; ; ] leveraged text-to-image models with inter-frame propagation to ensure consistency. For instance, Pix2Video [] replaces the key and value of the current frame with those of the first and previous frame. Video-P2P [] achieved local editing via video-specific fine-tuning and unconditional embedding optimization []. Follow-up studies [; ; ] also leveraged the edit-then-propagate framework with neatest-neighbor field [], estimated optical flow [], or temporal deformation field []. Despite the promising results, due to the constraint on the source video structure, these approaches are specialized in appearance editing and can not be applied to motion editing directly.

Recent studies have also explored video motion editing with text guidance [; ], user-provided motion [; ; ], or specific motion representation [; ; ]. For example, Dreamix [] proposed fine-tuning a pre-trained text-to-video model with mixed video-image reconstruction objectives for each source video. Then the editing is realized by conditioning the fine-tuned model on the given target prompt. MoCA [] decoupled the video into the first-frame appearance and the optical flow, and trained a diffusion model to generate video conditioned on the first frame and the text. However, it struggled to preserve the non-edited motion (e.g., background dynamics) as it generates the entire motion from the text.
Different from the aforementioned approaches that require fine-tuning or user-provided motion input, we are the first to achieve tuning-free motion and appearance editing with text guidance only.

## 3 Preliminaries: Video Diffusion Models

Our proposed UniEdit is built upon video diffusion models. Therefore, we first recap the architecture that is used in common text-guided video diffusion models [; ].

### 3.1 Overall Architecture

Modern text-to-video (T2V) diffusion models typically extend a pre-trained text-to-image (T2I) model [] to the video domain with the following adaptations. 1) Introducing additional temporal layers by inflating 2d convolutional layers to 3d form, or adding temporal self-attention layers [] to model the correlation between video frames. 2) Due to the extensive computational resources for modeling spatial-temporal joint distribution, these works typically first train video generation models on low spatial and temporal resolutions, and then upsampling the generated results with cascaded models. 3) Other improvements like efficiency [], training strategy [], or additional control signals [], etc.

During inference, given standard Gaussian distribution $z_{T}\sim\mathcal{N}(0,1)$, the trained denoising UNet is used to perform $T$ denoising steps to obtain the outputs [; ]. If the model is trained in latent space [], a decoder is employed to reconstruct videos from the latent domain.

### 3.2 Attention Mechanisms

In particular, for each block of the denoising UNet, there are four basic modules: a convolutional module, a spatial self-attention module (SA-S), a spatial cross-attention module (CA-S), and a temporal self-attention module (SA-T). Formally, the attention operation [] can be formulated as:

$$ $\texttt{attn}(Q,K,V)=\texttt{softmax}(\frac{QK^{T}}{\sqrt{d}})V,$ (1) $$

where $Q$ (query), $K$ (key), $V$ (value) are derived from the inputs, and $d$ is the dimension of the hidden states.

Intuitively, CA-S is in charge of fusing semantics from the text condition, SA-S models the intra-frame dependency, SA-T models the inter-frame dependency and ensures the generated results are temporally consistent. We leverage these intuitions in our designs as elaborated below.

Figure: Figure 2: Overview of UniEdit. It follows an inversion-then-generation pipeline and consists of a main editing path, an auxiliary reconstruction branch and an auxiliary motion-reference branch. The reconstruction branch produces source features for content preservation, and the motion-reference branch yields text-guided motion features for motion injection. The source features and motion features are injected into the main editing path through spatial self-attention (SA-S) and temporal self-attention (SA-T) modules respectively (Sec. 4.1). We further introduce spatial structure control to retain the coarse structure of the source video (Sec. 4.2).
Refer to caption: extracted/5521627/figs/fig_pipe.jpg

## 4 UniEdit

#### Method Overview.

As shown in Fig. 2, our main editing path is based on an inversion-then-generation pipeline: we use the latent after DDIM inversion [] as the initial noise $z_{T}$(^1^11For real source video, we set source prompt to null during both forward and inversion process to achieve high-quality reconstruction [].), then perform denoising process starting from $z_{T}$ with the pre-trained UNet conditioned on the target prompt $P_{t}$. For motion editing, to achieve source content preservation and motion control, we propose to incorporate an auxiliary reconstruction branch and an auxiliary motion-reference branch to provide desired source and motion features, which are injected into the main editing path to achieve content preservation and motion editing (as shown in Fig. 3). We propose the pipeline of motion editing and appearance editing in Sec. 4.1 & Sec. 4.2 respectively. To further alleviate the background inconsistency, we introduce a mask-guided coordination scheme in Sec. 4.3. We also extend UniEdit to text-image-to-video generation (TI2V) in Sec. 4.4.

### 4.1 Tuning-Free Video Motion Editing

#### Content Preservation on SA-S Modules.

One of the key challenges of editing tasks is to inherit the original content (e.g., textures and background) in the source video. To this end, we introduce an auxiliary reconstruction branch. The reconstruction path starts from the same inversed latent $z_{T}$ similar to the main editing path, and then conducts the denoising process with the pre-trained UNet conditioned on the source prompt $P_{s}$ to reconstruct the original frames. As verified in image editing [; ; ], the attention features in the denoising model during reconstruction contain the content of the source video. Hence, we inject attention features of the reconstruction path into the main editing path on spatial self-attention (SA-S) layers for content preservation. At denoising step $t$, the attention operation of the $l$-th SA-S module in the main editing path is formulated as:

$$ $\text{SA-S}^{l}_{\text{edit}}:=\begin{cases}\texttt{attn}(Q,K,V^{r}),&t>t_{0} \,\text{and}\,l>l_{0}\\ \texttt{attn}(Q,K,V),&\text{otherwise}\end{cases}$ (2) $$

where $Q$, $K$, $V$ are features in the main editing path, $V^{r}$ refer to the value feature of the corresponding SA-S layer in the reconstruction branch, $t_{0}$ and $l_{0}$ are hyper-parameters following previous work []. By replacing the value of spatial features, the video synthesized by the main editing path retains the non-edited characters (e.g., identity and background) of the source video, as exhibited in Fig. 7. Unlike previous video editing works [; ] which introduces a cross-frame attention mechanism (i.e., using the key and value of the first/last frame), we implement Eq. 2 frame-wisely to better tackle source video with large dynamics.

#### Motion Injection on SA-T Modules.

After implementing the content-preserving technique introduced above, we can obtain an edited video with the same content in the source video. However, it is observed that the output video could not follow the text prompt $P_{t}$ properly. A straightforward solution is to increase the value of $t_{0}$ and $l_{0}$ so that balancing between the impact of injected information and the conditioned text prompt. Nevertheless, this could result in a content mismatch with the original source video in terms of structures and textures.

To obtain the desired motion without sacrificing content consistency, we propose to guide the main editing path with reference motion. Concretely, an auxiliary motion-reference branch (which also starts from the inversed latent $z_{T}$) is involved during the denoising process. Different from the reconstruction branch, the motion-reference branch is conditioned on the target prompt $P_{t}$, which contains the description of the desired motion. To transfer the motion into the main editing path, our core insight here is that temporal layers model the inter-frame dependency of the synthesized video clip (as shown in Fig. 6). Motivated by the observations above, we design the attention map injection on temporal self-attention layers of the main editing path:

$$ $\text{SA-T}^{l}_{\text{edit}}:=\begin{cases}\texttt{attn}(Q^{m},K^{m},V),&t>t_ {1}\,\text{and}\,l>l_{1}\\ \texttt{attn}(Q,K,V),\quad&\text{otherwise}\end{cases}$ (3) $$

where $Q^{m}$ and $K^{m}$ refer to the query and key of the motion-reference branch, we simply set $t_{1}$ and $l_{1}$ to zero in practice. It’s observed that the injection of temporal attention maps can effectively facilitate the main editing path to generate motion aligned with the target prompt. To better fuse the motion with the content in the source video, we also implement spatial structure control (refer to Sec. 4.2) on the main editing path and motion-reference branch in the early steps.

Figure: Figure 3: Detailed illustration of the relationship between the main editing path, the auxiliary reconstruction branch and the auxiliary motion-reference branch. The content preservation, motion injection and spatial structure control are achieved by the fusion of $Q$ (query), $K$ (key), $V$ (value) features in spatial self-attention (SA-S) and temporal self-attention (SA-T) modules.
Refer to caption: extracted/5521627/figs/fig_detailed_pipe.jpg

### 4.2 Tuning-Free Video Appearance Editing

In Sec. 4.1, we introduce the pipeline of UniEdit for video motion editing. In this subsection, we aim to perform appearance editing (e.g., style transfer, object replacement, background changing) via the same framework.
In general, there are two main differences between appearance editing and motion editing. Firstly, appearance editing does not require changing the inter-frame relationships. Therefore, we remove the motion-reference branch and corresponding motion injection mechanism from the motion editing pipeline. Secondly, the main challenge of appearance editing is to maintain the structural consistency of the source video. To address this, we introduce spatial structure control between the main editing path and the reconstruction branch.

#### Spatial Structure Control on SA-S Modules.

Previous approaches on video appearance editing [; ] mainly realize spatial structure control with the assistance of additional network []. When the auxiliary control model fails, it may result in inferior performance in preserving the structure of the original video. Alternatively, we suggest extracting the layout information of the source video from the reconstruction branch. Intuitively, the attention maps in spatial self-attention layers encode the structure of the synthesized video, as verified in Fig. 6. Hence, we replace the query and key of SA-S module in the main editing path with those in the reconstruction branch:

$$ $\text{SA-S}^{l}_{\text{edit}}:=\begin{cases}\texttt{attn}(Q^{r},K^{r},V),&t<t_ {2}\,\text{and}\,l>l_{2},\\ \texttt{attn}(Q,K,V),\;&\text{otherwise},\end{cases}$ (4) $$

where $Q^{r}$ and $K^{r}$ refer to the query and key of the reconstruction branch, $t_{2}$ and $l_{2}$ are used to control the extent of editing. It is worth mentioning that the effect of spatial structure control is distinct from the content preservation mechanism in Sec. 4.1. Take stylization as an example, the proposed structure control in Eq. 4 only ensures consistency in terms of each frame’s composition, while enabling the model to generate the required textures and styles based on the text prompt. On the other hand, the content preservation technique inherits the textures and style of the source video. Therefore, we use structure control instead of content preservation for appearance editing.

In addition, the structure control technique also facilitates the fusion of source content and targeted motion in motion editing.
During motion editing, the structure control is only implemented at earlier denoising steps, and will not affect the model to generate the desired motion that matches the target prompt.

### 4.3 Mask-Guided Coordination

To further improve background consistency, we suggest leveraging the foreground/background segmentation mask $M$ to guide the denoising process [; ]. There are two possible ways to obtain the mask $M$: the attention maps of CA-S modules with a threshold []; or employing an off-the-shelf segmentation model [] on the source and generated videos.
The obtained segmentation masks can be leveraged to 1), alleviate the indistinction in foreground and background; 2), improve the consistency of content between edited and source videos.

To this end, we leverage mask-guided self-attention in the main editing path to coordinate the editing path with the motion-reference branch. Formally, we define:

$$ $\texttt{m-attn}(Q,K,V;M)=\texttt{softmax}(\frac{QK^{T}}{\sqrt{d}}+M)V.$ (5) $$

Then the mask-guided self-attention:

$$ $\displaystyle\text{SA}_{\text{mask}}:=$ $\displaystyle\texttt{m-attn}(Q,K,V;M^{f})\odot M_{m}$ (6) $\displaystyle+$ $\displaystyle\texttt{m-attn}(Q,K,V;M^{b})\odot(1-M_{m}),$ $$

where $M^{f},M^{b}\in\{-\infty,0\}$ indicate the foreground and background masks in the editing path respectively, $M_{m}\in\{0,1\}$ denotes the foreground mask from the motion-reference branch, and $\odot$ is Hadamard product.

In addition, we leverage the mask during the content preservation and motion injection for the features obtained from the reconstruction branch and the motion-reference branch (e.g., we replace $Q^{m}$ with $M_{m}\odot Q^{m}+(1-M_{m})\odot Q$).

Figure: Figure 4: Examples edited by UniEdit. For each case, the upper frames come from the source video, and the lower frames indicate the edited results with the target prompt. We encourage the readers to watch the [videos](https://jianhongbai.github.io/UniEdit/) and make evaluations.
Refer to caption: extracted/5521627/figs/fig_ours_in_exp.jpg

### 4.4 T2V Models are Zero-Shot TI2V Generators

To make our framework more flexible, we further derive a method to incorporate images as input and synthesize high-quality video conditioned on both image and text-prompt.
Different from some image animation techniques [], our method allows the user to guide the animation process with text prompts.
Concretely, we first achieve image-to-video (I2V) generation by: 1) transforming input images with simulated camera movement to form a pseudo-video clip [] or 2) leveraging existing image animation approaches (e.g., SVD [], AnimateDiff []) to synthesis a video with random motion (which may not consistent with the text prompt). Then, we perform text-guided editing with UniEdit on the vanilla video to obtain the final output video.

## 5 Experiments

Figure: Figure 5: Comparison with state-of-the-art methods for both video motion and appearance editing. It shows that UniEdit achieves better source content preservation, and outperforms baselines in motion editing by a large margin.
Refer to caption: extracted/5521627/figs/fig_compare_in_exp.jpg

### 5.1 Implementation Details

UniEdit is not limited to specific video diffusion models. In this section, we build UniEdit upon LaVie [] as an instantiation to verify the effectiveness of our method.
For each input video, we follow the pre-processing step in LaVie to the resolution of $320\times 512$. Then, the pre-processed video is fed into the UniEdit to perform video editing. It takes 1-2 minutes to edit on an NVIDIA A100 GPU for each video. More details can be found in Appendix A.

### 5.2 Comparison with State-of-the-Art Methods

#### Baselines.

To evaluate the performance of UniEdit, we compare the editing results of UniEdit with state-of-the-art motion and appearance editing approaches. For motion editing, due to the lack of open-source tuning-free (zero-shot) methods, we adapt the state-of-the-art non-rigid image editing technique MasaCtrl [] to a T2V model [] (denoted as MasaCtrl${}^{*}$ in Fig. 5) and a one-shot video editing method Tune-A-Video (TAV) [] as strong baselines. For appearance editing, we use the latest methods with strong performance, including FateZero [], TokenFlow [], and Rerender-A-Video (Rerender) [] as baselines.

#### Qualitative Results.

We give editing examples of UniEdit in Fig. 1, Fig. 4. (more examples in Fig. 12-17 of Appendix B). We recommend seeing more cases and videos on our [project page](https://jianhongbai.github.io/UniEdit/). It is observed that UniEdit can: 1) edit in different scenarios, including motion-changing, object replacement, style transfer, background modification, etc; 2) align with the target prompt; 3) demonstrate great temporal consistency.

Furthermore, we make comparisons with state-of-the-art methods in Fig. 5 (more comparisons in Fig. 10,11 of Appendix B). For appearance editing, i.e., turning the source video into an oil painting style, UniEdit outperforms baselines on content preservation. For example, the grassland still maintains its original appearance without any additional stones or paths. For motion editing, most baseline methods fail to output video that is aligned with the target prompt, or fail to preserve the source content.

**Table 1: Quantitative comparison (CLIP Score and User Preference) with state-of-the-art video editing techniques.**
| Method | Frame Consistency | Textual Alignment |  |  |
| --- | --- | --- | --- | --- |
| CLIP Score | User Pref. | CLIP Score | User Pref. |  |
| TAV [] | 96.88 | 3.86 | 32.93 | 3.34 |
| MasaCtrl${}^{*}$ [] | 96.52 | 4.32 | 31.93 | 3.20 |
| FateZero [] | 97.56 | 4.53 | 30.67 | 3.45 |
| Rerender [] | 97.17 | 4.18 | 32.18 | 3.56 |
| TokenFlow [] | 97.38 | 4.56 | 31.33 | 3.47 |
| UniEdit (Ours) | 98.37 | 4.74 | 36.29 | 4.88 |

#### Quantitative Results.

We quantitatively verify the effectiveness of our method from two aspects: temporal consistency and alignment with the target prompt. Following previous work [], we use the CLIP [] to calculate the score of Frame Consistency and Textual Alignment. We also conducted a user study between UniEdit and the baselines by asking $10$ participants to rate the edited videos at five grades ($1$-$5$). As shown in Tab. 1, UniEdit outperforms baseline methods by a large margin.

### 5.3 Ablation Study and Analysis

Figure: Figure 6: Visualization of spatial query in SA-S (second row), cross-frame temporal attention maps in SA-T (third row), and the magnitude of optical flow (fourth row).
Refer to caption: extracted/5521627/figs/fig_visualize.jpg

Figure: Figure 7: The proposed content preservation and motion injection both contribute to the final results.
Refer to caption: extracted/5521627/figs/fig_ablation1.jpg

Figure: Figure 8: Ablation on spatial structure control and controllable video editing.
Refer to caption: extracted/5521627/figs/fig_ablation2.jpg

#### How UniEdit Works?

To better understand how UniEdit works and reveal our insight on the spatial and temporal self-attention layers, we visualize the features in SA-S and SA-T modules and compare them with the magnitude of optical flow between adjacent frames in Fig. 6. It can be illustrated that, compared to the spatial query maps (second row), the temporal cross-frame attention maps (third row) show a significantly larger overlap with the optical flow (fourth row). Therefore, we leverage that the temporal self-attention layers encode the inter-frame dependency and conduct motion injection in temporal self-attention layers. We also conduct content preservation and structure control in spatial self-attention layers.

#### The Two Auxiliary Branches Both Contribute to the Final Results.

In UniEdit, we propose to preserve the content from the reconstruction branch and transfer motion from the motion-reference branch via spatial and temporal self-attention respectively. To better understand the impact of the two auxiliary branches, we visualize the edited results of utilizing each branch individually in Fig. 7. When performing the content preservation from the reconstruction branch, though the identity and background are well-preserved, the synthesized frames deviate from the target prompt. On the other hand, only implementing the motion injection results in dramatic changes in the background. Meanwhile, UniEdit achieves satisfying performance in both content preservation and adherence to text.

#### Spatial Structure Control Retains the Source Structure during the Appearance Editing.

We conduct the ablation study on spatial structure control in the upper two rows of Fig. 8, which demonstrates that the source structure is lost without the used structure control scheme, while our results (the lower three rows) adhere to the source structure.

#### Controllable Video Editing.

Our UniEdit supports controllable video editing by injecting the features into the main editing path at different layers and denoising steps. As exemplified in Fig. 8, compared with injection at more steps ($t_{2}$=50), injection at fewer ($t_{2}$=15) steps produces a more stylized output, enabling flexible video editing.

## 6 Conclusion and Limitations

In this paper, we design a novel tuning-free framework UniEdit for both video motion and appearance editing. By leveraging an auxiliary motion-reference branch and an auxiliary reconstruction branch and injecting features into the main editing path, it is capable of performing motion editing and various appearance editing.
There are nevertheless some limitations. Firstly, though editing on both motion and appearance can be achieved by calling UniEdit twice, it remains to be explored how to perform both types of editing simultaneously. Secondly, since we have multiple hyper-parameters, it is worth investigating the automatic determination scheme. We hope our work can promote the research toward unified content editing.

#### Ethical Consideration.

UniEdit is a tuning-free approach and is developed for research purposes only. It is suggested to add watermarks to prevent misuse.

## Appendix A Detailed Experimental Settings

#### Base T2V Model.

We instantiate the proposed method on LaVie [], which is a pre-trained text-to-video generation model that produces consistent and high-quality videos. To achieve a fair comparison, we only leverage the base T2V model in [LaVie](https://github.com/Vchitect/LaVie) and load the open-source pre-trained weights for video editing tasks in the experiments. Note that the edited video clip could further be seamlessly fed into the temporal interpolation model and the video super-resolution model to obtain video with a longer duration and higher resolution.

#### Video Preprocessing.

For each input video, we resize it to the resolution of $320\times 512$, followed by normalization, which is consistent with the training configuration of LaVie. Then, the pre-processed video is fed into the base model of Lavie to perform video editing. To maximize the generation power of LaVie, we set all input videos to 16 frames. For a source video, it takes 1-2 minutes to edit on an NVIDIA A100 GPU.

#### Configurations.

For real source videos, we inverse them with $50$ DDIM inversion steps and perform DDIM deterministic sampling with $50$ steps for generation.
For the generated videos, we use the same start latent of synthesizing the source video as the initial noise $z_{T}$ for the main editing path and two auxiliary branches. We use the commonly used classifier-free guidance technique [] with a scale of $7.5$.

#### Details of User Study.

As a text-guided editing task, in addition to CLIP scores, it is crucial to evaluate results through human subjective assessment. To achieve this, we utilized MOS (Mean Opinion Score) as our metric and collected feedback from $10$ experienced volunteers. We randomly selected $20$ editing samples and permuted results from different models. Volunteers were then tasked to evaluate the results based on two perspectives: frame consistency and textual alignment. They provided ratings for these aspects on a scale of $1$-$5$. Specifically, frame consistency measures the smoothness of the video, aiming to avoid dramatic jittering and ensure coherence between the content of each frame. Textual alignment assesses whether the editing results adhere to the text guidance and maintain the content of the source video. In the end, we computed the average user ratings for each method as our final results.

As illustrated in Tab. 1, UniEdit shows the best performance on frame consistency. Regarding textual alignment, UniEdit significantly outperforms all other baselines, demonstrating its capacity to support diverse editing scenarios.

#### Baselines.

We implement all baseline methods with their official repositories. For MasaCtrl [], we adapt it to video editing by first setting the base model to a T2V model [], then performing MasaCtrl on all frames of the source video. Moreover, since most baselines use StableDiffusion (SD) as the base model, we resize the source video to $512\times 512$ to align with the default configuration of SD, then feed it into the denoising model, which can maximize the power of SD.

## Appendix B Additional Experimental Results and Analysis

### B.1 More Ablation Study

Please refer to Fig. 9 for the ablation on mask-guided coordination, where the second line indicates the results without using mask-guided coordination and the last line is UniEdit results with the proposed mask-guided coordination. It’s observed that mask-guided coordination could further improve the editing performance.

Figure: Figure 9: Ablation on mask-guided coordination.
Refer to caption: extracted/5521627/figs/w_wo_mask.jpg

### B.2 More Comparison with State-of-the-Art Methods

Please refer to Fig. 10 and Fig. 11 for more comparison with the state-of-the-art methods.

Figure: Figure 10: More comparison with state-of-the-art methods.
Refer to caption: extracted/5521627/figs/compare1.jpg

Figure: Figure 11: More comparison with state-of-the-art methods.
Refer to caption: extracted/5521627/figs/compare2.jpg

### B.3 More Results of UniEdit

More edited results of UniEdit are provided in Fig. 12-17. Examples of TI2V generation are provided in Fig. 18.

Figure: Figure 12: More appearance editing results of UniEdit.
Refer to caption: extracted/5521627/figs/ours_appear1.jpg

Figure: Figure 13: More appearance editing results of UniEdit.
Refer to caption: extracted/5521627/figs/ours_appear2.jpg

Figure: Figure 14: More appearance editing results of UniEdit.
Refer to caption: extracted/5521627/figs/ours_appear3.jpg

Figure: Figure 15: More appearance editing results of UniEdit.
Refer to caption: extracted/5521627/figs/ours_appear4.jpg

Figure: Figure 16: More motion editing results of UniEdit.
Refer to caption: extracted/5521627/figs/ours_motion1.jpg

Figure: Figure 17: More motion editing results of UniEdit.
Refer to caption: extracted/5521627/figs/ours_motion2.jpg

Figure: Figure 18: Results of text-image-to-video synthesis in Sec. 4.4.
Refer to caption: extracted/5521627/figs/ours_ti2v.jpg

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Please check if it is reasonable to have conducted the experiment with 10 subjects.
soundness: 3
strengths: The method can be applied to various Text-to-Video models, clearly stating the intended goal of simultaneously editing the content's motion and appearance.
The paper presents results applied to multiple Text-to-Video models. Experiments using various prompts demonstrate that this method is not limited to specific prompts.
The video quality available on the project page is excellent, and the experimental results are easy to understand.
Videos generated using UniEdit outperformed existing methods in terms of CLIP score, Frame Quality, and Temporal Quality.
summary: The paper proposes a method to simultaneously edit motion and appearance through text guidance. Unlike previous approaches that required fine-tuning a pretrained generator model for the target video, the proposed UniEdit method is tuning-free. It utilizes the temporal self-attention layer within the generator to naturally express motion. Additionally, the spatial self-attention layer preserves the regions that are not being edited. Furthermore, the method ensures that the spatial structure of the video remains intact during appearance editing.
weaknesses: The user study involved only 10 participants, which may lead to highly subjective results. Including subjective results in the main performance table could confuse readers, and it would be beneficial to either evaluate with more participants or clearly indicate near the table that the results may be subjective due to the limited number of participants.
The subjective evaluation from only 10 participants being included in the quantitative results could cause confusion among readers.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses section for my questions. I may revise my rating based on the authors' rebuttal.
soundness: 3
strengths: - **Unified Framework for Motion and Appearance Editing**: UniEdit effectively addresses a key challenge in video editing by providing a single framework that can handle both motion and appearance modifications. This is achieved by leveraging the inherent properties of spatial and temporal self-attention layers in the T2V model. The motion-reference branch, conditioned on the target prompt, injects motion features into the temporal self-attention layers, enabling text-guided motion changes. The reconstruction branch focuses on preserving source content by injecting features into the spatial self-attention layers.

- **Zero-Shot Editing Capability**: UniEdit operates in a tuning-free manner, meaning it doesn't require fine-tuning the base T2V model for each new video or editing task. This makes it highly efficient and user-friendly, as it avoids the computational cost and complexity associated with model training.

- **Emphasis on Temporal Consistency:** Maintaining temporal consistency is crucial for high-quality video editing. UniEdit tackles this challenge by injecting features from the reconstruction branch into the spatial self-attention layers, which helps preserve the content and structure of the source video. The use of mask-guided coordination further enhances the consistency of unedited areas.
summary: UniEdit presents a novel approach to text-guided video editing, utilizing a tuning-free framework that leverages the capabilities of pre-trained text-to-video (T2V) diffusion models. The key strength of UniEdit lies in its ability to perform both motion and appearance editing within a unified architecture. This is a significant advancement compared to prior methods, which often focus on either motion or appearance editing, but not both. The core idea behind UniEdit is to invert the source video into the latent space of a pre-trained T2V model, and then guide the generation process using the target text prompt and auxiliary branches that control motion and appearance.
weaknesses: 1. The problem is challenging, and I believe this paper only partially addresses it. The editing capabilities need to be more clearly defined. For example, are all text prompts representing motion control or appearance control feasible? The scope requires further clarification. The inherent ambiguity of language and the complexity of video editing tasks make it practically impossible to guarantee success for every possible textual input.

**Defining Boundaries and Constraints:** To enhance clarity regarding its scope, the UniEdit paper should clearly state:

- **Types of Edits Supported:** Specify the categories of motion and appearance edits that UniEdit is designed to handle effectively.

- **Limitations in Prompt Complexity:** Provide guidelines or examples of text prompts that might pose challenges for UniEdit, particularly those involving complex scenes or multiple simultaneous edits.

- **Known Failure Cases:** Discuss specific scenarios where UniEdit is known to perform poorly, offering insights into the reasons for these limitations.

2. UniEdit's core techniques are indeed inspired by existing approaches:
   - **Auxiliary Branches:** The concept of using auxiliary branches for specific tasks, such as reconstruction or motion guidance, is not entirely new.
   - **Feature Injection via Self-Attention:** Injecting features into specific layers of a neural network, especially within attention mechanisms, is a common practice in various image and video editing tasks.
   - **Temporal and Spatial Self-Attention:** The observation that temporal self-attention layers encode inter-frame dependencies, while spatial self-attention layers capture intra-frame information, is a widely recognized principle reported in other works like I2VEdit (https://i2vedit.github.io/) for diffusion-based methods and SIFA (https://github.com/FuchenUSTC/SIFA) for other video editing tasks.

   While UniEdit's core techniques are not entirely novel, **the integration and adaptation of these mechanisms for tuning-free video motion editing demonstrate a degree of innovation.**

3. The improvements, while statistically significant, do not represent a leap in performance but rather a refinement of existing techniques. For instance, UniEdit's performance on Frame Quality and Temporal Quality metrics surpasses competing methods like TokenFlow and Rerender, but the margins are relatively modest (e.g., a 1-2% improvement in some metrics).

4. As the authors have pointed out, UniEdit has a **performance ceiling** when performing both motion and appearance editing simultaneously and struggles with complex scenes.

5. **Missing Critical Baseline:** ReVideo (https://mc-e.github.io/project/ReVideo/) should be included as a baseline. ReVideo's focus on remaking videos with control over both motion and content aligns well with UniEdit's capabilities. A direct comparison would reveal their strengths and weaknesses in tasks such as object replacement, style transfer, and motion modification.

   Including additional works for comparison will further strengthen this paper. 

  In addition, a discussion with the following related works is beneficial:
   - **AnyV2V:** Its compatibility with various image editing methods and emphasis on simplicity and consistency present an interesting comparison point for UniEdit.
   - **I2VEdit:** Comparing with I2VEdit would reveal the advantages and disadvantages of using an edited first frame versus direct text guidance for video editing. This comparison could be particularly insightful for tasks involving local edits, global style transfer, identity manipulation, and subject customization. The temporal consistency of I2VEdit appears to be quite robust.
   - **FLATTEN:** Comparing FLATTEN to UniEdit would showcase the differences between explicitly incorporating optical flow information and UniEdit's approach to motion injection.

6. **Metric Coverage:** The evaluation relies heavily on CLIP scores and user preferences. While these metrics are valuable, the absence of more comprehensive metrics (e.g., temporal coherence measures beyond flickering, user engagement metrics) limits the robustness of the evaluation.
   - **Flow Warping Error (Ewarp):** This metric can be used to assess visual consistency by comparing the edited video frames warped according to the estimated optical flow of the source video.
   - **Frame Variance Metric for Motion Detection (FVMD):** This may provide a more objective measure of how well the model preserves motion consistency across frames.

7. The evaluation dataset could include BalanceCC if possible.

8. **Sensitivity Analysis** can be more detailed:
   - **Sensitivity to Injection Steps:** The quantitative analysis clearly shows that the choice of injection steps influences the quality and consistency of the output.
   - **Sensitivity to Blend Layers and Steps:** The qualitative exploration indicates a trade-off between stylization and fidelity controlled by the blend layers and steps. This suggests a need for adjustment based on the desired outcome.

9. The related works are primarily categorized by broad topics (e.g., video generation, video editing) rather than by the specific innovations relevant to UniEdit (e.g., tuning-free frameworks, self-attention manipulation). This top-down organization makes it harder to discern how UniEdit differentiates itself from existing methods.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - [Ablation study] Why does the *content preservation* improve *textual alignment* a lot in Table 2?
- I believe different methods have their own "success rate" in generation. Is there a way to measure this rate for fair comparison, beyond just showcasing several visual examples? Additionally, how to know if this tuning-free method really works or not (to what degree), compared to potential methods with large-scale training?
soundness: 4
strengths: - The paper demonstrates good clarity in writing, making it easy for readers to follow the flow of ideas.
- The paper is well-motivated, offering various types of video editing while preserving non-target properties using a unified framework.
- The design of attention replacement for appearance, structure, and motion editing is technically sound, leveraging the insights and functionality of spatial and temporal self-attention, respectively.
- The paper and the auxiliary material show impressive editing results produced with their framework with substantial improvement on existing baselines, especially in temporal coherence.
summary: The authors propose UniEdit, addressing the task of video motion and appearance editing using pre-trained video diffusion models (VDMs) in a united and tuning-free manner. Specifically, they apply the common "inversion-and-generation" with two innovative auxiliary branches, i.e., the reconstruction branch for content preservation and the motion-reference branch for motion editing, in addition to the main editing path. The appearance editing, motion editing and structure preservation are realized by swapping attention features (QKV) in specific types and depth of attention layers, and denoising steps as well. UniEdit is compared with previous zero-shot/tuning-free methods and outperforms them both quantitatively and qualitatively.
weaknesses: - The authors claim in L217 that "better tackle source video with large dynamics". However, most visual examples are with small motion magnitudes. Does the model struggle with appearance and motion editing in videos with larger dynamics? Temporal inconsistencies are noticeable in videos with large motion (see Fig G.1).
- The authors claim in L312 that "UniEdit is not limited to specific video diffusion models", which is inaccurate. The proposed framework depends on the separate spatial and temporal attention design of video diffusion models, which is incompatible with full 3D attention DiT-based video generators. And 3D full attention is increasingly being proven to be more effective [1].
- This work still follows the common "inversion-and-genertation" paradigm, potentially performing better on generated videos but struggling with out-of-domain real videos.
- In Fig.4 (bottom-right case), motion-editing results show background changes, such as alterations in the lawn.
- The user study involved an insufficient number of participants (only 10), making it difficult to support the conclusions effectively with robust experimental results.

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer, Yang et al., 2024

Minor concerns:
- [Clarity] Consider including a note below the table or in the caption indicating that higher values are better.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses
soundness: 2
strengths: In this work, they present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversionthen-generation framework. To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, they introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers.
summary: Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, They present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversionthen-generation framework. However, this paper seems not latest version. Some new sota approaches are not included and had a comparison.
weaknesses: - Novelty.  Using ReferenceNet is not a novel approach in video generation. What is the difference between yours and previous work.
- Overclaim.  "UniEdit represents a pioneering leap in text-guided, tuning-free video motion editing."  Why this work is a pioneering leap? 
- Lack of comparison. Many new works, such as Revideo, COVE , accepted by NeurIPS 2024, are all not included in this paper. 
- Long video editing,  All results are only 16 frames，which is too short.
- Motion Editing. It seems that all motion are similar. How to editing it ?
- Inconsistency.  Such as cat -> dog. The consistency of the head turn is problematic and exhibits sudden changes.

Meta-review: The paper presents a tuning-free framework for video motion and appearance editing with a pre-trained text-to-video base model. A primary objective of the editing process is to change the motion while preserving source video content. This goal is accomplished by adding auxiliary branches for motion reference and reconstruction, which produce text-guided motion and source features. These features are then integrated into the temporal and spatial self-attention layers to control inter-frame and intra-frame dependencies.

The paper initially received mixed and somewhat negative ratings: 6, 6, 5, and 3. After the rebuttal, the ratings became more varied with scores of 8, 5, 5, and 1. Despite the authors' efforts to address the reviewers' comments and provide additional results, three of the reviewers continued to express concerns regarding the novelty and technical contributions of the work. The area chair reviewed the paper and found the reviewers' arguments somewhat convincing. Consequently, the area chair did not identify strong reasons to overrule the reviewers' decision and regretfully recommended rejecting the paper.

Decision: Reject