UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 07:38:06

Abstract: Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, we present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversion-then-generation framework.
To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, we introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers. Extensive experiments demonstrate that UniEdit covers video motion editing and various appearance editing scenarios, and surpasses the state-of-the-art methods.

Main paper: 
 introduction: 
 The advent of pre-trained diffusion-based [26, 53] text-to-image generators [49, 50, 48] has revolutionized the fields of design and filmmaking, opening new vistas for creative expression. These advancements, underpinned by seminal works in text-to-image synthesis, have paved the way for innovative text-guided editing techniques for both images [42, 24, 4, 5] and videos [65, 6, 39, 70, 17, 46] . Such techniques not only enhance creative workflows but also promise to redefine content creation within these industries.Video editing, in contrast to image editing, introduces the intricate challenge of ensuring frame-wise consistency. Efforts to address this challenge have led to the development of methods that leverage shared features and structures with the source video [6, 39, 37, 70, 46, 7, 33, 62, 18] through an inversion-then-generation pipeline [42, 53] , exemplified by Pix2Video's approach [6] to consistent appearance editing across frames. To transfer the edited appearance from the anchor frame to the remaining frames consistently, it employs a pre-trained image generator and extends the self-attention layers to cross-frame attention to generate each remaining frame. Despite these advancements in performing video appearance editing (e.g., stylization, object appearance replacement, etc.), these methodologies fall short in editing video motion (e.g., replacing the movement of playing guitar with waving), hampered by a lack of motion priors and limited control over inter-frame dependencies, underscoring a critical gap in video editing capabilities.Previous attempts [65, 44] at video motion editing through fine-tuning a pre-trained generator on the given source video and then editing motion through text guidance. Although effective, they necessitate a delicate balance between the generative prowess of the model and the preservation of the source video's content. This compromise often leads to restricted motion diversity and unwanted content variations, indicating a pressing need for a more robust solution.In response, our work aims to explore a tuning-free framework that adeptly navigates the complexities of editing both the motion and appearance of videos. To achieve this, we identify three technical challenges: 1) it is non-trivial to incorporate the text-guided motion into the source content, as directly applying video appearance editing [46, 18] or image editing [5] schemes leads to undesirable results (as shown in Fig. 5 ); 2) preserving the non-edited content of the source video; 3) inheriting the spatial structure of the source video during appearance editing.Our solution, UniEdit, harnesses the power of a pre-trained text-to-video generator (e.g., LaVie [63] ) within an inversion-then-generation framework [42] , tailored to overcome the identified challenges.Particularly, we introduce three key innovations: 1) To inject text-guided motion into the source content, we highlight the insight that the temporal self-attention layers of the generator encode the inter-frame dependency. Acting in this way, we introduce an auxiliary motion-reference branch to generate text-guided motion features, which are then injected into the main editing path via temporal self-attention layers. 2) To preserve the non-edited content of the source video, motivated by the image editing technique [5] , we follow the insight that the spatial self-attention layers of the generator encode the intra-frame dependency. Therefore, we introduce an auxiliary reconstruction branch, and inject the features obtained from the spatial self-attention layers of the reconstruction branch into the main editing path. 3) To retain the spatial structure during the appearance editing, we replace the spatial attention maps of the main editing path with those in the reconstruction branch.To our best knowledge, UniEdit represents a pioneering leap in text-guided, tuning-free video motion editing. In addition, its unified architecture not only facilitates a wide array of video appearance editing tasks, as shown in Fig. 1 , but also empowers image-to-video generators for zero-shot text-image-to-video generation. Through comprehensive experimentation, we demonstrate UniEdit's superior performance relative to existing state-of-the-art methods, highlighting its potential to significantly advance the field of video editing.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In this work, we present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained textto-video generator within an inversion-then-generation framework.Extensive experiments demonstrate that UniEdit covers video motion editing and various appearance editing scenarios, and surpasses the state-of-the-art method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
video generation: 
 Researchers have achieved video generation with generative adversarial networks [58, 51, 61] , language models [69, 71] , or diffusion models [28, 52, 25, 23, 3, 60, 72, 19, 63, 8, 47] . To make the generation more controllable, recent endeavors have also incorporated additional structure guidance (e.g., depth map) [16, 10, 74, 11, 20, 64] , or conducted customized generation [65, 67, 34, 75, 59, 41] .These models have generally learned real-world video distribution from large-scale data, and achieved promising results on text-to-video or image-to-video generation. Based on their success, we leverage the learned prior in the pre-trained model to achieve tuning-free video motion and appearance editing.Video editing aims to produce a new video that is aligned with the provided editing instructions (e.g., text) while maintaining the other characteristics of the source video. It can be categorized into appearance and motion editing.For appearance editing [70, 15, 17, 35, 12] , like turn a video into the style of Van Gogh, the main challenge is to achieve temporal-consistent generation across different frames. Early attempts [6, 37, 46, 7, 33, 62 ] leveraged text-to-image models with inter-frame propagation to ensure consistency.For instance, Pix2Video [6] replaces the key and value of the current frame with those of the first and previous frame. Video-P2P [39] achieved local editing via video-specific fine-tuning and unconditional embedding optimization [43] . Follow-up studies [18, 70, 45 ] also leveraged the editthen-propagate framework with neatest-neighbor field [18] , estimated optical flow [70] , or temporal deformation field [45] . Despite the promising results, due to the constraint on the source video structure, these approaches are specialized in appearance editing and can not be applied to motion editing directly.Recent studies have also explored video motion editing with text guidance [65, 44] , user-provided motion [32, 54, 15] , or specific motion representation [55, 36, 22] . For example, Dreamix [44] proposed fine-tuning a pre-trained text-to-video model with mixed video-image reconstruction objectives for each source video. Then the editing is realized by conditioning the fine-tuned model on the given target prompt. MoCA [68] decoupled the video into the first-frame appearance and the optical flow, and trained a diffusion model to generate video conditioned on the first frame and the text. However, it struggled to preserve the non-edited motion (e.g., background dynamics) as it generates the entire motion from the text. Different from the aforementioned approaches that require fine-tuning or user-provided motion input, we are the first to achieve tuning-free motion and appearance editing with text guidance only.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We discussed the potential limitations of the method in Sec. 6 and presented failed cases in Appendix B.6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: video diffusion models: 
 Our proposed UniEdit is built upon video diffusion models. Therefore, we first recap the architecture that is used in common text-guided video diffusion models [63, 2] .Overall Architecture Modern text-to-video (T2V) diffusion models typically extend a pre-trained text-to-image (T2I) model [49] to the video domain with the following adaptations. 1) Introducing additional temporal layers by inflating 2d convolutional layers to 3d form, or adding temporal self-attention layers [57] to model the correlation between video frames. 2) Due to the extensive computational resources for modeling spatial-temporal joint distribution, these works typically first train video generation models on low spatial and temporal resolutions, and then upsampling the generated results with cascaded models. 3) Other improvements like efficiency [1] , training strategy [19] , or additional control signals [16] , etc. During inference, given standard Gaussian distribution z T ∼ N (0, 1), the denoising UNet is used to perform T denoising steps to obtain the outputs [26, 53] . If the model is trained in latent space [49] , a decoder is employed to reconstruct videos from the latent domain.In particular, for each block of the denoising UNet, there are four basic modules: a convolutional module, a spatial self-attention module (SA-S), a spatial cross-attention module (CA-S), and a temporal self-attention module (SA-T). Formally, the attention operation [57] can be formulated as:EQUATIONwhere Q (query), K (key), V (value) are derived from inputs, and d is the dimension of hidden states.Intuitively, CA-S is in charge of fusing semantics from the text condition, SA-S models the intraframe dependency, SA-T models the inter-frame dependency and ensures the generated results are temporally consistent. We leverage these intuitions in our designs as elaborated below. some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
uniedit: 
 Method Overview. As shown in Fig. 2 , our main editing path is based on an inversion-thengeneration pipeline: we use the latent after DDIM inversion [53] as the initial noise z T1 , then perform denoising process starting from z T with the pre-trained UNet conditioned on the target prompt P t . For motion editing, to achieve source content preservation and motion control, we propose to incorporate an auxiliary reconstruction branch and an auxiliary motion-reference branch to provide desired source and motion features, which are injected into the main editing path to achieve content preservation and motion editing (as shown in Fig. 3 ). We propose the pipeline of motion editing and appearance editing in Sec. 4.1 & Sec. 4.2 respectively. To further alleviate the background inconsistency, we introduce a mask-guided coordination scheme in Sec. 4.3. We also extend UniEdit to text-image-to-video generation (TI2V) in Sec. 4.4.Content Preservation on SA-S Modules. One of the key challenges of editing tasks is to inherit the original content (e.g., textures and background) in the source video. To this end, we introduce an auxiliary reconstruction branch. The reconstruction path starts from the same inversed latent z T similar to the main editing path, and then conducts the denoising process with the pre-trained UNet conditioned on the source prompt P s to reconstruct the original frames. As verified in image editing [56, 24, 5] , the attention features in the denoising model during reconstruction contain the content of the source video. Hence, we inject attention features of the reconstruction path into the main editing path on spatial self-attention (SA-S) layers for content preservation. At denoising step t, the attention operation of the l-th SA-S module in the main editing path is formulated as:EQUATIONwhere Q, K, V are features in the main editing path, V r refer to the value feature of the corresponding SA-S layer in the reconstruction branch, t 0 = 50 and L = 10 are hyper-parameters following previous work [5] . By replacing the value of spatial features, the video synthesized by the main editing path retains the non-edited characters (e.g., identity and background) of the source video, as exhibited in Fig. 7a . Unlike previous video editing works [37, 29] which introduces a cross-frame attention mechanism (i.e., using the key and value of the first/last frame), we implement Eq. 2 frame-wisely to better tackle source video with large dynamics.Motion Injection on SA-T Modules. After implementing the content-preserving technique introduced above, we can obtain an edited video with the same content in the source video. However, it is observed that the output video could not follow the text prompt P t properly. A straightforward solution is to increase the value of L so that balancing between the impact of injected information and the conditioned text prompt. Nevertheless, this could result in a content mismatch with the original source video in terms of structures and textures.To obtain the desired motion without sacrificing content consistency, we propose to guide the main editing path with reference motion. Concretely, an auxiliary motion-reference branch (which also starts from the inversed latent z T ) is involved during the denoising process. Different from the reconstruction branch, the motion-reference branch is conditioned on the target prompt P t , which contains the description of the desired motion. To transfer the motion into the main editing path, our core insight here is that temporal layers model the inter-frame dependency of the synthesized video clip (as shown in Fig. 6 ). Motivated by the observations above, we design the attention map injection on temporal self-attention layers of the main editing path:EQUATIONwhere Q m and K m refer to the query and key of the motion-reference branch, note that we replace the query and key of SA-T modules in the main editing path with those in the motion-reference branch on all layers and denoising steps. It's observed that the injection of temporal attention maps can effectively facilitate the main editing path to generate motion aligned with the target prompt.To better fuse the motion with the content in the source video, we also implement spatial structure control (refer to Sec. 4.2) on the main editing path and motion-reference branch in the early steps. Previous approaches on video appearance editing [70, 18] mainly realize spatial structure control with the assistance of additional network [73] . When the auxiliary control model fails, it may result in inferior performance in preserving the structure of the original video. Alternatively, we suggest extracting the layout information of the source video from the reconstruction branch. Intuitively, the attention maps in spatial self-attention layers encode the structure of the synthesized video, as verified in Fig. 6 . Hence, we replace the query and key of SA-S module in the main editing path with those in the reconstruction branch:EQUATIONwhere Q r and K r refer to the query and key of the reconstruction branch, t 1 is used to control the extent of editing. It is worth mentioning that the effect of spatial structure control is distinct from the content preservation mechanism in Sec. 4.1. Take stylization as an example, the proposed structure control in Eq. 4 only ensures consistency in terms of each frame's composition, while enabling the model to generate the required textures and styles based on the text prompt. On the other hand, the content preservation technique inherits the textures and style of the source video. Therefore, we use structure control instead of content preservation for appearance editing. In addition, using the proposed structure control technique in motion editing can make the layout of the output video similar to the source video (shown in Fig. 11b in Appendix). Users have the flexibility to adjust the consistency between the edited video and the source video layout based on their specific requirements.To further improve the editing performance, we suggest leveraging the foreground/background segmentation mask M to guide the denoising process [14, 13] . There are two possible ways to obtain the mask M : the attention maps of CA-S modules with a threshold [24] ; or employing an off-the-shelf segmentation model [38] on the source and generated videos. The obtained segmentation masks can be leveraged to 1), alleviate the indistinction in foreground and background; 2), improve content consistency between edited and source videos. To this end, we leverage mask-guided self-attention in the main editing path to coordinate the editing process. Formally, we define:EQUATIONThen the mask-guided self-attention:EQUATIONwhereM f , M b ∈ {-∞, 0}indicate the foreground and background masks in the editing path respectively, M m ∈ {0, 1} denotes the foreground mask from the motion-reference branch, and ⊙ is Hadamard product. In addition, we leverage the mask during the content preservation and motion injection for the features obtained from the reconstruction branch and the motion-reference branch (e.g., we replaceQ m with M m ⊙ Q m + (1 -M m ) ⊙ Q).To make our framework more flexible, we further derive a method to incorporate images as input and synthesize high-quality video conditioned on both image and text-prompt. Different from some image animation techniques [2] , our method allows the user to guide the animation process with text prompts. Concretely, we first achieve image-to-video (I2V) generation by: 1) transforming input images with simulated camera movement to form a pseudo-video clip [44] or 2) leveraging existing image animation approaches (e.g., SVD [2] , AnimateDiff [21] ) to synthesis a video with random motion (which may not consistent with the text prompt). Then, we perform text-guided editing with UniEdit on the vanilla video to obtain the final output video. 
comparison with state-of-the-art methods: 
 Implementation Details UniEdit is not limited to specific video diffusion models. In this section, we build UniEdit upon LaVie [63] as an instantiation to verify the effectiveness of our method. To demonstrate the flexibility of UniEdit across different base models, we also implement the proposed method on VideoCrafter2 [9] and exhibit the editing results in Appendix B.1. For each input video, we follow the pre-processing step in LaVie to the resolution of 320 × 512. Then, the pre-processed video is fed into the UniEdit to perform video editing. It takes 1-2 minutes to edit on an NVIDIA A100 GPU for each video. More details can be found in Appendix A.Baselines. To evaluate the performance of UniEdit, we compare the editing results of UniEdit with state-of-the-art motion and appearance editing approaches. For motion editing, due to the lack of open-source tuning-free (zero-shot) methods, we adapt the state-of-the-art non-rigid image editing technique MasaCtrl [5] to a T2V model [63] (denoted as MasaCtrl * in Fig. 5 ) and a one-shot video editing method Tune-A-Video (TAV) [65] as strong baselines. For appearance editing, we use the latest methods with strong performance, including FateZero [46] , TokenFlow [18] , and Rerender-A-Video (Rerender) [70] (further comparisons in Fig. 13 ,14,15 of Appendix B.7). For a fair comparison, we also migrated all baselines to LaVie [63] , using the same base model as our method. The results are presented in Fig. 15 . For appearance editing, we showcase two scenarios: non-rigid object replacement and stylization. In object replacement, our method outperforms baselines in terms of prompt alignment and background consistency. In stylization, UniEdit excels in preserving content. For example, the grassland retains its original appearance without any additional elements. In motion editing, UniEdit surpasses baselines in aligning the video with the target prompt and preserving the source content.Quantitative Results. We quantitatively evaluate our method using two approaches: 1) CLIP scores and user preference, as employed in previous work [65] ; and 2) VBench [31] scores, a recently proposed benchmark suite for T2V models. The summarized results are in Tab. 1. Following previous work [65] , we assess the effectiveness of our method in terms of temporal consistency and alignment with the target prompt. Additionally, we conducted a user study involving 10 participants who rated the edited videos on a scale of 1 to 5. We also utilize the recently proposed VBench [31] benchmark to provide a more comprehensive assessment, which includes 'Frame Quality' metrics and 'Temporal Quality' metrics. UniEdit outperforms the baseline methods across all metrics. Furthermore, the mask-guided coordination technique introduced in Sec. 4.3 further enhances performance (see Appendix B.3). For more detailed quantitative results, please refer to Appendix B.2&B.3&B.5.How UniEdit Works? To better understand how UniEdit works and reveal our insight on the spatial and temporal self-attention layers, we visualize the features in the SA-S and SA-T modules and compare them with the magnitude of optical flow between adjacent frames in Fig. 6a . It is evident that, in comparison to the spatial query maps (2nd row), the temporal cross-frame attention maps (3rd row) exhibit a notably higher degree of overlap with the optical flow (4th row). This indicates that the temporal self-attention layers encode inter-frame dependencies and facilitate motion injection, while content preservation and structure control are carried out in the spatial self-attention layers. Output Visualization of the Two Auxiliary Branches. Recall that to perform motion editing, we propose to transfer the targeted motion from the motion-reference branch and realize content preservation via feature injection from the reconstruction branch. To verify the effectiveness, we visualized the output of each branch in Fig. 6b . It is observed that the motion-reference branch (4th row) generates video with the target motion, and effectively transfers it to the main path (3rd row); meanwhile, the main path inherits the content from the reconstruction branch (2nd row), thus enhancing the consistency of unedited parts. The (a) Ablation study on t0 in Eq. 2.(b) Ablation study on t1 in Eq. 4.Figure 7 : Ablation study on hyper-parameters.Ablation on Hyper-parameters. We utilize content preservation in Eq. 2 to maintain the original content from the source video. By varying the feature injection steps in Fig. 7a , we observe that replacing the value features at a few steps introduces inconsistencies in the background (footprints on the beach). In practice, we adhere to the hyper-parameter selection outlined in [5] (last row).Simultaneously, we note that adjusting the blend layers and steps in Eq. 4 can effectively regulate the extent to which the edited image adheres to the original image. For instance, in the stylization demonstrated in Fig. 7b , injecting the attention map into fewer (15) steps yields a stylized output that may not retain the same structure as the input, while injecting into all 50 steps results in videos with nearly identical textures but less stylization. Users have the flexibility to adjust the blended steps to achieve their preferred balance between stylization and fidelity.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Due to company policy reasons, we are currently unable to upload the code.The code will be publicly available after the paper is published.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion and limitations: 
 In this paper, we design a novel tuning-free framework UniEdit for both video motion and appearance editing. By leveraging a motion-reference branch and a reconstruction branch and injecting features into the main editing path, it is capable of performing motion editing and various appearance editing. There are nevertheless some limitations. Firstly, we observe performance degradation when performing both types of editing simultaneously. Secondly, since our work is based on T2V models, the proposed method also inherits some of the shortcomings of the existing models, such as inferior performance in understanding complex prompts. We exhibit the failure cases in Appendix B.6.As illustrated in Tab. 1, UniEdit shows the best performance on frame consistency. Regarding textual alignment, UniEdit significantly outperforms all other baselines, demonstrating its capacity to support diverse editing scenarios.Baselines. We implement all baseline methods with their official repositories. For MasaCtrl [5] ,we adapt it to video editing by first setting the base model to a T2V model [63] , then performing MasaCtrl on all frames of the source video. Moreover, since most baselines use StableDiffusion (SD)as the base model, we resize the source video to 512 × 512 to align with the default configuration of SD, then feed it into the denoising model, which can maximize the power of SD.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper provides detailed information on the models, parameters, hyperparameter selection, computational resources in Sec. 5 and Appendix A to ensure reproducibility.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The common practice in video editing does not including error bars, and we follow the previous papers.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This paper provides detailed information on the computational resources in Sec. 5 and Appendix A and inference time comparison in Tab. 6.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The broader impacts are discussed in Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, the creators or original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have uploaded the code of this paper to an anonymous repository and provided the corresponding link in Appendix. The code will be made publicly available after the paper is published.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We additionally implement our method on VideoCrafter2 [9] , a concurrent work on T2V generation to demonstrate the flexibility of UniEdit. The results are shown in Fig. 8 . In practice, we empirically found set these values to fixed values, i.e., t 0 = 50, L = 10 (same as MasaCtrl [5] ) and t 1 = 25 can achieve satisfying results on most cases, and we further perform a quantitative study when applying different hyper-parameters in Tab. 3&4. To investigate the impact of mask-guided coordination, we begin by visualizing masks obtained from 1) the attention map in CA-S modules; 2) the off-the-shelf segmentation model SAM [38] ,followed by presenting both qualitative and quantitative results of implementing UniEdit with or without mask-guided coordination.As verified by previous work [24] , the attention maps in CA-S modules contain correspondence information between text and visual features. The underlying intuition is that the attention maps between each word and the spatial features at point (i, j) indicate 'how similar this token is to the spatial feature at this location'. We visualize the text-image cross attention map alongside the synthesized frame in Fig. 9 . We observe spatial correspondences that align with the video output from the attention map. For instance, areas with higher values of the token 'man' and 'NYC' correspond to the foreground and background, respectively. We further employ a fixed threshold (0.4 in practice)to derive binary segmentation maps from the attention maps. For comparison, we also display the segmentation mask obtained by point prompt on SAM. It's observed that the cross-attention mask is generally accurate and could serve as a reliable proxy in practice when an external segmentor is not available.We examine the impact of mask-guided coordination through both qualitative and quantitative results across 4 settings: {w/o UniEdit, UniEdit w/o mask, UniEdit with mask from CA-S, UniEdit with mask from SAM}. Qualitatively, shown in Fig. 10 , the implementation of UniEdit significantly enhances the consistency between the edited videos and the original video. The application of the mask-guided coordination technique further improves the consistency of unedited areas (e.g., color and texture). The quantitative results in Tab. 5 align coherently with this analysis. We conduct a theoretical analysis of the additional cost of UniEdit and an empirical comparison with baseline methods in terms of inference speed.Theoretically, our method primarily involves feature replacement operations in attention modules, achieved through forward hook registration and introducing minimal additional computation. Therefore, the main difference between synthesizing a video from random noise and editing a video with UniEdit lies in the batch size of the denoising process (i.e., vanilla generation: batchsize=1, appearance editing: batchsize=2, motion editing: batchsize=3), and this process could be further accelerated through multi-GPU parallel processing techniques. Additionally, we utilize LaVie [63] as the base T2V model in the paper, which takes approximately 45 seconds to synthesize a 16-frame video. Our method can be even faster when adapted to more efficient base models.Empirically, UniEdit demonstrates comparable speed with baseline methods. The comparison of inference time on a single 16-frame source video clip with a resolution of 320x512 on 1 NVIDIA A100 GPU is as follows: We exhibit failure cases in Fig. 12 . Fig. 12a showcase when editing multiple elements simultaneously, and we observe a relatively large inconsistency with the source video. A naive solution is to perform editing with UniEdit multiple times. Fig. 12b visualizes the results when editing video with complex scenes, and the model sometimes could not understand the semantics in the target prompt, resulting in incorrect editing. This may be caused by the base model's limited text understanding power, as discussed in [30] . It could be alleviated by leveraging the reasoning power of MLLM [30] , or adapting approaches in complex scenario editing [40] . Please refer to Fig. 13 and Fig. 14 for more comparison with the state-of-the-art methods. For a fair comparison, we also migrated all baselines to LaVie [63] , using the same base model as our method.The results are presented in Fig. 15 , and they are found to be inferior compared to those in Fig. 5 (based on Stable Diffusion).More edited results of UniEdit are provided in Fig. 16 UniEdit is a tuning-free approach and is intended for advancing AI/ML research on video editing.We encourage users to use the model responsibly. We discourage users from using the codes to generate intentionally deceptive or untrue content or for inauthentic activities. It is suggested to add watermarks to prevent misuse.Justification: The research strictly adheres to the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).For real source video, we set source prompt to null during both forward and inversion process to achieve high-quality reconstruction[43]. 


Decision: The paper initially received mixed reviews, with ratings of 4, 6, and 7.  
**Reviewer UJWt** increased the score to *borderline accept* after the rebuttal but expressed some concerns about the novelty. Specifically, **Reviewer UJWt** noted that *"Content Preservation, Motion Injection, and Spatial Structure Control (Equations 2, 3, and 4) are all different forms of feature injection and a direct extension of feature injection [56] from image diffusion models to video diffusion models."* 

**Reviewer rXMa** maintained the *weak accept* rating and had no further comments.

**Reviewer t6q7**, who initially gave an *accept* rating, flagged the confidence as less certain. After the rebuttal, **Reviewer t6q7** lowered the rating to *borderline accept*. 

The paper's final ratings are, therefore, $(5, 6, 5)$. Although the final ratings are on the positive side, there is much uncertainty and concerns about the analysis in the paper.  The rebuttal presents additional helpful analysis.  However, due to the large number of submissions and limited capacity, preference must be given to papers that do not require substantial revision to be acceptable and, therefore, due to the remaining concerns and significant needed revision, the area chair unfortunately feels compelled to recommend rejecting the paper. This decision was also reviewed by the senior AC.

Reviews: review: paper_summary: paper_summary: This paper focuses on developing a tuning-free framework capable of editing both the motion and appearance of videos. They introduce UniEdit, an approach designed for text-guided motion editing that maintains the original content of the source video. By utilizing two branches—an auxiliary reconstruction branch and an auxiliary motion-reference branch—they achieve both content preservation and effective motion editing.
strength_weakness: strength_weakness: 1. The number of the participants in the user study might not be representative enough.
questions: questions: 1. The first row of the table 2 is missing. What is that?
2. It will be more informative if you could ablate with motion injection and structure control solely in Table 2 as well.
3. How do you measure texture alignment and frame consistency in Table 2?
limitations: limitations: This method is inherently influenced by the T2V model used.
paper_summary: paper_summary: This paper suggests UniEdit, a tuning-free method for editing the motion of a given video. The authors use a pre-trained text-to-video diffusion model and utilize its motion prior, to performing motion editing on a video while keeping the appearance of the original video. During the denoising process, they apply structural/content features injection from the reconstruction branch of the original video to maintain the input video's structure or content. The motion is edited according to a text description used to denoise another reference branch which is then used for injecting features into the editing videos. The results improve over the existing methods.
strength_weakness: strength_weakness: 1. Novelty.  Feature injection for image editing is a known technique [56].  Applying injection to video models is important and challenging, but not novel enough in my view.
Showing that the injection of motion features from the reference motion branch into the edited video, constrains the output motion, is important, but not surprising given the observation of [56]. 

2. Given that the main insight of the paper is that “the temporal self-attention layers of the generator encode the inter-frame dependency”, there is not enough analysis of this besides the visual results and Figure 6, which shows the relation between the optical flow magnitude and the temporal attention on one example qualitatively. Showing a quantitative analysis, and analyzing the features during the denoising process for different layers, could support this claim better and show the importance of this insight.
questions: questions: Can the authors respond to the two weaknesses written above?
limitations: limitations: Yes, the authors discuss both, limitations, and broader impact.
paper_summary: paper_summary: The paper proposes UniEdit, a framework that allows the editing of videos. More specifically, UniEdit allows manipulation via text prompts to change the visual style or the motion pattern that is visible in the video. Moreover, it also targeted steering, e.g. via segmentation masks. They achieve this by introducing an additional reconstruction branch and a motion-reference branch into a u-net based diffusion network and share the values of the attention layers which are party designed specifically for this work. The method allows editing videos without retraining and creates very good results.
strength_weakness: strength_weakness: The implementation may not be entirely reconstructable. Hopefully, this issue will be fixed when they publish the code as promised.

Even if the method description is understandable, the math is sometimes not entirely correct. For example in line 212/212, M is a matrix but the notation says that it is a scalar from a set $\{ -\inf, 1 \} $ or $\{ 0, 1 \} $. The authors should be encouraged to revise the math present in the paper.

**Minor weaknesses:**
Sometimes the English writing is a bit weak. For example: 
 - 100-110: Some parts are not forming complete sentences, e.g. "Other improvements like efficiency [1], training strategy [19], or additional control signals [16], etc."
 - 187: I think it should be "an additional network" or "additional networks"
questions: questions: - Is there a reference that also handles the evaluation process as described in lines 242-245?
limitations: limitations: Unfortunately, there is no benchmark to evaluate the method. This is not the author's fault and they tried to do their best to create baselines. However, this makes it harder to rate the results.

Comments: title: Further Response to Reviewer rXMa
Further Response to Reviewer UJWt
comment: Thanks for your reply! We are delighted to see that our responses addressed your questions and concerns. We will incorporate these classifications and experiments into the final version.
Thank you for your response and appreciation! We are dedicated to continually enhancing our work, and we will incorporate these analyses and experiments into the final version.
Thanks for the efforts of addressing all the questions and concerns. I have no further questions, and will keep the current rating.
In my opinion, Content Preservation, Motion Injection, and Spatial Structure Control (Equations 2, 3, and 4) are all different forms of feature injection and a direct extension of feature injection [56] from image diffusion models to video diffusion models. While I appreciate the quality of the results and the technical contribution, I do not find it particularly novel. However, I find the analysis of the temporal self-attention layers provided here interesting. Assuming this analysis will be included in the revised version, I would like to increase my score to borderline accept.