ReVideo: Remake a Video with Motion and Content Control
Authors: Chong Mou
Mingdeng Cao
Xintao Wang
Zhaoyang Zhang
Ying Shan
Jian Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model
Video editing
Primary area: diffusion_based_models

Abstract: Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-area editing without specific training, demonstrating its flexibility and robustness.

Re Video: Remake a Video with Motion and Content Control
Chong Mou
Mou 1.2.4 ,iingenng Cao Xintao Wang Zhaoyang Zhang', 22,1+ -2,1+
Ving Shan Jian Zhang-
School of Electronic and Computer Engineering, Peking University
2 Pekking University Shenzhen Graduate School- -abbbiae AIGC Joint Research Laboratory
ARC Lab, Tencent PCG University of Tokyo
A GGanggong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology
https h/cc///cceeeee////// gi thub. ttriii//poope project/ReVideo.
Change conteot
Castemine trajectory
Trajectory:
Original
Change content &
Keep trajectory
Trajectury:
Keep conticat
Castenice trajectory
Trajectwy::
Original
object-lvvel Internttions
Trajectory:
Editiigg
Trajectury:
Original
The capability of method to locally modify video content and motion. This ability can also be easily extended to multi-area editing. The motion control is labeled in colorful lines in videos.
Figure I:
Abstract
Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial chal- lenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification both content and motion. Content editing facilitated by modify-
*Corresponding author
This work was also supported by the National Natural Science Foundation of China (Grant No. 62372016)and Guangdong Provincial Key Laboratory Ultra High Definition Immersive Media Technology 202224112120100000
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ing the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising per- formance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-are editing without specific training, demonstrating its flexibility and robustness.
Introduction
Thanks to the large-scale training data and huge computing power, there have been significant advancements in diffusion-based [1]] image and video generation. For personalization purposes,
many works add control signals the generation process, such as text-guided image (39
40 38
and video 18
17
13 45
generation, as well as image-guided video generation 4 544 Based on
these base models, extensive works explore how to transfer their generation capabilities to video editing. Early works based on text-to-image diffusion models implement video editing through zero-shot strategies (e.g., Fate-Zero 36), Flatten (T]) or one-shot tuning (e.g.. Tune-A- -iideo (.I). However, these methods are limited by excessive manual design and a lack of video generation priors. Moreover, text prompt only provide coarse condition, limiting the editing accuracy. Compared to text, more recent methods adopt image conditions which can provide more accurate editing guidance. For instance, VideoComposer [47] generates tranffrranfo videos by providing spatial attributes (e.g.. edge, depth) of the target video and a style reference. Dream Video [49 and Make-a-protagonis [62] can modify a specific object in the video by providing a reference object. However, these methods still struggle with local editing and introducing new elements, such as adding new objects to video. Recent work EVE [42 proposes a diffusion distillation strategy to achieve video editing while keeping unedited content unchanged. Nevertheless, the editing region and target controlled by text, which is challenging in complex scenarios. Any [26] edit a video by modifying the first frame, enabling accurate customization of local content. Pika [1] regenerate specific area in the video by selecting an editing region. Although these methods improve the performance of local video editing, they only focus visual content editing and cannot customize the motion of new content.
Motion another crucial aspect of video, yet research on video motion editing remains limited. While some methods explore motion-guided video generation using sejjectory-based motion guidance (e.g., DragNUWA [57], Dryhhnn [52], MotionCtrl 48, VideoSwap (16) and box-based motion guidance (e.g., Boximator [44), Peckaboo (20)), they do not support motion editing. Additionally, other works [56 30 61] can transfer motion from one video to another but cannot modify as well.
In this paper, our goal is to accurately edit content and motion in specific areas of video. We create an easy-to-inteed tnteettccct pipeline by setting the content editing as modifying the first frame, with trajectory lines [57) as the motion control signal. Other unedited content in all frames should be maintained in editing results and merged with the editing effect. However, we find that fusing unedited content with motion-customized new content is challenging, mainly for two reasons: (1) Training imbalance: Unedited content is dense and easier to learn, while motion trajectories are sparse and abstract, making them harder to learn. (2) Condition coupling: Unedited content provides both visual and inter-frame motion information, leading the model to rely on it for motion estimation, thereby ignoring the hard-to-learn trajectory lines.
To address these challenges, we design a three-stage training strategy to harmonize unedited content and motion-customizn new content, enabling harmonious control of different conditions. Besides, we design a spatiotemporal adaptive fusion module fuse these two conditions different diffusion sampling steps and spatial locations. Furthermore, our method can compactly inject motion and content conditions into the diffusion video generation through a single control module. With these techniques, users can conveniently edit specific regions in the video by modifying the first frame and drawing trajectory lines. Notably, Re Video not limited to single-region editing and can customize multiple areas in parallel.
2

---

In summary, this work makes the following contributions:
To the best of our knowledge, this is the first attempt to explore local editing of both content and motion in videos. Our method can also be easily extended multi-area video editing.
We propose a three-stage training strategy and a spatiotemporal adaptive fusion module to address the coupling of content and motion control in video editing, enabling compact control through single module.
Extensive experiments demonstrate that ReVideo performs well in several precise video editing applications, including changing content in a specific region while keeping motion constant, maintaining content while customizing new motion trajectories, and modifying both content and motion trajectories. Some examples are presented in Fig.
2 Related Works
2.1 Controllable Image and Video Generation
19 drive the rapid development of image and video generation. In the community of image generation, some notable works, such as Stable Diffusion [Р) Imagen 40.. and DALL-E2 38. utilize text as the generation condition. To achieve accurate generation control, some methods, e.g., ControlNet [58) and T21-Adapter [34), propose adding control modules on pre- pre-trained diffusion models. Similarly, initial efforts in controllable video generation concentrate the text condition, such as Video LDM [5]. Imagen Video [18), VideoCrafter [8] and AnimateDiff [17. Recognizing the limitations of text prompts in capturing complex scenarios, some recent works modules on pre-trained foundation models are also popular in controllable video generation. Such as video ControlNet [9 60] extends the ControlNet [58) in image generation to video generation conditioned on a sequence of control signals, like edge maps and depth maps. In addition to spatial structure control, precise temporal motion control is also important in controllable video generation. This process is similar to the drag- based image editing 35|
Recent advancements in diffusion models
59 13
leverage image conditions for more direct approach. External control
32 33 Several recent works study this topic, such video generation with trajectory- based motion guidance (e.g., DragNUWA (57], MotionCtrl [48), Motion-I2V [1), DragAnything [52)) and generation with box-based motion
guidance (e.g., TrailBlazer [29] Boximator 44
20 These methods perform the control by
training extra motion controllers on pre-trained video diffusion models.
2.2 Diffusion-based Video Editing
Due to the lack of training data, the common approach in video editing is via training-free strate-
gies 77 15 21 25 46 (36] or one-shot tuning [51] 3 24 For instance, the prior work Tune-A- Video 511 overfits some diffusion model parameters to specific video. Then, it uses the overfitting parameters to produce the editing result conditioned on the target prompt. To enable cohesive global appearance among edited frames, many methods extend the attention module of Stable Diffusion (39 to encompass multiple frames and conduct cross-frame attention. For instance, Pix2Video edits the first frame and performs cross-frame attention of each frame on the first frame to preserve appearance consistency. TokenFlow [13] and Fairy [50] jointly edit a few key frames at each denoising step and propagate them throughout the video based on the nearest-neighhor field extracted from the original video. Inspired by the initial zero-shot image editing method SDEdit [31], the recent video foundation model SORA [6] achieves video editing by adding noise to the input video and then denoising it under the target description. Although these methods can preserve the general structure of original videos, the information loss and the lack of consistency constraints the original video make them unfit for precise video editing but suitable for global editing like style transfer.
Another strategy is to train a control module to guide the generation with some characters that should persist in the editing result, such as depth [4 27 533, sketch [47), and optical flow [55]. However, existing methods primarily focus on preserving spatial structure and are unsuitable for precise video editing. the community of precise video editing, some works, such as InsV2V [10] and the recent EVE [42] edit the video by providing editing instructions. However, the text-based editing instruction struggles to locate a target region in some complex scenarios. Vny (26) can edit a video by editing the first frame. Pika [ is designed to regenerate selected area in video by text guidance.
3

---

A
B
Treinable
Centrol
Traieale Medule
Content
Content
Reference Image
Outpet
tre-froised
Reference Image
Output Video
Trabnable
Pre. trained SVD
Centrol
Zero Conv
Figure 2: Two potential structures inject motion and content control. Structure A is a compact and efficient mode that integrates motion and content control via single module. Structure B features in- dependent control, structurally decoupling motion and content conditions, causing higher complexity.
Unlike these works, we aim achieve accurate customization in local areas of video. The editing target includes locally modifying content and motion and keeping the unedited content unchanged.
Method
3.1
Preliminaries
Stable Video Diffusson (SVD) 4 is high-quality and commonly used image-to-video generation model. To utilize the priors of high-quality video generation, employ SVD the base model and add control modules to achieve our editing target. Given a reference image c₁, SVD will generate a video frame sequence is conducted on a latent denoising diffusion process 399
{x", x! x < of length N, starting with c₁. The sampling of SVD
At each denoising step, conditional 3D
UNet 4 is used iteratively denoise this sequence:
2n Lz
(I)
where z₁ the latent representation of xg. ău is the predication Zy. There are two condition paths for the reference image cy: (1) It is embedded into tokens by CLIP (37] image encoder and injected into the diffusion model vis cross-attentin
[39] (2) It is encoded into a latent representation by the
VAE encoder, and concatenated with the latent of each frame in channel dimension. SVD follows the
EIDM-peccontitioningg framework (23,, which parameterizes the learnable denoiser Pp as:
( 1, Cr:aa = Cakip
Coua(a) (e (a 2 L Cri Cnosee (a));
where o is the noise level, and F& is the network to be trained. Cakips Couts Cius and Cnoise are preconditioning hyper- parameters. 48 is trained via denoising score matching (DSM):
(.z..t.n (o n, n,t,er)
2o1
Task Formulation and Some Insights
The purpose of this paper is to locally edit video, including visual information and motion information. In addition, the unedited content in the video should remain unchanged. Therefore, our conditional video generation involves three control signals: (1) the edited content, (2) the content the unedited area, and (3) the motion condition the edited area. We implement content editing by modifying the first frame of the video and then broadcasting it to subsequent video frames. Here, we denote the edited first frame as Cref E g3x x For the motion condition, we use interaction-friennt friendly trajectory lines (57 (2)
Task formulation.
the control signal. Specifically, the motion condition also contains maps for a N-frame video. Each map consists of channels, indicating the movement of the tracked points in the horizontal and vertical directions relative to the previous frame. The motion condition in this paper is represented as Cymo E R x2x W H The unedited content Coon can be conveniently provided by the masked video, i.e., Ccon = V M, where V Nx3x 3xxxxxx and M E R x refer to the original video and the editing region mask, respectively.
Since we adopt SVD as the pre-trained base model, its toage-to-videe capability can naturally serve as the import port for the edited first frame. For unedited content and customized motion trajectories, we train additional control modules to import them into the generation process.

---

Toy experiment
experiment
Cattinn LEP0
Original video
Motion trajectory
Structure & Direct training
training
experiment
Toy experiment 4
Stage
Stage
Structure B & Motion prior training
Structure & Coarse-to-fine training
Figure 3: The motion control capability of two structures in Fig. 2 with different training strategies. We visualize trajectory lines in a specific area (red box) and label the editing area with a black box. Toy experiments present the coupling issue of customized motion and unedited content.
Trajectory sampling. During training, it is essential to extract trajectories from videos to provide motion condition Cmot. At the beginning of trajectory sampling, we use grid [57] to sparsify dense sampling points, obtaining Nivit initial points. Among these points, those with larger motions beneficial to train trajectory control. To filter out these points, we first apply motion tracking on each point to obtain their path lengths, i.e., flo, l1,.... NN..a- We the mean of these lengths the threshold lyn to extract points whose motion length is greater than ąyy. Then, we use the normalized lengths of these points sampling probabilities to sample N points randomly. Because the high sparsity is not conducive for the model to learn from these trajectories, we apply a Gaussian filter [57] to obtain the smooth trajectory map Cmot- More details presented in Appendix.
Insights. naive implementation of our editing target directly training an extra control module, like ControlNet (58), to inject motion and content conditions into the diffusion generation process. We present this design in structure A of Fig. 2 Specifically, at the input, a content encoder E, and motion encoder E. embed the content condition Ccon of the unedited area and motion condition Cmot of the editing area. These two embeddings are merged by direct summing to obtain the fused condition feature fo. Then, copy of the UNet encoder extracts multiscale intermediate features from f., which are added to the corresponding layers in the diffusion model. This process is formulated as:
(4) where Ye is the new diffusion features. Z is the function of zero-conv (58) + and O are the parameters of the SVD model and extra control module. We conduct several toy experiments based on this idea, as illustrated in Fig 3 The input video contains woman initially moving to the left, followed by shift the right. The editing target is alter the facial motion towards the right while keeping the other content unchanged. In the toy experiment 1, we fix SVD and train the control module with Eq. The result shows that the content condition precisely controls the unedited area of the generated video. But the motion condition has no control effect, and the trajectory lines in the editing area (labeled with black box) are consistent with the unedited area. A possible reason is that single control branch has difficulty handling two control conditions simultaneously. To verify this hypothesis, we train structure
Ye F(z,, I, Crefi θ) (FF(z; + 2 (ff.), Crefi O2)),
Fig. 2 to handle these two conditions separately. The toy experiment 2 in F shows that the motion control is still ineffective, suggesting that the problem more attributed to the control training rather than the network structure. To enhance the motion control training, we split the training of structure B into two stages. In the first stage, we only train the motion control module to endow it with motion control prior. In the second stage, we train the motion control and content control together. The result in toy experiment 3 shows that although the motion prior training produces good motion control capability, the control accuracy is weakened and affected by the unedited content after introducing the content control. After these toy experiments, we have the following insights:
o The condition of unedited content not only contains visual information but also has rich inter-frame motion information. As more easily learned condition, the diffusion model tends to predict the motion of the editing area through unedited content, ignoring the sparse motion trajectory control.
5

---

M
V V2 M V22 (1
M)
Editing result from decoupling training
Training data in decoupling training
Figure 4: The data construction strategy for decoupling training and editing results from this stage.
The coupling between notion-customizer new content and unedited content is strong, making difficult to overcome even using the motion prior and separate control branches.
Motion prior training is helpful in decoupling motion-customizze content and unedited content.
3.3 Coarse-to-fine Training Strategy
To rectify the ignoring of the motion control, we design a coarse-to- training strategy. addition, structure in Fig. 2 has high computational cost, and hope to joint control the unedited content and motion-customized new content on the concise structure A.
Motion prior training. As discussed above, motion trajectory is a sparse and difficult-to-eearn control signal. Toy experiment Fig. 3 shows that the motion prior training can alleviate the coupling between train the motion trajectory control, allowing the control module to have good motion control prior.
content and unedited content. Hence, in the first stage, only
Decoupling training. Based on the control module from the first stage, the training in the second stage aims to add content control of unedited areas. Toy experiment good motion control priors, the precision of motion control still degrades after introducing unedited content condition. Therefore, we design training strategy to decouple motion and content control in this stage. Specifically, we set the editing part and the unedited part in a training sample V to be two different videos, i.e., V, and V2. As shown in Fig. 4 V, and V2 are combined through the editing mask M, i.e., V V1 M + V2 (1 M). Since the editing region and the unedited region come from two different videos, the motion information of the editing region cannot be predicted through the unedited content. Therefore, it can decouple content control and motion control during training.
Fig. B] that even with
Deblocking training. As shown in the right part Fig. 4 although the decoupling training achieves joint control of customized motion and unedited content with high accuracy, it breaks the consistency between the edited and unedited regions, producing block artifacts in the boundary. To rectify this issue, we design the third training stage to remove block artifacts. The training in this stage is initialized with the model from the second stage and trained on normal video data. To preserve the decoupled motion and content control prior from the second stage, we only fine-tune the key embedding We and value embedding W temporal self-attention layers of the control module and SVD model. The toy experiment 4 in Fig 3 shows that after the training of this stage, the model removes the block artifacts and retains joint control of unedited content and motion customization.
3.4 Spatiotemporal Adaptive Fusion Module
Although the coarse-to-fine training strategy achieves decoupling of content control and motion control, we observe considerable failure cases some complex motion trajectories. To further distinguish the control roles of unedited content and motion trajectories in the generation. a spatiotemporal adaptive fusion module (SAFM) as shown in Fig. 5 Specifically, SAFM predict weight map I' through the editing mask M to fuse motion and content control instead of direct summing. Moreover, because diffusion generation a multi-step iterative process, the fusion of control conditions between time steps should have adaptive adjustment. Therefore, concatenate timestep and M in the channel dimension to form a spatiotemporal condition to guide the prediction. Mathematically, the fusion of motion and content conditions is formulated as follows:
design
f. (Ceon) I Em (Cmot) (1- F), T= H(M,t).
(5)

---

Content Encoder
Content
Conws
Sigmoid
1000
t=800
1-600
f
Editing Region
"Encoder
Motton
t=400
F00
F0
Motion
Spatiotemporal Adaptive Fusion Module (SAFM)
Visualization of I different timesteps
Figure 5: The architecture of our proposed spatiotemporal adaptive fusion module (left), and the visualization of fusion weight at different timesteps (right).
Table 1:
Quantitative comparison between Re Video and other related works. We employ automatic metrics (i.e., CLIP 37 score, PSNR) and human evaluation to evaluate the performance.
Automatic Metries
Human Evaluation
Method
Time +
PSNR t
Text Alignment
Consistency
Overall t Editing Target t
InsV2V [10]
29.77
0.2022
0.9808
10.2%
5.1%
132s
Any [26]
29.80
0.2143
0.9836
2.8%
4.0%
380s
Pika
33.07
0.2184
0.9956
27.9%
23.9%
ReVideo
32.85
0.2304
0.9864
59.1%
67.0%
26s
where is the function of spatiotemporal embedding. H needs be jointly trained with Wa W and W, a in the deblocking training stage. We visualize I different time steps in the right part of Fig. can be seen that r learns the spatial characteristics the motion condition in the editing area and higher weight to the content condition in the unedited area. In addition, learns to distinguish different sampling steps and linearly adjusts with t.
the editing area. It assigns a higher weight to
Experiments
4.1 Implementation Details
In this work, choose Stable Video Diffusion (SVD) the base model. Our three training stages
are completed on the Web Vid (2] dataset, which contains 10 million text-video pairs. These three
stages are optimized for 40K, 30K, and 20K iterations, respectively, with Adam (88 optimizer NVIDIA A100 GPUs. The batch size for each GPU is set as 4, with the resolution being 512 x 320. CoTracker [22
takes about 6 days complete all training stages. During the training process, to extract motion trajectories. the first training stage, trajectory sampling is performed throughout the video. In the second and third training stages, rectangular editing area randomly selected in the video with the minimum size being 64 x 64, and trajectory sampling is performed within it. The number of trajectory lines for each training sample is randomly selected between 1 and 10.
4.2 Comparison
Among existing methods, Pika 1 is the most similar to ours. Pika can perform local video editing by defining an editing area. The difference is that Pika controls the new content in the editing area by text and has no motion control. In addition, the recent work Any V2V (26) proposes editing the first frame of the video to achieve entire video editing, which has similarities with our Re Video. Vns [10), using editing instructions to edit the video, can also maintain unedited content. Therefore. in this paper, we compare our VeVideo with these three methods. The visual comparison Fig. shows that in some fine-gn -eee--rrindd editing scenarios, such as putting sunglasses on man, Any V2V has loss of edited content. addition, the unedited area Ins and AnyV2V suffers from content distortion. Although Pika can generate smooth and high-fidelity results, it is difficult to accurately customize new content by text, especially adding new objects, e.g., adding dog on the soccer field. Adding new objects to the scene is also challenging for Ins V2V. Due to the lack of motion

---

"A man wearing sunglasses"
"A car running on the road"
InsV2V
Any
Pika
Original
"A robot lizard"
"A dog soccer field"
InsV
AnyV2V
Pika
Ours
Original
Figure 6: The visual comparison between V2V 10
V2V
26
Pika
and our Re Video.
control, V2V and Pika usually produce static motion of the edited content, such as a car driving on road. In comparison, our ReVideo can effectively broadcast the edited content throughout entire video while allowing users to customize the motion in editing areas.
In addition to visual comparison, we employ automatic metries and human evaluation measure the performance of different methods. For this task, we build a test set containing 16 videos, with the resolution being 720 1280. Following previous works 7 26). automatic metrics employ CLIP score (37 to measure text alignment and temporal consistency. The text alignment is obtained by calculating the average CLIP cosine similarity between each frame and editing description. Temporal consistency is computed by average CLIP cosine similarity between every pair of consecutive frames. employ PSNR [43] to measure the reconstruction quality unedited content. The human evaluation considers two aspects, i.e., overall video quality, and whether the editing target is achieved. We allow 20 volunteers to choose the best method for each test sample on each aspect. The results in Tab. D show that our Re Video performs better than Ins V2V and Any V2V in all evaluation terms.

---

Refercnce
Original Video
w/o SAFM
SAFM wio time adaptation
Tuning control module stage
Tuning spatial layers stage
ReVideo
Figure 7: Ablation study of our ReVideo.
Compared with Pika, our performance is slightly lower in the evaluation of temporal consistency and the quality of unedited content. Notably, Any V2V and Pika usually generate static motion of new content due to the lack motion control. Static motion tends to score higher consistency evaluation, measured by CLIP similarity of adjacent video frames. Our method has obvious advantages over Pika text alignment and human evaluation, reflecting the significant gap between rext-guided local editing and user-specified local editing. Our ReVideo can precisely specify the appearance and motion of the editing area, better meeting requirements for accurate customization.
The time complexity of different methods is also presented in Tab. The experiment is conducted on an A100 GPU, with the video resolution being 768x768. Results show that our method has significantly lower time costs compared to other methods.
4.3 Ablation Study
In our ReVideo, design the spatiotemporal adaptive fusion module (SAFM) to help decouple the control of unedited content and motion customization in diffusion generation. It predicts a fusion weight I conditioned the editing area M and time step Then, the fusion of content and motion control is achieved through Eq. [5] In this part, we conduct an ablation study on this fusion mechanism. In addition, we only fine-tune the key embedding and value embedding of the temporal self-attention layers in the SVD model and control module in the stage of deblocking training. In the ablation study, discuss the impact of tuning parameters in deblocking training.
The effectiveness of SAFM. To demonstrate the effectiveness of SAFM, we replace SAFM with direct summing of motion and content control. The results in Fig. 7 show that the direct summing fusion cannot accurately control the motion in some complex motion trajectories, e.g., wavy lines. In comparison, using SAFM can help decouple content and motion control in the editing area, achieving more accurate trajectory guidance.
The effectiveness of time adaptation in SAFM. We remove the time condition in the SAFM module, ie., using the same weight map I' fuse content and motion control in each diffusion sampling step. The results in Fig. show that not distinguishing I in different sampling steps leads to unsatisfactory artifacts at the boundary of the editing area.
Tuning parameters in deblocking training. Although the training in stages I and 2 enables the control module to have good local motion control capabilities, we find that there is still an ignoring of motion control in the training of stage 3, i.e., deblocking training. As shown in Fig. 7 the local motion control capability is degraded after we tune the entire control module in stage 3. Therefore. we optimize part of the parameters to maintain the prior of local motion control. Experiments show that fine-tuning spatial layers still triggers the ignoring of motion control. In comparison, fine-tuning key embedding and value embedding of the temporal layer in the control module and the base model has minimal impact on local motion control capability. The edited and unedited areas are also harmoniously fused. More ablations of tuning parameters presented in Appendix.

---

5 Conclusion
In this paper, we aim to solve the problem of local video editing. The editing target includes visual content and motion trajectory modifications. To the best of our knowledge, this is the first attempt at this task. In this new task, We find a coupling problem between unedited content and motion customization. Directly training these two control conditions on the video generation model will cause the ignoring of motion control. To address this issue, we develop a three-stage training strategy to combine these two conditions coarse to fine. In addition, fusion module to further decouple unedited content and motion- costion- content in different diffusion sampling steps and spatial locations. Extensive experiments demonstrate that Re Video has promising performance on several accurate video editing applications, i.e. (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also easily extend these applications to multi-area editing without specific training.
design spatiotemporal adaptive
Limitations. Although our method can regenerate local areas of the video, the regeneration quality is limited by the base model. In some scenarios where the generation prior of SVD is not ideal, some unexpected artifacts may occur in the editing results.
References
[1] http://www.pika.rt/
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder end-to-end retrieval. In Proceedigs of the IEEF/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[3] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference computer vision, pages 707-723. Springer. 2022.
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint 1551, 2023.
[5] Andreas Blattmann. Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[7] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13206-23211 2023.
[8] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun. Shaoshu Yang, Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality vidco generation. arXiv preprint arXiv:2310. 19512, 2023.
[9] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia. Xuefeng Xiao, and Liang Lin. Control- a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305. 13840, 2023.
[10] Jiaxin Cheng, Tianjun Xiao, and Tong Hc. Consistent video-to-viden transfer using synthetic dataset. In The Twelfth International Conference Learning Representations, 2023.
[11] Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-viden editing. arXiv preprint arXiv:2310.0592,, 2023.
[12] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
10

---

[13] Patrick Esser, Johnathan Chiu, Parmida Arighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[14] Patrick Esser, Johnathan Chiu, Parmida Atihhehchian, Jonathan Granskog. and Anastasis Germanidis. Structure and content guided video synthesis with diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[15] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arViv: 10373, 2023.
[16] Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjic Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point correspondence. arXiv preprint trXiv:2312.02083 2023.
[17] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint m0Xiv:2307.04722. 2023.
[18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023:5 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[20] Yash Jain, Anshul Nasery. Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusion. arXiv preprint arXiv:2312.07509, 2023.
[21] Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M Rehg, and Pinar Yanardag. Rave: Randomized noise shuftling for fast and consistent video editing with diffusion models. arXiv preprint 241444322..22 2023.
[22] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better track together. arXiv preprint arXiv:2307.07635, 2023.
[23] Tero Karras, Milka Aittala. Timo Aila, and Samuli Laine. Elucidating the design space of diffision- generative models. Advances in Neural Information Processing Systems, 266267655-2677,, 2022.
[24] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG), 40(6): 12, 2021.
[25] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero, Text-to-imas image diffusion models are zero-shot video generators. In Proceedings of the 1EEE/CVF International Conference on Computer Vision, pages 15954-15964, 2023.
[26] Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play framework for any video-to-video editing tasks. arXiv preprint arXiv:2403, 14468, 2024.
[27] Feng Liang, Bichen Wu, Jialiang Wang. Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. arXiv preprint 23 2023.
[28] Ilya Loshchilov and Frank Hutter. 17X.007711.01101,00 2017.
Decoupled weight decay regularization.
arXiv preprint
[29] Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-baase video generation. arXiv preprint 24X1:2401.008000 2023.
[30] Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, and Bryan Russell. Customizing motion in sext-to-vider diffusion models. arXiv preprint arXiv:23/2.04966, 2023.
[31] Chenlin Meng. Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 20..0.200..01073 2021.
[32] Chong Mou, Xintao Wang. Jiechong Song, Ying Shan, and Jian Zhang, Dragondiffusion: Enabling drag-style manipulation diffusion models. arXiv preprint 222222377002211,1 2023.
11

---

[33] Chong Mou, Xintao Wang, Jiechong Song. Ying Shan, and Jian Zhang. Diffeditor; Boosting accuracy and flexibility diffusion- based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8488-8497, 2024.
[34] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296-4304, 2024.
[35] Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjic Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. arXiv preprint arXiv:2335. 10973, 2023.
[36] Chenyang Qi. Xiaodong Cun. Yong Zhang, Chenyang Lei, Xintao Wang. Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15932-15942, 2023.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 2021.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 06225,,, 2022.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 11.36479-34944 2022.
[41] Xiaoyu Shi, Zhaoyang Huang, Yun Wang, Weikang Bian, Dasong Li. Yi Zhang, Manyuan Zhang, Ka Chun Cheung. Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. arXiv preprint arXiv:2401. 2024.
[42] Uriel Singer, Amit Zohar, Yuval Kirstain, Shelly Sheymin, Adam Polyak, Devi Parikh, and Yaniv Taigman. Video editing via factorized diffusion distillation. arXiv preprint 2403::2403.99334 2024.
[43] Yule Sun, Ang Lu, and Lu Yu. video. IEEE signal processing letters, 24(9):1408 1412, 2017.
quality evaluation for omnidirectional
[44] Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.011 01566. 2024.
[45] Qian Wang. Weiqi Li, Chong Mou, Xinhua Cheng, and Jian Zhang, 360dvd: Controllable panorama video generation with 360-degree video diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6913-6923, 2024.
[46] Wen Wang. Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot vidco editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599 2023.
[47] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2023.
[48] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint 23X 2023.
[49] Yujic Wei, Shiwei Zhang, Zhiwu Qing, Hangjic Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Drcamvideo: Composing your dream videos with customized subject and motion. arXiv preprint 213.44331 2023.
[50] Bichen Wu, Ching Yao Chuang. Xiaoyan Wang, Yichen Jia, Kapil Krishnakumar, Tong Xiao, Feng Liang, Licheng Yu, and Peter Vajda. Fairy: Fast parallelized instruction instucction-uidde guided video-to- voieo synthesis. arXiv preprint arXiv:2312.1385. 2023.
12

---

[S1] Jay Zhangjie Wu, Yixiao Ge. Xintao Wang, Stan Weixian Lei. Yuchao Gu. Yufei Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 1623-7633, 2023.
[52] Wejia Wu, Zhuang Li. Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. arXiv preprint Xrx::24033.074 2024.
[53] Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Y He, Liu, Chen, X Cun, X Wang, Y Shan, et al. Make-your- Customized video generation using textual and structural guidance. IEEE Transactions Visualization and Computer Graphics. 2024.
[54] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang. Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with vidco diffusion priors. arXiv preprint arXiv:2310. 12100 2023.
[55] Wilson Yan, Andrew Brown, Picter Abbeel, Rohit Girdhar, and Samanch Azadi. Motion-conditioned image animation for video editing. arXiv preprint (rXiv:2311.8882. 2023.
[56] Danah Yatim, Rafail Fridman, Omer Bar Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. arXiv preprint arXiv:2311.77000 2023.
[57] Shengming Yin, Chenfei Wu, Jian Liang, Jic Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: emeegrained control video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308. 08089, 2023.
[58] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IFEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[59] Shiwei Zhang, Jiayu Wang. Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:231 1,0445, 0,04145,
2023.
[60] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian, Controlvideo: Training-fice free controllable text-to-vidco generation. arXiv preprint arXiv:2305.13007 2023,
[61] Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-viden diffusion models. arXiv preprint arXiv:2310.088465 2023.
[62] Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lec. Make-a- prottagnoot Generic video editing with an ensemble of experts. arXiv preprint arXiv:2305.0m 08850, 2023.
13

---

A
Appendix
Sparsify
DDEVO
VIDEOO
Threshold
Probabilistic sampling
L VIDEVO
D MIDEVO
Figure 8: The trajectory sampling pipeline in Re training.
A.1 Details of Trajectory Sampling
As described in main paper, trajectory sampling in the training process includes three stages, ie., sparsifying, threshold filtration, and probabilistic sampling. We present the visualization this pipeline in Fig. 8 In sparsifying, we grid [57] to sparsify the dense sampling points, obtaining Niuit initial points. In threshold filtration, we use the mean of the tracking length these Ninit points the threshold ITA filter points with large motion. In probabilistic sampling, the normalized lengths of these sampling points as sampling probabilities to sample N points from them. N randomly selected from 10.
Bokonnss leAN
Irregutar Editing Area
Edited Frame
Edited Frame
Edited Frame 14
Figure 9: The robustness of our Re Video for irregular editing areas.
A.2 Robustness for Irregular Editing Area
In our main paper, we demonstrate the robustness of method on multi-area editing without specific training. In Fig. 9 we present another robustness our method for irregular editing areas. As can be seen, though method is trained rectangular editing areas, it has stable content and motion editing capabilities when facing a hand-drawn irregular editing area.
A.3 Extension Application of Video Inpainting
In some video editing scenarios, specifying motion trajectories is challenging, such as when erasing an object video. We find that by leaving the motion control empty in these cases, our method can automatically generate the motion state of the editing area and match it with the motion in the unedited area. This due to our inherent capability to predict the motion in the editing area via unedited content. An example is presented in Fig. 10
14

---

Original
Editing wio motion control
Figure 10: The application of object removing without specifying the motion trajectory. Note that the background not static is moving the right.
Dynamic Background with Complex Lighting
Scene Change
First
Center frame
Last
Figure The editing results some complex scenarios. The first row has dynamic background complex lighting, and the second row has scene change.
A.4 Additional Discussion in Complex Scenarios
In Fig- II| we try some complex scenarios. The result shows that our method can handle dynamic lighting and texture, but scene change affects the content quality, which is failure case.
A.5 Details of Content Encoder and Motion Encoder
At the input of our spatiotemporal adaptive fusion module (SAFM), two encoders, i.e., Eg and Em- separately encode the content and motion conditions. E, and E,, have the same low-cost structure. This structure contains three sub-blocks, each consisting of convolution and downsampling operation, mapping the condition map to the same size as the latent 2,-
A.6 More Editing Results of Re Video
In Fig. 12 we present more editing results of our ReVideo, including adding new objects to video, modifying the motion trajectory of existing content in the video, editing existing content while maintaining the motion trajectory, and multi-region editing. As can be seen, the editing results achieve the editing goals, and motion control and content control coexist harmoniously.
15

---

Original Video
Editing Results
Motion
Control:
Motion
Control:
Motion
Control:
Motion
Control:
Motion
Control:
Motion
Control:
Motion
Control:
Motion Control:
Motion
Control:
Motion Control:
Figure 12: More editing results of our ReVideo
A.7 Necessity of Fine-tuning the Base Model
In the training process of stage 3, we fine-tune the key embedding We and value embedding W, of the temporal self-attention layer in the control module and base model. In Fig. [3] we demonstrate the necessity of fine-tuning the base model in two scenarios. Specifically, in some complex scenarios, such as the forest shown the first row, not fine-tuning disjunction, e.g., the misaligned tree trunk. The second row shows the case where there is a high coupling between the unedited content and editing content, such as retaining the motion of hair and only editing the facial movement. Fine-tuning the base model can alleviate artifacts brought by the
base model would result content
16

---

Content
Reference
Image
Content
Reference
Image
Tuning W and W, in control module
Tuning WR and W, control module and base model
Figure 13: The necessity of fine-tuning key embedding and value embedding in the base model, i.e, SVD.
motion conflict between the highly coupled edited and unedited areas. Therefore, jointly fine-tuning base model helps to produce more harmonious editing results.
A.8 More Frame Editing
In addition to editing a fixed number of frames based on the base model SVD, our ReVideo can process more frames. implementation, we use the sliding window strategy, where the last frame of the editing result in the previous window is used as the reference image for the current window. Fig. shows the editing results of method on 9-second video containing 90 frames. One can see that our Re Video broadcasts the editing of the first frame into the 90-frame video while controlling the motion of the new content be consistent with the original video. At the same time, we also observe that the error accumulation affects the editing quality. This is an inherent issue in long video editing, and more powerful base model can alleviate this issue.
Frame 1
Frame 10
Frame 30
Frame 90
Figure 14: The ability of our Re Video to extend the number of editing frames. The results demonstrate the performance of our Re Video in processing 9-second video containing 90 frames.
A.9 More Discussion Pika in Adding New Object
In the comparison section of the main paper, we find that Pika 1 has weak editing capabilities in adding new objects. To eliminate the influence of randomness, we generate 5 times with random seeds in the case of adding new objects in Fig. [5] i.c. adding plane in the sky. We set the strength of text
17

---

Foeme stoed
slowstatte
PiKa
Pika
Secd-8195767444755444
Seed-42
Editing interface of Pika
Pika
Pika
Pika
Stedf2244199172006166688
Sedd-584996802737733
Sedd-275587336662371
Figure 15: More failure cases of Pika in adding objects to video. We set the text consistency control parameter to the highest level during testing. The editing target is to add a plane in the sky.
consistency the highest level, but all 5 editing results failed. This indicates the inaceuracy text as the control signal of local redrawing. In comparison, editing the first frame and then broadcasting the entire video can accurately specify the content of the editing area.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In the abstract and introduction, we detail our task objectives, weaknesses in the current methods, and the improvements we propose for these weaknesses. Therefore, the scope and contributions in abstract and introduction are clear.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the end part of our main paper, we discuss the limitations of this paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We conduct experimental verification for each assumption in the paper and display the experimental results. For instance, in the insight section (Sec. 3.2) of this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
This paper provides a detailed description of the model structure and training details.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
20

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code for this paper is open-sourced at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
/ctee//guuddeeeodeduuubmiiii wwwpsubmmssiioPPooooo ssioniolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In the experiment section the paper, we provide detailed description of the training and testing details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
For the task of video editing, the error bar uncommon, and there is currently no mature error evaluation system in this community.
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In experiment details, we provide the time complexity and computational cost in training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ftttpp:://neurppee http:///wcc/ccttcccsssudaaaaaaaa
Answer: [Yes]
Justification:
We have carefully reviewed the NeurTPS Code of Ethics to ensure that submission complies with all regulations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
22

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The pre-trained model used in our paper is image-to-vine video model, and the content of the generated video is specified by the input image. The video editing in this paper is achieved by editing the first frame, and the image editing method is not part of this paper. Therefore, there is no issue of data misuse in this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
23

---

Answer: [Yes]
Justification:
The models and data used in this paper are open-sourced and authorized.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwpprswithcoteeoo thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- ee-pacagedd, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper aims to locally edit a video, thereby does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
24

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper aims to locally edit a video, thereby does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "h3FIUCzk0A", "reviewer": "Reviewer_F4MH", "summary": "This paper presents ReVideo, a new approach for precise local video editing of both content and motion. It introduces a coarse-to-fine training strategy to progressively decouple content and motion control, and a spatiotemporal adaptive fusion module to integrate them effectively. Experiments show ReVideo can modify local video content, customize motion trajectories, or change both simultaneously, and extend to multi-region editing.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This appears to be the first attempt at exploring local editing of both content and motion in videos using diffusion models. Being able to modify content and motion trajectories in specific regions is a novel capability compared to prior work.\n- The proposed three-stage coarse-to-fine training strategy to progressively decouple content and motion control is an interesting technical approach to deal with the core challenge.\n- The spatiotemporal adaptive fusion module is another novel component to integrate the content and motion conditions across sampling - steps and spatial locations.\n- Extending the approach to allow multi-area editing without requiring specific training demonstrates flexibility.\n- Most of the visual and quantitative results show improvements over prior methods \n\nOverall, this paper addresses a timely and important topic with significant potential benefits for the community. Despite some weaknesses, the reviewer recommends acceptance, considering this is a relatively new area and the paper presents promising results. The score may be adjusted based on the quality of the rebuttal.", "weaknesses": "## Practicality of the Editing Workflow\n\nThe current editing interface requires users to specify both a target content image and a set of motion trajectories. While this allows for fine-grained control, it may not be the most intuitive or efficient workflow for common editing tasks. Consider the scenario of object removal - the user would need to carefully craft a content image with the object removed and ensure that the remaining motion trajectories are consistent. An alternative approach could be to directly specify the regions to remove and have the model infer the appropriate content and motion changes automatically. The paper would benefit from a more detailed discussion of the practical trade-offs and usability considerations of the proposed editing framework.\n\n## Limited Motion Control\nWhile the method allows for editing the motion of individual objects, it assumes that the overall scene motion (camera movement, background motion) remains fixed. This limits the applicability of the approach in scenarios where the goal is to modify the global motion patterns (e.g. stabilizing shaky footage, changing the camera viewpoint).\n\n## Precise Placement and Key Contributions of this Paper\n\nWhile the individual technical components (e.g. coarse-to-fine training, adaptive fusion) are well-motivated, it's worth considering whether similar strategies have been explored in related domains. For instance, progressive training to handle multi-factor variation has been used in GANs, and spatially-adaptive normalization is common in style transfer. Drawing more connections to such related work would clarify the novelty of the specific adaptations made here.\n\n## Content-Motion Entanglement\n\n- The key technical contribution of the paper is the decoupling of content and motion information through a coarse-to-fine training strategy. However, it's not clear if this decoupling is complete or if there are still some residual entanglements between the two factors. For instance, the edited content may still contain some motion information that could interfere with the specified motion trajectories, leading to artifacts or inconsistencies. A more thorough analysis of the content-motion separation and its impact on the editing quality would be informative.\n\n- Is decoupling content and motion the only way to address the issue - could a joint representation learning approach work instead? Acknowledging alternate strategies would help justify the chosen approach.\n\n- **Figure 4 is not very intuitive. It would benefit from additional justification, theoretical analysis, and insights into why such a simple composition from two videos is effective.** This is a key concern.\n\n## Multi-area Editing \n- The extension to multi-area editing is a nice addition, but the paper could go further in characterizing the challenges involved. Are there issues with preserving global coherence across multiple edited regions? How does the method scale with the number of regions? Providing such details would give a more complete picture of the capability.\n\n## Clarity and Reproducibility\n- Implementation details: There are some missing specifics that could hamper reproducibility. For instance:\n\n   - How exactly are the editing regions defined during training - what is the procedure for randomly sampling them?\n   - What metrics are used for the \"threshold filtering\" of motion trajectories and how were the thresholds chosen?\n   - Are there any data augmentation, regularization or optimization tricks used during training?\n\n## Evaluation Metrics\n\nThe quantitative evaluation relies primarily on low-level metrics like PSNR and LPIPS, which may not fully capture the perceptual quality and coherence of the edited videos. Additional metrics could provide a more comprehensive assessment:\n\n- Metrics that specifically measure the consistency of the edited regions with the target content and motion (e.g. using an object detector or tracker).\n- Metrics that evaluate the temporal stability and smoothness of the edited videos (e.g. some metrics that are used in video inpainting tasks, Please refer to [this repo](https://github.com/MichiganCOG/video-inpainting-evaluation) for details).\n- Human evaluations of the overall realism, coherence, and faithfulness to the editing inputs (e.g. through user studies).\n\n\n\n## Robustness Evaluation and Ablation Studies\n\nWhile the paper does include ablations for a few key components (e.g. SAFM, training stages), there are other design choices that are not fully explored. For instance:\n\n   - How important is the choice of motion representation (trajectory vs. alternatives)? Testing with different motion inputs would reveal the sensitivity to this factor.\n   - What is the impact of the trajectory sampling strategy and hyperparameters? Varying the number and selection of trajectories could provide insight into the robustness.\n   - How does the performance vary with the size and shape of the editing regions? A systematic evaluation across different region properties would be informative.\n   - Only the end-to-end video editing pipelines are compared, but not the individual technical components. For instance, how does SAFM compare to simpler fusion schemes used in prior work?\n  - Input noise and perturbations (e.g. in the content image or motion trajectories)\n\n## Dataset Complexity \n\n-  While the approach achieves good results on the chosen datasets, it's unclear how well it would generalize to more complex video content (e.g. with dynamic backgrounds, scene changes, occlusions etc.). Discussing the potential failure modes and current limitations would help scope the contribution appropriately.\n\n- The examples shown in the paper are largely limited to simple object-level edits in relatively constrained scenarios (e.g. clean backgrounds, single objects). It's unclear how well the method would perform on more challenging videos with complex scenes, multiple objects, occlusions, camera motion, etc. Testing on a wider range of video complexity would help establish the generality of the approach.\n\n## Editing Scenarios\nThe paper demonstrates a few key editing applications (e.g. object addition/removal, motion editing), but there are other important scenarios that are not explored, such as: performing semantic-level edits (e.g. changing the action or interaction between objects).\nShowcasing the method's performance across a fuller range of editing tasks would demonstrate its versatility.\n\n## Open Source\nWill the code for training and inference be released?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1 Practicality of Workflow\n**Fig.1** and **Fig.2** in the attached PDF show that our method can still produce smooth results without specifying trajectory. This is due to our inherent capability to predict the motion in editing area via unedited content, enabling automatic motion generation when trajectory is difficult to specify. However, the content reference is essential in current framework. We will discuss the practicality in our paper.\n## Q2 Limited Motion Control\nWe need to clarify that global motion pattern in the editing region is not our editing target but needs to be consistent with unedited areas. The consistency is automatically produced without fixed motion assumption.\n## Q3 Precise Placement\nThanks for the suggestion. We will discuss related works in progressive learning and multi-condition control.\n## Q4 Entanglement\n- Residual entanglement. Yes, it still exists and can be observed by reducing the weight of motion condition (**Fig.3** in the PDF). We believe it is useful and mild. **(1)Useful.** Inspired by **Q.1**, the entanglement can automatically produce the motion, when it is difficult to specify the motion of editing region. **(2)Mild.** Various editing results show that this entanglement is almost unnoticeable in default inference.\n- Other solutions. Yes, that's certain. But we think decoupled representations are necessary.\n- Decoupling training. The insight of why it works mainly comes from Sec.3.2 in our paper. In Sec.3.2, we use several toy experiments to show the strong coupling issue. The method in Fig.4 ensures no motion correlation between the editing and unediting region, preventing the model from estimating the motion in editing region via unedited content. We will add a more intuitive explanation in the paper.\n## Q5 Multi-area Editing\nIn **Fig.4** in the PDF, we select a complex wavy line as the editing target and gradually add editing regions. We compute editing accuracy by Euclidean distance between the trajectory in editing result (extracted by CoTracker) and the target. Results below show slight impact of region numbers on accuracy and consistency. We will add this in our paper.\n|Region Number|1|2|4|8|10|\n|-|:-|:-|:-|:-|:-|\n|Acc.|4.68|4.57|4.67|5.15|5.40|\n|Temporal Consistency|0.9926|0.9920|0.9918|0.9905|0.9908|\n## Q6 Clarity\nIn Sec.3.2 and A.1 of our paper, we describe the trajectory sampling. Threshold $l_{Th}$ is empirically defined as the mean length of sparsed trajectories. Each video has one editing area, the minimum bounding rectangle of sampled trajectories. It is expanded to at least 64x64 pixels. Data augmentation includes sampling frames at varying intervals and randomly reversing the video. We will update these details in our paper.\n## Q7 Metrics\n- Measurement in editing region. Using a tracker helps evaluate our motion control, especially in ablation. Other methods do not have motion control. The content consistency in editing region can be compared among different methods. Below table shows CLIP Score between the editing region and target description.\n|Methods|InsV2V|AnyV2V|Pika|ReVideo|\n|-|:-:|:-:|:-:|:-:|\n|Local CLIP Score|0.2255|0.2378|0.2402| 0.2516|\n- Temporal stability. In our paper, we use CLIP-Image score to evaluate the temporal consistency, which is used in several related works (e.g., AnyV2V, Pix2Video). The reference repo is helpful, but most metrics need a target video, which is unachievable in editing task. After reviewing, PSNR Consistency can measure the temporal stability, as shown below.\n|Methods|InsV2V|AnyV2V|Pika|ReVideo|\n|-|:-:|:-:|:-:|:-:|\n|PSNR Consistency|34.94|34.12|40.51|39.26|\n- Human evaluation. In Tab.1 in our paper, we conduct a user study for overall video quality and editing accuracy. We agree that more evaluation terms is helpful. We will extend it accordingly.\n## Q8 Robustness and Ablation\n- Choice of motion representation. Yes, trying more motion representations is helpful. But we only find the trajectory that meets our sparse and interactable requirement.\n- Impact of trajectory sampling. Training with significantly moved trajectory is crucial. Below table shows its significant impact. In contrast, trajectory numbers have little impact.\n||Random sampling|ReVideo|\n|-|:-:|:-:|\n|Acc.|13.38|5.21|\n- Impact of size and shape. In Sec.A.2 of our paper, we show our support for irregularly shaped regions. Below table shows the performance after randomly expanding the width/height of editing regions on test set. The size and shape have little impact on performance, even when the editing region is entire video.\n||Default|+$l\\in$[32,64]|+$l\\in$[64,128]|Global|\n|-|:-:|:-:|:-:|:-:|\n|Acc.|5.21|5.06|5.32|5.48|\n- Ablation of SAFM. \"w/o SAFM\" in Sec.4.3 is a simpler fusion scheme that fuses motion and content embedding by adding and convolution. It fails in complex motion control. This result and weight visualization in Fig.5 can verify the need of region-adaptive fusion.\n- Input perturbations. We try two content perturbations in **Fig.5** in the PDF. The base model's prior on high-quality videos can mitigate content degradation, and motion control works normally. If the trajectory is perturbed, the motion will follow the perturbation.\n\nWe will update the above discussion in our paper.\n## Q9 Dataset Complexity\nIn **Fig.6** and **Fig.7** in the PDF, we try some complex scenarios. Our method can handle dynamic lighting and texture, but scene change affects the content quality, which is a failure case. We will include a discussion in our paper.\n## Q10 Editing Scenario\nOur test has simple semantic interactions, such as wearing a hat or glasses. In **Fig.8** in the PDF, we test a more complex interaction where two balloons collide. They do not bounce off each other but overlap. This failure is because our base model (SVD) is weak in physical world modeling. We believe our method can achieve such interactions with a more powerful base model. We will discuss this failure in the paper.\n## Q11\nYes, we will release all code.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKgfI7TS8x", "reviewer": "Reviewer_ZSh9", "summary": "The paper presents a video editing method that enables precise localized adjustments to content and motion within specific areas of a video. It introduces a three-stage training strategy and a spatiotemporal adaptive fusion module to integrate edits across frames and locations effectively. This method allows for complex editing tasks such as changing content while maintaining motion, adding new motion to static content, and simultaneously modifying both elements.", "questions": "- What kind of GPU does one need to perform training and testing? \n- Will the authors release the training and testing code along with pre-trained models upon acceptance?", "limitations": "Quality limited by SVD.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a novel challenge of editing both content and motion in specific video areas and combines techniques from diffusion models and video editing to achieve nuanced control.\n- The three-stage training strategy enhances the robustness and effectiveness of the edits, supported by experimental validation that demonstrates superior performance compared to existing methods.\n- The paper is well-organized and clearly explains complex concepts, including the innovative spatiotemporal adaptive fusion module and detailed training strategy.", "weaknesses": "- The decoupling training could cause some artifacts. Although the paper demonstrates these artifacts could mostly be alleviated by deblocking training. I can still see some blocky/unnatural results in the result videos.\n- The training is quite complicated and separated into three stages. I feel the training strategy could 'overfit' this particular video dataset.\n- This method is more like a direct combination of video diffusion and ControlNet.\n- More detailed implementation specifics, particularly regarding parameter settings and the architecture of the spatiotemporal adaptive fusion module, are needed.\n- The method's computational demands and potential scalability issues are not adequately addressed. For example, what kind of GPU does one need to perform training and testing? \n- The paper focuses heavily on technical aspects with less consideration of user interaction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1 About Artifact\nWe agree that our method still has room for improvement. We want to clarify this concern from two points:\n\n(1) **The challenge of this task and our novelty.** Our method is the first attempt at local content and motion editing for videos. In Section 3.2 of the main paper, we conduct extensive toy experiments to demonstrate the challenge of this task, as it requires overcoming significant coupling between control conditions. In implementation, we meticulously design both the training strategy and the model architecture to address this challenging task.\n\n(2) **Editing Performance.** We present several useful applications and high-quality editing results in the paper, which is unachievable for existing methods with a significant gap. Although some artifacts exist in the edges, as you concerned, they are often difficult to detect in the result videos. \n\nWe will continue to improve our method in the future.\n\n## Q2 Overfitting in Training\nWe need to clarify that although the three-stage training strategy might be a bit complicated, our extensive experiments in the paper (Section 3.2) demonstrate that the problem we face is indeed challenging. Therefore, we believe that using a progressive learning approach to tackle this difficult problem is reasonable, and the role of each training stage is demonstrated in the paper. We would greatly appreciate any references or suggestions you could provide to help us improve our current training strategy. \n\nIn Section 4.1, we show that our training data includes 10 million videos from the WebVid dataset. Therefore, overfitting is not a concern under such a dataset scale. The test results also demonstrate the diversity of the editing scenarios.\n\n## Q3 Like a ControlNet\nWe need to clarify that ControlNet is a commonly used method for condition inputs. In this paper, we also introduce that our method uses conditional injection of ControlNet. Moreover, combining video diffusion with ControlNet cannot solve our problem, and it is not a contribution of this work. The key contribution of our work lies in the proposing of a novel training strategy to solve the unexplored fine-grained video editing task. Additionally, we design a spatiotemporal adaptive fusion module to improve the fusion of motion and content control.\n\n## Q4 Detailed Implementation of SAFM\nOur SAFM consists of four convolutional layers, each followed by a SiLU activation layer, with a Sigmoid function applied at the end. The SAFM has 2 million parameters, which is negligible compared to the 671 million parameters in the SVD model. We will add this detailed description to the paper.\n\n## Q5 Computation Demand\nIn Section 4.1 of the paper, we describe that our training is conducted on four A100 GPUs. During the inference, it requires 22GB of VRAM to edit a 16x768x1344 video. In the table below, we present a comparison of computational complexity for different methods. The experiment is conducted on an A100 GPU, with the editing target being a 16x768x768 video. The results demonstrate the advantage of our method in inference efficiency. Therefore, we believe our approach is efficient and flexible. We will add a detailed description of complexity demand in the paper.\n|                | InsV2V | AnyV2V                     | ReVideo |\n|-|:-:|:-:|:-:|\n| Inference Time (s) |  132   | 303 (DDIM Inv) + 80.9 (Inference) |  26.8  |\n\n## Q6 Focuses Heavily on Technical Aspects\nWe want to clarify this concern via the interactive design and capability of our method. To offer an intuitive user interaction, we chose the sparse and easy-to-interact trajectory lines as the motion control. In the editing mode, users can selectively customize content or motion. In scenarios where motion control is difficult to specify, our method can also automatically generate the motion in the editing region. Therefore, our method is user-friendly in controlling inputs and functions. We will include a detailed description of user interaction in the paper.\n\n## Q7 Open Source & Limitation \nYes, we will open-source all the code. \n\nWe need to clarify that our method is not limited to a specific base model. The reason we use SVD as the base model is that SVD is the best open-source model currently available to us. Using a better base model can further improve our editing results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J76UKADBko", "reviewer": "Reviewer_YHcN", "summary": "ReVideo presents a novel view of video editing by modifying content with input trajectory to create new content. It designs a three-stage strategy to wrestle out the problem of ignoring motion control when direct training. The main contribution of this work relies on the new task of editing motion via user-specified trajectory while keeping the original video movement. The editing results are superior and photorealistic.", "questions": "1. The method to edit the first frame needs to be declared.\n2. Non-rigid motion-like side view wheels movement of cars.\n3. Qualitative video comparisons with other methods. \n4. The inference time/training cost comparison with other similar methods.\n5. What about ReVideo performing in editing multiple objects simultaneously in the same video?\n6. Can ReVideo work on text-to-video generation models?", "limitations": "no significant limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The first video editing work on creating new motion and content.\n2. Good writing; the paper is easy to follow, and the motivation and three-stage training strategy on decoupling content and motion control is reasonable. The proposed SAFM learned a dynamic fusion weight at different timesteps.\n3. The editing results are photorealistic and adhere to the original motion or follow user-specified trajectory with no artifacts.", "weaknesses": "1. The author did not provide the method or explanation of how ReVideo edits the first frame, making the total editing pipeline not end-to-end for users.\n2. Part of the original video motion, like mouth movement in the Zuckerberg->robot (head6) and tail movement in dog->lion, is not kept in the edited video.\n3. I would like to know how the drag-based editing method handles non-rigid motion, such as the rotation of car tires from a side view. In examples like sea2 and sea2_2, where a shark and a dinosaur are added, the limbs of the animal seem unable to move, making the video look unrealistic. However, in soccer and some human-centric examples, the legs of dogs and people can move normally. Therefore, I would like the authors to add an example of a vehicle moving on the road from a side view, including the movement of the wheels, to address my concerns. This may be a limitation of the drag-based method.\n3. There is no quantitative comparison of the ablation study; I understand that the image results in Fig 7 are clear, but only one video qualitative ablation is not reasonable. \n4. There are no qualitative video comparisons with other methods in the supp or project page, but only Fig 6, and the automatic metrics are worse than pika even though I understand the clip scores are not accurate, which can not reflect temporal consistency accurately. I suggest the author supply the comparison video between Revideo and other methods in the rebuttal phase.\n5. The training cost of three stages: even though Revideo makes great progress in creating new motion, training cost like GPU costs, time costs, memory costs and so on, is still a problem since users prefer to edit a video in a zero-shot manner when using a pretrained video generation model and the compared methods like AnyV2V is training-free.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1 The Editing of the First Frame\nThanks for this suggestion. The editing method for the first frame is arbitrary, like the setting in AnyV2V. The results presented in the paper utilize text-guided image inpainting tools and InstructPix2Pix. Note that in our framework, editing the first frame is not mandatory. Users can choose to keep the content and modify the motion of the video. We will add a detailed explanation of the editing workflow in the paper.\n## Q2 Some Motion is Not Kept\nWe need to clarify that our editing framework uses sparse motion control (trajectory lines) in the editing region, which is challenging to handle slight motions as you mentioned. To support such motion control, our framework may need to incorporate dense motion representations, such as optical flow. A potential solution is to mix sparse trajectories with dense optical flow during the training process, allowing our method to apply sparse and dense motion control selectively. We will explore this design in future work.\n## Q3 Non-rigid Motion\nThank you for this constructive suggestion. We add a video of editing a side-view driving car in **Fig.9** in the attached PDF. We can see that while the car's rigid body is moving, the car's wheels are also rolling (although it might not look entirely realistic). Similar phenomena can be observed in some demo videos in our supplementary, such as adding a tank on the desert. The tank's wheels are moving in a regular pattern, and the shadow behind the tank moves accordingly. We believe that this phenomenon, including the natural motion of dogs and humans that you mentioned, is due to the priors of physical world that the base model (SVD) learned from large-scale video data. The unnatural movement of the shark is also caused by the limitations of SVD's prior. To verify this, we use SVD to perform image-to-video generation based on the shark image. The result in **Fig.9** in the attached PDF shows that while the water surface has natural ripples, the shark remains stationary. We believe that by using a more powerful base model (such as the Sora architecture), we can achieve more natural editing results.\n## Q4 Quantitative Comparison of the Ablation Study\nTo address this concern, we measure the editing accuracy under different settings in our ablation study, as shown in the table below. We use a point tracking model (CoTracker) to extract trajectory in the edited results and then calculate the Euclidean distance with the target trajectory. We will add this quantitative comparison to our paper.\n|    | w/o SAFM | SAFM w/o time adaptation | Tuning all control module in stage3 | Tuning spatial layers in stage3 | ReVideo |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| Accuracy (Pixel)      |   37.34  |           8.92            |               42.26                |              44.78              |   5.21  |\n## Q5 Qualitative Video Comparisons\nYes, it is necessary to provide comparison videos of different methods. However, due to the rebuttal policy, we cannot provide external links. Instead, we select two videos to present in **Fig.10** in the attached PDF. One can see that AnyV2V has gradual changes in video content in some editing samplings, leading to instability. Pika and AnyV2V both tend to produce static editing results, due to the lack of motion control. While such static results may achieve higher scores in CLIP, it is not natural enough. Additionally, Pika tends to fail in some challenging editing scenarios, such as transforming a part of a lizard into a robot.\n## Q6 Training Cost\nWe agree that our implementation involves some training costs compared with training-free methods. However, our approach offers significant performance gains:\n\n(1) Our method allows precise customization of local content and motion in videos, which is not achievable with existing training-free methods.\n\n(2) Our method can produce high-quality and stable outputs. Training-free methods often struggle with video generation quality and stability of the edited content.\n\n(3) Our method has an advantage in inference complexity. Some training-free methods, such as AnyV2V, require a long time of ddim inversion to ensure the editing quality. The table below shows the time costs of different methods during the inference stage. The experiment is conducted on an A100 GPU, with the video resolution being 768x768. Results show that our method has significantly lower time costs compared to other methods. Therefore, we believe that our training cost (4 A100 GPUs) is reasonable and necessary. We will add the complexity analysis to our paper.\n|                | InsV2V | AnyV2V                     | ReVideo |\n|-|:-:|:-:|:-:|\n| Inference Time (s) |  132   | 303 (DDIM Inv) + 80.9 (Inference) |  26.8  |\n## Q7 Performance in Editing Multiple Objects Simultaneously\nIn the demo video in our supplementary, we show examples of editing six individual petals in the same video, as well as editing multiple flowers and balloons in one video. This demonstrates the robustness of our method in multi-object editing. Additionally, **R1** raises a concern in **Q.5** about the impact of the number of editing regions on performance. Results in **R1-Q.5** show that each editing region has a high independence, and the impact of the region numbers is almost negligible.\n## Q8 Whether works on text-to-video models\nYes, it is possible, but the implementation would be more complex than with an Image-to-Video model. The image condition port in the Image-to-Video model can naturally be used as the input of the edited first frame, whereas the Text-to-Video model lacks this port and would require additional training to incorporate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the efforts of all the reviewers, ACs, and PCs. We have carefully read and addressed all concerns. **Since we are limited to 6,000 characters per reviewer during the rebuttal phase, we could only provide brief responses to some questions.** If there are any further issues, we are happy to address them during the discussion phase.

We include the figures and videos in the attached PDF. Please note that you need to use **Adobe Acrobat** to open the PDF to watch the videos.

Meta-review: Overall, the reviewers agree that the paper tackles an unexplored and challenging task, significantly advancing the capability to modify content and motion trajectories in specific regions using diffusion models. The experiments demonstrate promising results and show improvements over prior methods. The reviewers have offered suggestions for revision, including a more precise description of the task scope and editing capabilities, a discussion of potential artifacts introduced by the proposed method, and detailed documentation of the training process. Additionally, to facilitate reproducibility and further research on this new task, it is strongly recommended that the authors release the code for training and inference, as they have committed to doing.
Decision: Accept (poster)