ReVideo: Remake a Video with Motion and Content Control

Authors: chong  mou
mingdeng  cao
xintao  wang
zhaoyang  zhang
ying  shan
jian  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 09:19:46

Abstract: Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-area editing without specific training, demonstrating its flexibility and robustness.

Main paper: 
 introduction: 
 Thanks to the large-scale training data and huge computing power, there have been significant advancements in diffusion-based [19] image and video generation. For personalization purposes, many works add control signals to the generation process, such as text-guided image [39, 40, 38] and video [18, 17, 13, 45] generation, as well as image-guided video generation [4, 54] . Based on these base models, extensive works explore how to transfer their generation capabilities to video editing. Early works based on text-to-image diffusion models implement video editing through zero-shot strategies (e.g., Fate-Zero [36] , Flatten [11] ) or one-shot tuning (e.g., Tune-A-Video [51] ). However, these methods are limited by excessive manual design and a lack of video generation priors. Moreover, text prompt only provide coarse condition, limiting the editing accuracy. Compared to text, more recent methods adopt image conditions which can provide more accurate editing guidance. For instance, VideoComposer [47] generates style-transformed videos by providing spatial attributes (e.g., edge, depth) of the target video and a style reference. DreamVideo [49] and Make-a-protagonist [62] can modify a specific object in the video by providing a reference object. However, these methods still struggle with local editing and introducing new elements, such as adding new objects to a video. Recent work EVE [42] proposes a diffusion distillation strategy to achieve video editing while keeping unedited content unchanged. Nevertheless, the editing region and target are controlled by text, which is challenging in complex scenarios. AnyV2V [26] edit a video by modifying the first frame, enabling accurate customization of local content. Pika [1] can regenerate a specific area in the video by selecting an editing region. Although these methods improve the performance of local video editing, they only focus on visual content editing and cannot customize the motion of new content.Motion is another crucial aspect of video, yet research on video motion editing remains limited. While some methods explore motion-guided video generation using trajectory-based motion guidance (e.g., DragNUWA [57] , DragAnything [52] , MotionCtrl [48] , VideoSwap [16] ) and box-based motion guidance (e.g., Boximator [44] , Peekaboo [20] ), they do not support motion editing. Additionally, other works [56, 30, 61] can transfer motion from one video to another but cannot modify it as well.In this paper, our goal is to accurately edit content and motion in specific areas of a video. We create an easy-to-interact pipeline by setting the content editing as modifying the first frame, with trajectory lines [57] as the motion control signal. Other unedited content in all frames should be maintained in editing results and merged with the editing effect. However, we find that fusing unedited content with motion-customized new content is challenging, mainly for two reasons: (1) Training imbalance: Unedited content is dense and easier to learn, while motion trajectories are sparse and abstract, making them harder to learn. (2) Condition coupling: Unedited content provides both visual and inter-frame motion information, leading the model to rely on it for motion estimation, thereby ignoring the hard-to-learn trajectory lines.To address these challenges, we design a three-stage training strategy to harmonize unedited content and motion-customized new content, enabling harmonious control of different conditions. Besides, we design a spatiotemporal adaptive fusion module to fuse these two conditions at different diffusion sampling steps and spatial locations. Furthermore, our method can compactly inject motion and content conditions into the diffusion video generation through a single control module. With these techniques, users can conveniently edit specific regions in the video by modifying the first frame and drawing trajectory lines. Notably, ReVideo is not limited to single-region editing and can customize multiple areas in parallel.• To the best of our knowledge, this is the first attempt to explore local editing of both content and motion in videos. Our method can also be easily extended to multi-area video editing.• We propose a three-stage training strategy and a spatiotemporal adaptive fusion module to address the coupling of content and motion control in video editing, enabling compact control through a single module.• Extensive experiments demonstrate that ReVideo performs well in several precise video editing applications, including changing content in a specific region while keeping motion constant, maintaining content while customizing new motion trajectories, and modifying both content and motion trajectories. Some examples are presented in Fig. 1 .2 Related WorksQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification:In the abstract and introduction, we detail our task objectives, weaknesses in the current methods, and the improvements we propose for these weaknesses. Therefore, the scope and contributions in abstract and introduction are clear.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
controllable image and video generation: 
 Recent advancements in diffusion models [19, 12] drive the rapid development of image and video generation. In the community of image generation, some notable works, such as Stable Diffusion [39] , Imagen [40] , and DALL-E2 [38] , utilize text as the generation condition. To achieve accurate generation control, some methods, e.g., ControlNet [58] and T2I-Adapter [34] , propose adding control modules on pre-trained diffusion models. Similarly, initial efforts in controllable video generation concentrate on the text condition, such as Video LDM [5] , Imagen Video [18] , VideoCrafter [8] , and AnimateDiff [17] . Recognizing the limitations of text prompts in capturing complex scenarios, some recent works [4, 54, 59, 13] leverage image conditions for a more direct approach. External control modules on pre-trained foundation models are also popular in controllable video generation. Such as video ControlNet [9, 60] extends the ControlNet [58] in image generation to video generation conditioned on a sequence of control signals, like edge maps and depth maps. In addition to spatial structure control, precise temporal motion control is also important in controllable video generation. This process is similar to the drag-based image editing [35, 32, 33] . Several recent works study this topic, such as video generation with trajectory-based motion guidance (e.g., DragNUWA [57] , MotionCtrl [48] , Motion-I2V [41] , DragAnything [52] ) and generation with box-based motion guidance (e.g., TrailBlazer [29] , Boximator [44] , [20] ). These methods perform the control by training extra motion controllers on pre-trained video diffusion models.Due to the lack of training data, the common approach in video editing is via training-free strategies [7, 15, 21, 25, 46, 36] or one-shot tuning [51, 3, 24] . For instance, the prior work Tune-A-Video [51] overfits some diffusion model parameters to a specific video. Then, it uses the overfitting parameters to produce the editing result conditioned on the target prompt. To enable a cohesive global appearance among edited frames, many methods extend the attention module of Stable Diffusion [39] to encompass multiple frames and conduct cross-frame attention. For instance, Pix2Video [7] edits the first frame and performs cross-frame attention of each frame on the first frame to preserve appearance consistency. TokenFlow [15] and Fairy [50] jointly edit a few key frames at each denoising step and propagate them throughout the video based on the nearest-neighbor field extracted from the original video. Inspired by the initial zero-shot image editing method SDEdit [31] , the recent video foundation model SORA [6] achieves video editing by adding noise to the input video and then denoising it under the target description. Although these methods can preserve the general structure of original videos, the information loss and the lack of consistency constraints on the original video make them unfit for precise video editing but suitable for global editing like style transfer.Another strategy is to train a control module to guide the generation with some characters that should persist in the editing result, such as depth [14, 27, 53] , sketch [47] , and optical flow [55] . However, existing methods primarily focus on preserving spatial structure and are unsuitable for precise video editing. In the community of precise video editing, some works, such as InsV2V [10] and the recent EVE [42] , edit the video by providing editing instructions. However, the text-based editing instruction struggles to locate a target region in some complex scenarios. AnyV2V [26] 3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification:In the end part of our main paper, we discuss the limitations of this paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Stable Video Diffusion (SVD) [4] is a high-quality and commonly used image-to-video generation model. To utilize the priors of high-quality video generation, we employ SVD as the base model and add control modules to achieve our editing target. Given a reference image c I , SVD will generate a video frame sequence x = {x 0 , x 1 , ..., x N -1 } of length N , starting with c I . The sampling of SVD is conducted on a latent denoising diffusion process [39] . At each denoising step, a conditional 3D UNet Φ θ is used to iteratively denoise this sequence:EQUATIONwhere z t is the latent representation of x t . ẑ0 is the predication of z 0 . There are two condition paths for the reference image c I : (1) It is embedded into tokens by CLIP [37] image encoder and injected into the diffusion model vis cross-attention [39] ; (2) It is encoded into a latent representation by the VAE encoder, and concatenated with the latent of each frame in channel dimension. SVD follows the EDM-preconditioning framework [23] , which parameterizes the learnable denoiser Φ θ as:EQUATIONwhere σ is the noise level, and F θ is the network to be trained. c skip , c out , c in , and c noise are preconditioning hyper-parameters. Φ θ is trained via denoising score matching (DSM):E z0,t,n∼N (0,σ 2 ) λ σ ||Φ θ (z 0 + n, t, c I ) -z 0 || 2 2 .(3)Task formulation. The purpose of this paper is to locally edit a video, including visual information and motion information. In addition, the unedited content in the video should remain unchanged. Therefore, our conditional video generation involves three control signals:(1) the edited content, (2) the content of the unedited area, and (3) the motion condition in the edited area. We implement content editing by modifying the first frame of the video and then broadcasting it to subsequent video frames. Here, we denote the edited first frame as c ref ∈ R 3×W ×H . For the motion condition, we use interaction-friendly trajectory lines [57, 52] as the control signal. Specifically, the motion condition also contains N maps for a N-frame video. Each map consists of 2 channels, indicating the movement of the tracked points in the horizontal and vertical directions relative to the previous frame. The motion condition in this paper is represented as c mot ∈ R N ×2×W ×H . The unedited content c con can be conveniently provided by the masked video, i.e., c con = V • M, where V ∈ R N ×3×W ×H and M ∈ R 1×1×W ×H refer to the original video and the editing region mask, respectively.Since we adopt SVD as the pre-trained base model, its image-to-video capability can naturally serve as the import port for the edited first frame. For unedited content and customized motion trajectories, we train additional control modules to import them into the generation process.Trajectory sampling. During training, it is essential to extract trajectories from videos to provide motion condition c mot . At the beginning of trajectory sampling, we use a grid [57] to sparsify dense sampling points, obtaining N init initial points. Among these points, those with larger motions are beneficial to train trajectory control. To filter out these points, we first apply motion tracking on each point to obtain their path lengths, i.e., {l 0 , l 1 , ..., l Ninit-1 }. We use the mean of these lengths as the threshold l T h to extract points whose motion length is greater than l T h . Then, we use the normalized lengths of these points as sampling probabilities to sample N points randomly. Because the high sparsity is not conducive for the model to learn from these trajectories, we apply a Gaussian filter [57] to obtain the smooth trajectory map c mot . More details are presented in Appendix.Insights. A naive implementation of our editing target is directly training an extra control module, like ControlNet [58] , to inject motion and content conditions into the diffusion generation process.We present this design in structure A of Fig. 2 . Specifically, at the input, a content encoder E c and a motion encoder E m embed the content condition c con of the unedited area and motion condition c mot of the editing area. These two embeddings are merged by direct summing to obtain the fused condition feature f c . Then, a copy of the UNet encoder extracts multiscale intermediate features from f c , which are added to the corresponding layers in the diffusion model. This process is formulated as:EQUATIONwhere y c is the new diffusion features. Z is the function of zero-conv [58] . Θ and Θ c are the parameters of the SVD model and extra control module. We conduct several toy experiments based on this idea, as illustrated in Fig. 3 . The input video contains a woman initially moving to the left, followed by a shift to the right. The editing target is to alter the facial motion towards the right while keeping the other content unchanged. In the toy experiment 1, we fix SVD and train the control module with Eq. 3. The result shows that the content condition precisely controls the unedited area of the generated video. But the motion condition has no control effect, and the trajectory lines in the editing area (labeled with a black box) are consistent with the unedited area. A possible reason is that a single control branch has difficulty handling two control conditions simultaneously. To verify this hypothesis, we train structure B in Fig. 2 to handle these two conditions separately. The toy experiment 2 in Fig. 3 shows that the motion control is still ineffective, suggesting that the problem is more attributed to the control training rather than the network structure. To enhance the motion control training, we split the training of structure B into two stages. In the first stage, we only train the motion control module to endow it with motion control prior. In the second stage, we train the motion control and content control together. The result in toy experiment 3 shows that although the motion prior training produces good motion control capability, the control accuracy is weakened and affected by the unedited content after introducing the content control. After these toy experiments, we have the following insights:⋄ The condition of unedited content not only contains visual information but also has rich inter-frame motion information. As a more easily learned condition, the diffusion model tends to predict the motion of the editing area through unedited content, ignoring the sparse motion trajectory control. ⋄ The coupling between motion-customized new content and unedited content is strong, making it difficult to overcome even using the motion prior and separate control branches.⋄ Motion prior training is helpful in decoupling motion-customized content and unedited content.To rectify the ignoring of the motion control, we design a coarse-to-fine training strategy. In addition, structure B in Fig. 2 has a high computational cost, and we hope to joint control the unedited content and motion-customized new content on the concise structure A.As discussed above, motion trajectory is a sparse and difficult-to-learn control signal. Toy experiment 3 in Fig. 3 shows that the motion prior training can alleviate the coupling between motion-customized content and unedited content. Hence, in the first stage, we only train the motion trajectory control, allowing the control module to have good motion control prior.Decoupling training. Based on the control module from the first stage, the training in the second stage aims to add content control of unedited areas. Toy experiment 3 in Fig. 3 shows that even with good motion control priors, the precision of motion control still degrades after introducing unedited content condition. Therefore, we design a training strategy to decouple motion and content control in this stage. Specifically, we set the editing part and the unedited part in a training sample V to be two different videos, i.e., V 1 and V 2 . As shown in Fig. 4 , V 1 and V 2 are combined through the editing mask M, i.e.,V = V 1 • M + V 2 • (1 -M).Since the editing region and the unedited region come from two different videos, the motion information of the editing region cannot be predicted through the unedited content. Therefore, it can decouple content control and motion control during training.Deblocking training. As shown in the right part of Fig. 4 , although the decoupling training achieves joint control of customized motion and unedited content with high accuracy, it breaks the consistency between the edited and unedited regions, producing block artifacts in the boundary. To rectify this issue, we design the third training stage to remove block artifacts. The training in this stage is initialized with the model from the second stage and trained on normal video data. To preserve the decoupled motion and content control prior from the second stage, we only fine-tune the key embedding W k and value embedding W v in temporal self-attention layers of the control module and SVD model. The toy experiment 4 in Fig. 3 shows that after the training of this stage, the model removes the block artifacts and retains joint control of unedited content and motion customization.Although the coarse-to-fine training strategy achieves decoupling of content control and motion control, we observe considerable failure cases in some complex motion trajectories. To further distinguish the control roles of unedited content and motion trajectories in the generation, we design a spatiotemporal adaptive fusion module (SAFM) as shown in Fig. 5 . Specifically, SAFM predict a weight map Γ through the editing mask M to fuse motion and content control instead of direct summing. Moreover, because diffusion generation is a multi-step iterative process, the fusion of control conditions between time steps should have adaptive adjustment. Therefore, we concatenate timestep t and M in the channel dimension to form a spatiotemporal condition to guide the Γ prediction. Mathematically, the fusion of motion and content conditions is formulated as follows: where H is the function of spatiotemporal embedding. H needs to be jointly trained with W k and W v in the deblocking training stage. We visualize Γ at different time steps in the right part of Fig. 5 .EQUATIONIt can be seen that Γ learns the spatial characteristics of the editing area. It assigns a higher weight to the motion condition in the editing area and a higher weight to the content condition in the unedited area. In addition, Γ learns to distinguish different sampling steps t and linearly adjusts with t.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
implementation details: 
 In this work, we choose Stable Video Diffusion (SVD) as the base model. Our three training stages are completed on the WebVid [2] dataset, which contains 10 million text-video pairs. These three stages are optimized for 40K, 30K, and 20K iterations, respectively, with Adam [28] optimizer on 4 NVIDIA A100 GPUs. The batch size for each GPU is set as 4, with the resolution being 512 × 320. It takes about 6 days to complete all training stages. During the training process, we use CoTracker [22] to extract motion trajectories. In the first training stage, trajectory sampling is performed throughout the video. In the second and third training stages, a rectangular editing area is randomly selected in the video with the minimum size being 64 × 64, and trajectory sampling is performed within it. The number of trajectory lines for each training sample is randomly selected between 1 and 10.Among existing methods, Pika [1] is the most similar to ours. Pika can perform local video editing by defining an editing area. The difference is that Pika controls the new content in the editing area by text and has no motion control. In addition, the recent work AnyV2V [26] proposes editing the first frame of the video to achieve entire video editing, which has similarities with our ReVideo. InsV2V [10] , using editing instructions to edit the video, can also maintain unedited content. Therefore, in this paper, we compare our ReVideo with these three methods. The visual comparison in Fig. 6 shows that in some fine-grained editing scenarios, such as putting sunglasses on a man, AnyV2V has a loss of edited content. In addition, the unedited area of InsV2V and AnyV2V suffers from content distortion. Although Pika can generate smooth and high-fidelity results, it is difficult to accurately customize new content by text, especially in adding new objects, e.g., adding a dog on the soccer field. Adding new objects to the scene is also challenging for InsV2V. Due to the lack of motion control, AnyV2V and Pika usually produce static motion of the edited content, such as a car driving on the road. In comparison, our ReVideo can effectively broadcast the edited content throughout the entire video while allowing users to customize the motion in editing areas.In addition to visual comparison, we employ automatic metrics and human evaluation to measure the performance of different methods. For this task, we build a test set containing 16 videos, with the resolution being 720 × 1280. Following previous works [7, 26] , automatic metrics employ CLIP score [37] to measure text alignment and temporal consistency. The text alignment is obtained by calculating the average CLIP cosine similarity between each frame and editing description. Temporal consistency is computed by average CLIP cosine similarity between every pair of consecutive frames. We employ PSNR [43] to measure the reconstruction quality of unedited content. The human evaluation considers two aspects, i.e., overall video quality, and whether the editing target is achieved. We allow 20 volunteers to choose the best method for each test sample on each aspect. The results in Tab. 1 show that our ReVideo performs better than InsV2V and AnyV2V in all evaluation terms. Compared with Pika, our performance is slightly lower in the evaluation of temporal consistency and the quality of unedited content. Notably, AnyV2V and Pika usually generate static motion of new content due to the lack of motion control. Static motion tends to score higher in consistency evaluation, measured by CLIP similarity of adjacent video frames. Our method has obvious advantages over Pika in text alignment and human evaluation, reflecting the significant gap between text-guided local editing and user-specified local editing. Our ReVideo can precisely specify the appearance and motion of the editing area, better meeting requirements for accurate customization.The time complexity of different methods is also presented in Tab. 1. The experiment is conducted on an A100 GPU, with the video resolution being 768x768. Results show that our method has significantly lower time costs compared to other methods.In our ReVideo, we design the spatiotemporal adaptive fusion module (SAFM) to help decouple the control of unedited content and motion customization in diffusion generation. It predicts a fusion weight Γ conditioned on the editing area M and time step t. Then, the fusion of content and motion control is achieved through Eq. 5. In this part, we conduct an ablation study on this fusion mechanism. In addition, we only fine-tune the key embedding and value embedding of the temporal self-attention layers in the SVD model and control module in the stage of deblocking training. In the ablation study, we discuss the impact of tuning parameters in deblocking training.The effectiveness of SAFM. To demonstrate the effectiveness of SAFM, we replace SAFM with direct summing of motion and content control. The results in Fig. 7 show that the direct summing fusion cannot accurately control the motion in some complex motion trajectories, e.g., wavy lines. In comparison, using SAFM can help decouple content and motion control in the editing area, achieving more accurate trajectory guidance.The effectiveness of time adaptation in SAFM. We remove the time condition in the SAFM module, i.e., using the same weight map Γ to fuse content and motion control in each diffusion sampling step. The results in Fig. 7 show that not distinguishing Γ in different sampling steps leads to unsatisfactory artifacts at the boundary of the editing area.Tuning parameters in deblocking training. Although the training in stages 1 and 2 enables the control module to have good local motion control capabilities, we find that there is still an ignoring of motion control in the training of stage 3, i.e., deblocking training. As shown in Fig. 7 , the local motion control capability is degraded after we tune the entire control module in stage 3. Therefore, we optimize a part of the parameters to maintain the prior of local motion control. Experiments show that fine-tuning spatial layers still triggers the ignoring of motion control. In comparison, fine-tuning key embedding and value embedding of the temporal layer in the control module and the base model has minimal impact on local motion control capability. The edited and unedited areas are also harmoniously fused. More ablations of tuning parameters are presented in Appendix. 
conclusion: 
 In this paper, we aim to solve the problem of local video editing. The editing target includes visual content and motion trajectory modifications. To the best of our knowledge, this is the first attempt at this task. In this new task, We find a coupling problem between unedited content and motion customization. Directly training these two control conditions on the video generation model will cause the ignoring of motion control. To address this issue, we develop a three-stage training strategy to combine these two conditions coarse to fine. In addition, we design a spatiotemporal adaptive fusion module to further decouple unedited content and motion-customized content in different diffusion sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also easily extend these applications to multi-area editing without specific training.Limitations. Although our method can regenerate local areas of the video, the regeneration quality is limited by the base model. In some scenarios where the generation prior of SVD is not ideal, some unexpected artifacts may occur in the editing results. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification:The code for this paper is open-sourced at https://github.com/MC-E/ReVideo.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification:In the experiment section of the paper, we provide a detailed description of the training and testing details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: For the task of video editing, the error bar is uncommon, and there is currently no mature error evaluation system in this community.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification:In experiment details, we provide the time complexity and computational cost in training.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:The pre-trained model used in our paper is an image-to-video model, and the content of the generated video is specified by the input image. The video editing in this paper is achieved by editing the first frame, and the image editing method is not part of this paper. Therefore, there is no issue of data misuse in this paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : The capability of our method to locally modify video content and motion. This ability can also be easily extended to multi-area editing. The motion control is labeled in colorful lines in videos.Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and 
 
 Appendix: Sparsify Threshold Probabilistic samplingAs described in our main paper, trajectory sampling in the training process includes three stages, i.e., sparsifying, threshold filtration, and probabilistic sampling. We present the visualization of this pipeline in Fig. 8 . In sparsifying, we use a grid [57] to sparsify the dense sampling points, obtaining N init initial points. In threshold filtration, we use the mean of the tracking length of these N init points as the threshold l T h to filter out points with large motion. In probabilistic sampling, we use the normalized lengths of these sampling points as sampling probabilities to sample N points from them. N is randomly selected from 1 to 10.Edited Frame 1 Edited Frame 7 Edited Frame 14 In our main paper, we demonstrate the robustness of our method on multi-area editing without specific training. In Fig. 9 , we present another robustness of our method for irregular editing areas. As can be seen, even though our method is trained on rectangular editing areas, it has stable content and motion editing capabilities when facing a hand-drawn irregular editing area.In some video editing scenarios, specifying motion trajectories is challenging, such as when erasing an object in a video. We find that by leaving the motion control empty in these cases, our method can automatically generate the motion state of the editing area and match it with the motion in the unedited area. This is due to our inherent capability to predict the motion in the editing area via unedited content. An example is presented in Fig. 10 . In Fig. 11 , we try some complex scenarios. The result shows that our method can handle dynamic lighting and texture, but scene change affects the content quality, which is a failure case.At the input of our spatiotemporal adaptive fusion module (SAFM), two encoders, i.e., E c and E m , separately encode the content and motion conditions. E c and E m have the same low-cost structure. This structure contains three sub-blocks, each consisting of a convolution and a downsampling operation, mapping the condition map to the same size as the latent z t .In Fig. 12 , we present more editing results of our ReVideo, including adding new objects to the video, modifying the motion trajectory of existing content in the video, editing existing content while maintaining the motion trajectory, and multi-region editing. As can be seen, the editing results achieve the editing goals, and motion control and content control coexist harmoniously. consistency to the highest level, but all 5 editing results failed. This indicates the inaccuracy of text as the control signal of local redrawing. In comparison, editing the first frame and then broadcasting it to the entire video can accurately specify the content of the editing area.NeurIPS Paper ChecklistJustification:We have carefully reviewed the NeurIPS Code of Ethics to ensure that our submission complies with all regulations.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: Overall, the reviewers agree that the paper tackles an unexplored and challenging task, significantly advancing the capability to modify content and motion trajectories in specific regions using diffusion models. The experiments demonstrate promising results and show improvements over prior methods. The reviewers have offered suggestions for revision, including a more precise description of the task scope and editing capabilities, a discussion of potential artifacts introduced by the proposed method, and detailed documentation of the training process. Additionally, to facilitate reproducibility and further research on this new task, it is strongly recommended that the authors release the code for training and inference, as they have committed to doing.

Reviews: review: paper_summary: paper_summary: This paper presents ReVideo, a new approach for precise local video editing of both content and motion. It introduces a coarse-to-fine training strategy to progressively decouple content and motion control, and a spatiotemporal adaptive fusion module to integrate them effectively. Experiments show ReVideo can modify local video content, customize motion trajectories, or change both simultaneously, and extend to multi-region editing.
strength_weakness: strength_weakness: ## Practicality of the Editing Workflow

The current editing interface requires users to specify both a target content image and a set of motion trajectories. While this allows for fine-grained control, it may not be the most intuitive or efficient workflow for common editing tasks. Consider the scenario of object removal - the user would need to carefully craft a content image with the object removed and ensure that the remaining motion trajectories are consistent. An alternative approach could be to directly specify the regions to remove and have the model infer the appropriate content and motion changes automatically. The paper would benefit from a more detailed discussion of the practical trade-offs and usability considerations of the proposed editing framework.

## Limited Motion Control
While the method allows for editing the motion of individual objects, it assumes that the overall scene motion (camera movement, background motion) remains fixed. This limits the applicability of the approach in scenarios where the goal is to modify the global motion patterns (e.g. stabilizing shaky footage, changing the camera viewpoint).

## Precise Placement and Key Contributions of this Paper

While the individual technical components (e.g. coarse-to-fine training, adaptive fusion) are well-motivated, it's worth considering whether similar strategies have been explored in related domains. For instance, progressive training to handle multi-factor variation has been used in GANs, and spatially-adaptive normalization is common in style transfer. Drawing more connections to such related work would clarify the novelty of the specific adaptations made here.

## Content-Motion Entanglement

- The key technical contribution of the paper is the decoupling of content and motion information through a coarse-to-fine training strategy. However, it's not clear if this decoupling is complete or if there are still some residual entanglements between the two factors. For instance, the edited content may still contain some motion information that could interfere with the specified motion trajectories, leading to artifacts or inconsistencies. A more thorough analysis of the content-motion separation and its impact on the editing quality would be informative.

- Is decoupling content and motion the only way to address the issue - could a joint representation learning approach work instead? Acknowledging alternate strategies would help justify the chosen approach.

- **Figure 4 is not very intuitive. It would benefit from additional justification, theoretical analysis, and insights into why such a simple composition from two videos is effective.** This is a key concern.

## Multi-area Editing 
- The extension to multi-area editing is a nice addition, but the paper could go further in characterizing the challenges involved. Are there issues with preserving global coherence across multiple edited regions? How does the method scale with the number of regions? Providing such details would give a more complete picture of the capability.

## Clarity and Reproducibility
- Implementation details: There are some missing specifics that could hamper reproducibility. For instance:

   - How exactly are the editing regions defined during training - what is the procedure for randomly sampling them?
   - What metrics are used for the "threshold filtering" of motion trajectories and how were the thresholds chosen?
   - Are there any data augmentation, regularization or optimization tricks used during training?

## Evaluation Metrics

The quantitative evaluation relies primarily on low-level metrics like PSNR and LPIPS, which may not fully capture the perceptual quality and coherence of the edited videos. Additional metrics could provide a more comprehensive assessment:

- Metrics that specifically measure the consistency of the edited regions with the target content and motion (e.g. using an object detector or tracker).
- Metrics that evaluate the temporal stability and smoothness of the edited videos (e.g. some metrics that are used in video inpainting tasks, Please refer to [this repo](https://github.com/MichiganCOG/video-inpainting-evaluation) for details).
- Human evaluations of the overall realism, coherence, and faithfulness to the editing inputs (e.g. through user studies).



## Robustness Evaluation and Ablation Studies

While the paper does include ablations for a few key components (e.g. SAFM, training stages), there are other design choices that are not fully explored. For instance:

   - How important is the choice of motion representation (trajectory vs. alternatives)? Testing with different motion inputs would reveal the sensitivity to this factor.
   - What is the impact of the trajectory sampling strategy and hyperparameters? Varying the number and selection of trajectories could provide insight into the robustness.
   - How does the performance vary with the size and shape of the editing regions? A systematic evaluation across different region properties would be informative.
   - Only the end-to-end video editing pipelines are compared, but not the individual technical components. For instance, how does SAFM compare to simpler fusion schemes used in prior work?
  - Input noise and perturbations (e.g. in the content image or motion trajectories)

## Dataset Complexity 

-  While the approach achieves good results on the chosen datasets, it's unclear how well it would generalize to more complex video content (e.g. with dynamic backgrounds, scene changes, occlusions etc.). Discussing the potential failure modes and current limitations would help scope the contribution appropriately.

- The examples shown in the paper are largely limited to simple object-level edits in relatively constrained scenarios (e.g. clean backgrounds, single objects). It's unclear how well the method would perform on more challenging videos with complex scenes, multiple objects, occlusions, camera motion, etc. Testing on a wider range of video complexity would help establish the generality of the approach.

## Editing Scenarios
The paper demonstrates a few key editing applications (e.g. object addition/removal, motion editing), but there are other important scenarios that are not explored, such as: performing semantic-level edits (e.g. changing the action or interaction between objects).
Showcasing the method's performance across a fuller range of editing tasks would demonstrate its versatility.

## Open Source
Will the code for training and inference be released?
questions: questions: Please refer to the weakness section.
limitations: limitations: Please refer to the weakness section.
paper_summary: paper_summary: The paper presents a video editing method that enables precise localized adjustments to content and motion within specific areas of a video. It introduces a three-stage training strategy and a spatiotemporal adaptive fusion module to integrate edits across frames and locations effectively. This method allows for complex editing tasks such as changing content while maintaining motion, adding new motion to static content, and simultaneously modifying both elements.
strength_weakness: strength_weakness: - The decoupling training could cause some artifacts. Although the paper demonstrates these artifacts could mostly be alleviated by deblocking training. I can still see some blocky/unnatural results in the result videos.
- The training is quite complicated and separated into three stages. I feel the training strategy could 'overfit' this particular video dataset.
- This method is more like a direct combination of video diffusion and ControlNet.
- More detailed implementation specifics, particularly regarding parameter settings and the architecture of the spatiotemporal adaptive fusion module, are needed.
- The method's computational demands and potential scalability issues are not adequately addressed. For example, what kind of GPU does one need to perform training and testing? 
- The paper focuses heavily on technical aspects with less consideration of user interaction.
questions: questions: - What kind of GPU does one need to perform training and testing? 
- Will the authors release the training and testing code along with pre-trained models upon acceptance?
limitations: limitations: Quality limited by SVD.
paper_summary: paper_summary: ReVideo presents a novel view of video editing by modifying content with input trajectory to create new content. It designs a three-stage strategy to wrestle out the problem of ignoring motion control when direct training. The main contribution of this work relies on the new task of editing motion via user-specified trajectory while keeping the original video movement. The editing results are superior and photorealistic.
strength_weakness: strength_weakness: 1. The author did not provide the method or explanation of how ReVideo edits the first frame, making the total editing pipeline not end-to-end for users.
2. Part of the original video motion, like mouth movement in the Zuckerberg->robot (head6) and tail movement in dog->lion, is not kept in the edited video.
3. I would like to know how the drag-based editing method handles non-rigid motion, such as the rotation of car tires from a side view. In examples like sea2 and sea2_2, where a shark and a dinosaur are added, the limbs of the animal seem unable to move, making the video look unrealistic. However, in soccer and some human-centric examples, the legs of dogs and people can move normally. Therefore, I would like the authors to add an example of a vehicle moving on the road from a side view, including the movement of the wheels, to address my concerns. This may be a limitation of the drag-based method.
3. There is no quantitative comparison of the ablation study; I understand that the image results in Fig 7 are clear, but only one video qualitative ablation is not reasonable. 
4. There are no qualitative video comparisons with other methods in the supp or project page, but only Fig 6, and the automatic metrics are worse than pika even though I understand the clip scores are not accurate, which can not reflect temporal consistency accurately. I suggest the author supply the comparison video between Revideo and other methods in the rebuttal phase.
5. The training cost of three stages: even though Revideo makes great progress in creating new motion, training cost like GPU costs, time costs, memory costs and so on, is still a problem since users prefer to edit a video in a zero-shot manner when using a pretrained video generation model and the compared methods like AnyV2V is training-free.
questions: questions: 1. The method to edit the first frame needs to be declared.
2. Non-rigid motion-like side view wheels movement of cars.
3. Qualitative video comparisons with other methods. 
4. The inference time/training cost comparison with other similar methods.
5. What about ReVideo performing in editing multiple objects simultaneously in the same video?
6. Can ReVideo work on text-to-video generation models?
limitations: limitations: no significant limitations.

Comments: comment: Thank you for your efforts in the review. Your comments help improve our method, and we will continue to enhance our approach in future work.
Dear authors,

Thank you very much for the clarification. However, I still feel the paper is somewhat incremental by combining a ControlNet-like motion injection module with augmented (motion) training data. The approach and task themselves are new and novel. Furthermore, the result videos are impressive, with barely noticeable artifacts. I would like to raise my rating to borderline accept.
Thanks again to the authors for the clarification and discussions!
Thank you for your efforts in the review, improving our paper to a higher standard. We will revise our paper based on your comments and suggestions.
My concerns have been thoroughly addressed in the rebuttal. Overall, this paper is well-structured, offering significant contributions and demonstrating impressive performance. It brings fresh perspectives to the research field, so I have decided to upgrade my final rating to accept.
This understanding is correct. In our experiments, we find a significant motion coupling between the foreground (editing region) and the background (please refer to Section 3.2). The model tends to learn how to estimate the motion state of the foreground (editing region) by relying on the motion in the background, thereby neglecting the condition of motion trajectory. To address this issue, we propose the motion decoupling training as you mentioned. We may not describe this clearly enough in the paper, and we will make a revision for this part.
I found out I severely misunderstood the operation of the motion decoupling part. The goal here is to manipulate different motions between the foreground (editing region) and background. Then, the purpose is largely different from MixCut or other cut-and-paste approaches. Could the authors further claim the purpose of the motion decoupling? Is my current understanding correct?
Dear reviewers, I would like to further explain our proposed three-stage training strategy. Its contributions mainly lie in two aspects:

1. We discover a significant coupling between content and motion in this task, which is an attribute difficult to detect. As described in Section 3.2, we find this attribute/issue after many failed attempts. This priori exploration can inspire future research in this community.

2. The proposed method successfully tackles an unexplored and challenging task. Each stage is validated and analyzed in Section 3.2. The design of each stage is effective and innovative given the current state of this research field.
Thank you for your constructive suggestions. Your very detailed comments in the review improve our paper to a higher standard. We will make a revision accordingly.
Thank you for your rebuttal and efforts to address the concerns raised. I still believe this work is timely and addresses an important topic.

After considering the comments from other reviewers and your rebuttal, my opinion and suggestions are as follows:

1. The task scope and editing capabilities need to be defined more precisely. The initial submission did not clearly discuss the limitations and capabilities in the introduction and conclusion sections.

2. While the decoupling method is well-motivated, the proposed solution is straightforward and similar to proposals seen in many common computer vision tasks. There is little technical analysis and theoretical explanation provided, especially regarding potential artifacts introduced by the proposed method.

3. Given the current state of the field, the architecture is reasonable and enables motion control by trajectory, despite its complexity and the need for tuning many hyperparameters. However, a more concise solution may emerge in the future. To facilitate reproducibility, I recommend providing more detailed documentation of the training details in the paper.
1. ControlNet cannot accomplish our task. This is the fundamental difference. The following works all used ControlNet for conditional injection. We do not claim ControlNet as one of our contributions. It is a commonly used model, just like Stable Diffusion.

[1] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models

[2] ControlVideo: Training-free Controllable Text-to-Video Generation

[3] CameraCtrl: Enabling Camera Control for Text-to-Video Generation

[4] EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing

[5] CCEdit: Creative and Controllable Video Editing via Diffusion Models

[6] Text-Animator: Controllable Visual Text Video Generation

[7] IMAGE CONDUCTOR: PRECISION CONTROL FOR INTERACTIVE VIDEO SYNTHESIS

2. [1] is a paper on watermark removal, where a watermark image refers to the watermark being overlaid on the original image. This overlay is a task, not a solution. We still cannot find any connection between this and our method.

[1] Liu, Yang, Zhen Zhu, and Xiang Bai. "Wdnet: Watermark-decomposition network for visible watermark removal." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.

3. Our method is the first attempt at this task and has already achieved state-of-the-art (SOTA) results. Even if adjustments are needed for other models and datasets, the issues found and solutions proposed in this work are still insightful. 

4. CutMix you referenced is entirely different from our approach in both purpose and methodology. We do not think combining two images to enhance image classification performance can replace our contribution. What we propose is a motion-decoupled training strategy, not data augmentation. This decoupling strategy is effective and does not exist in our community.
1. Novelty: Modifying existing methods doesn't automatically constitute novelty. Your paper shows good designs and results but appears incremental to ControlNet. Clarify what fundamentally differentiates your approach from ControlNet beyond architectural changes.
2. Methodology: The cut-and-paste approach you're using is common in various video processing tasks, e.g., [1], not unique to your method. Referencing MixCut was to highlight this point, not to directly compare classification and video editing tasks.
3. Generalizability: You claim your method can generalize beyond SVD and WebVid, but haven't demonstrated this. Without broader validation, the method's applicability to other datasets or base models remains unclear.
4. Contributions: While some components in your work are new, they appear to be engineering efforts rather than fundamental contributions. The reference to CutMix was to point out similar data augmentation techniques in video processing, not to equate it directly with your method.

[1] Liu, Yang, Zhen Zhu, and Xiang Bai. "Wdnet: Watermark-decomposition network for visible watermark removal." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.
We need to clarify that:

1. **ControlNet is a commonly used method for condition injection, not a solution for our problem (as verified in Section 3.2). The contribution of this paper is to propose effective solutions for local video editing.** Additionally, we modify ControlNet by designing a region-adaptive fusion method to make the condition embedding more suitable for this task.

2. **Why is the decoupling training strategy not considered one of our contributions?** Dear reviewer, have you ever seen other similar solutions for video editing? The reference you provided is an image classification method—what relevance does it have to our paper? Just because two different images are mixed together?

3. **Our method is successfully validated on SVD and WebVid data, but this does not mean that our method can only be implemented using SVD and WebVid.** It can be applied to other data and base models. We believe that addressing a challenging problem by a progressive learning approach is reasonable. In many complex generation tasks, such as EMU2[1], the training difficulty is much greater than ours.

[1] Generative Multimodal Models are In-Context Learners

4. Firstly, we are addressing an unexplored and challenging task that requires carefully designed training processes and fusion modules. This is not a simple engineering patchwork. Our core components did not previously exist. Secondly, what is the relevance of the CutMix to our work? Why can it replace our core contribution?

Dear R2, we are eager to address these misunderstandings and look forward to your response.
I really appreciate the author's rebuttal. However:

1. The authors claim that the proposed method's novelty is beyond ControlNet. However, I am still not convinced. From Figure 2 in the paper, the two potential structures are just two slightly different ways of conditioning ControlNet. The trainable control module and zero convolution layers are exactly the same as ControlNet. The only difference is that you replace the condition (originally depth, canny edge map, normal, etc.) with a content map specifying the editing region and the motion trajectory. I still cannot say this analysis is a contribution. I'd say this is rather than an ablation on network architectural design.

2. In order to keep the original motion from the original video. This paper further proposes a data construction strategy, as shown in Figure 4, to decouple the motion between the editing region and the original region. However, this approach is still too naive, as it just performs CutMix [1] to combine two videos into one. However, in this case, the goal is to manipulate motion. I am still not convinced this stage is a contribution.

3. The training pipeline, separated into three stages, is very complex. There will be many hyperparameters that could influence the training in each stage. Also, the entire pipeline relies on many existing off-the-shelf techniques/datasets such as SVD and WebVid (This dataset is actually banned and no longer available due to some legal issue. Thus, the proposed method will not be reproducible. However, this is not the fault of this method.), and CoTracker. If more advanced techniques show up, one needs to tweak all the parameters of the entire training process to make the proposed method work. This makes this method not general 

4. Following the previous comment, the proposed pipeline is more like an engineering work by combining existing techniques (SVD, ControlNet, CoTracker, CutMix) and using a massive computational power (4x A100), directly training for a long time (6 days) on a largescale dataset (WebVid). I really appreciate the edited video results. Some of them are amazing. In some of them, I still see artifacts (which might be the cons from SVD). In the entire paper, I do not see many novelties other than architectural designs and training data augmentation.

Due to the above reasons, I am still leaning towards rejection. However, I would like to hear and discuss more with the authors and other reviewers.

Best,

Reviewer ZSh9

[1] Yun, Sangdoo, et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features." Proceedings of the IEEE/CVF international conference on computer vision. 2019.
Dear R2, we would like to know if your concerns are addressed. If any questions, we can discuss them in this phase.