Title: ReVideo: Remake a Video with Motion and Content Control
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion model

Video editing
Abstract: Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-area editing without specific training, demonstrating its flexibility and robustness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReVideo presents a novel view of video editing by modifying content with input trajectory to create new content. It designs a three-stage strategy to wrestle out the problem of ignoring motion control when direct training. The main contribution of this work relies on the new task of editing motion via user-specified trajectory while keeping the original video movement. The editing results are superior and photorealistic.\n\nSTRENGTHS:\n1. The first video editing work on creating new motion and content.\n2. Good writing; the paper is easy to follow, and the motivation and three-stage training strategy on decoupling content and motion control is reasonable. The proposed SAFM learned a dynamic fusion weight at different timesteps.\n3. The editing results are photorealistic and adhere to the original motion or follow user-specified trajectory with no artifacts.\n\nWEAKNESSES:\n1. The author did not provide the method or explanation of how ReVideo edits the first frame, making the total editing pipeline not end-to-end for users.\n2. Part of the original video motion, like mouth movement in the Zuckerberg->robot (head6) and tail movement in dog->lion, is not kept in the edited video.\n3. I would like to know how the drag-based editing method handles non-rigid motion, such as the rotation of car tires from a side view. In examples like sea2 and sea2_2, where a shark and a dinosaur are added, the limbs of the animal seem unable to move, making the video look unrealistic. However, in soccer and some human-centric examples, the legs of dogs and people can move normally. Therefore, I would like the authors to add an example of a vehicle moving on the road from a side view, including the movement of the wheels, to address my concerns. This may be a limitation of the drag-based method.\n3. There is no quantitative comparison of the ablation study; I understand that the image results in Fig 7 are clear, but only one video qualitative ablation is not reasonable. \n4. There are no qualitative video comparisons with other methods in the supp or project page, but only Fig 6, and the automatic metrics are worse than pika even though I understand the clip scores are not accurate, which can not reflect temporal consistency accurately. I suggest the author supply the comparison video between Revideo and other methods in the rebuttal phase.\n5. The training cost of three stages: even though Revideo makes great progress in creating new motion, training cost like GPU costs, time costs, memory costs and so on, is still a problem since users prefer to edit a video in a zero-shot manner when using a pretrained video generation model and the compared methods like AnyV2V is training-free.\n\nQUESTIONS:\n1. The method to edit the first frame needs to be declared.\n2. Non-rigid motion-like side view wheels movement of cars.\n3. Qualitative video comparisons with other methods. \n4. The inference time/training cost comparison with other similar methods.\n5. What about ReVideo performing in editing multiple objects simultaneously in the same video?\n6. Can ReVideo work on text-to-video generation models?\n\nLIMITATIONS:\nno significant limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a video editing method that enables precise localized adjustments to content and motion within specific areas of a video. It introduces a three-stage training strategy and a spatiotemporal adaptive fusion module to integrate edits across frames and locations effectively. This method allows for complex editing tasks such as changing content while maintaining motion, adding new motion to static content, and simultaneously modifying both elements.\n\nSTRENGTHS:\n- The paper introduces a novel challenge of editing both content and motion in specific video areas and combines techniques from diffusion models and video editing to achieve nuanced control.\n- The three-stage training strategy enhances the robustness and effectiveness of the edits, supported by experimental validation that demonstrates superior performance compared to existing methods.\n- The paper is well-organized and clearly explains complex concepts, including the innovative spatiotemporal adaptive fusion module and detailed training strategy.\n\nWEAKNESSES:\n- The decoupling training could cause some artifacts. Although the paper demonstrates these artifacts could mostly be alleviated by deblocking training. I can still see some blocky/unnatural results in the result videos.\n- The training is quite complicated and separated into three stages. I feel the training strategy could 'overfit' this particular video dataset.\n- This method is more like a direct combination of video diffusion and ControlNet.\n- More detailed implementation specifics, particularly regarding parameter settings and the architecture of the spatiotemporal adaptive fusion module, are needed.\n- The method's computational demands and potential scalability issues are not adequately addressed. For example, what kind of GPU does one need to perform training and testing? \n- The paper focuses heavily on technical aspects with less consideration of user interaction.\n\nQUESTIONS:\n- What kind of GPU does one need to perform training and testing? \n- Will the authors release the training and testing code along with pre-trained models upon acceptance?\n\nLIMITATIONS:\nQuality limited by SVD.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ReVideo, a new approach for precise local video editing of both content and motion. It introduces a coarse-to-fine training strategy to progressively decouple content and motion control, and a spatiotemporal adaptive fusion module to integrate them effectively. Experiments show ReVideo can modify local video content, customize motion trajectories, or change both simultaneously, and extend to multi-region editing.\n\nSTRENGTHS:\n- This appears to be the first attempt at exploring local editing of both content and motion in videos using diffusion models. Being able to modify content and motion trajectories in specific regions is a novel capability compared to prior work.\n- The proposed three-stage coarse-to-fine training strategy to progressively decouple content and motion control is an interesting technical approach to deal with the core challenge.\n- The spatiotemporal adaptive fusion module is another novel component to integrate the content and motion conditions across sampling - steps and spatial locations.\n- Extending the approach to allow multi-area editing without requiring specific training demonstrates flexibility.\n- Most of the visual and quantitative results show improvements over prior methods \n\nOverall, this paper addresses a timely and important topic with significant potential benefits for the community. Despite some weaknesses, the reviewer recommends acceptance, considering this is a relatively new area and the paper presents promising results. The score may be adjusted based on the quality of the rebuttal.\n\nWEAKNESSES:\n## Practicality of the Editing Workflow\n\nThe current editing interface requires users to specify both a target content image and a set of motion trajectories. While this allows for fine-grained control, it may not be the most intuitive or efficient workflow for common editing tasks. Consider the scenario of object removal - the user would need to carefully craft a content image with the object removed and ensure that the remaining motion trajectories are consistent. An alternative approach could be to directly specify the regions to remove and have the model infer the appropriate content and motion changes automatically. The paper would benefit from a more detailed discussion of the practical trade-offs and usability considerations of the proposed editing framework.\n\n## Limited Motion Control\nWhile the method allows for editing the motion of individual objects, it assumes that the overall scene motion (camera movement, background motion) remains fixed. This limits the applicability of the approach in scenarios where the goal is to modify the global motion patterns (e.g. stabilizing shaky footage, changing the camera viewpoint).\n\n## Precise Placement and Key Contributions of this Paper\n\nWhile the individual technical components (e.g. coarse-to-fine training, adaptive fusion) are well-motivated, it's worth considering whether similar strategies have been explored in related domains. For instance, progressive training to handle multi-factor variation has been used in GANs, and spatially-adaptive normalization is common in style transfer. Drawing more connections to such related work would clarify the novelty of the specific adaptations made here.\n\n## Content-Motion Entanglement\n\n- The key technical contribution of the paper is the decoupling of content and motion information through a coarse-to-fine training strategy. However, it's not clear if this decoupling is complete or if there are still some residual entanglements between the two factors. For instance, the edited content may still contain some motion information that could interfere with the specified motion trajectories, leading to artifacts or inconsistencies. A more thorough analysis of the content-motion separation and its impact on the editing quality would be informative.\n\n- Is decoupling content and motion the only way to address the issue - could a joint representation learning approach work instead? Acknowledging alternate strategies would help justify the chosen approach.\n\n- **Figure 4 is not very intuitive. It would benefit from additional justification, theoretical analysis, and insights into why such a simple composition from two videos is effective.** This is a key concern.\n\n## Multi-area Editing \n- The extension to multi-area editing is a nice addition, but the paper could go further in characterizing the challenges involved. Are there issues with preserving global coherence across multiple edited regions? How does the method scale with the number of regions? Providing such details would give a more complete picture of the capability.\n\n## Clarity and Reproducibility\n- Implementation details: There are some missing specifics that could hamper reproducibility. For instance:\n\n   - How exactly are the editing regions defined during training - what is the procedure for randomly sampling them?\n   - What metrics are used for the \"threshold filtering\" of motion trajectories and how were the thresholds chosen?\n   - Are there any data augmentation, regularization or optimization tricks used during training?\n\n## Evaluation Metrics\n\nThe quantitative evaluation relies primarily on low-level metrics like PSNR and LPIPS, which may not fully capture the perceptual quality and coherence of the edited videos. Additional metrics could provide a more comprehensive assessment:\n\n- Metrics that specifically measure the consistency of the edited regions with the target content and motion (e.g. using an object detector or tracker).\n- Metrics that evaluate the temporal stability and smoothness of the edited videos (e.g. some metrics that are used in video inpainting tasks, Please refer to [this repo](https://github.com/MichiganCOG/video-inpainting-evaluation) for details).\n- Human evaluations of the overall realism, coherence, and faithfulness to the editing inputs (e.g. through user studies).\n\n\n\n## Robustness Evaluation and Ablation Studies\n\nWhile the paper does include ablations for a few key components (e.g. SAFM, training stages), there are other design choices that are not fully explored. For instance:\n\n   - How important is the choice of motion representation (trajectory vs. alternatives)? Testing with different motion inputs would reveal the sensitivity to this factor.\n   - What is the impact of the trajectory sampling strategy and hyperparameters? Varying the number and selection of trajectories could provide insight into the robustness.\n   - How does the performance vary with the size and shape of the editing regions? A systematic evaluation across different region properties would be informative.\n   - Only the end-to-end video editing pipelines are compared, but not the individual technical components. For instance, how does SAFM compare to simpler fusion schemes used in prior work?\n  - Input noise and perturbations (e.g. in the content image or motion trajectories)\n\n## Dataset Complexity \n\n-  While the approach achieves good results on the chosen datasets, it's unclear how well it would generalize to more complex video content (e.g. with dynamic backgrounds, scene changes, occlusions etc.). Discussing the potential failure modes and current limitations would help scope the contribution appropriately.\n\n- The examples shown in the paper are largely limited to simple object-level edits in relatively constrained scenarios (e.g. clean backgrounds, single objects). It's unclear how well the method would perform on more challenging videos with complex scenes, multiple objects, occlusions, camera motion, etc. Testing on a wider range of video complexity would help establish the generality of the approach.\n\n## Editing Scenarios\nThe paper demonstrates a few key editing applications (e.g. object addition/removal, motion editing), but there are other important scenarios that are not explored, such as: performing semantic-level edits (e.g. changing the action or interaction between objects).\nShowcasing the method's performance across a fuller range of editing tasks would demonstrate its versatility.\n\n## Open Source\nWill the code for training and inference be released?\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}