Title: ROSE: Remove Objects with Side Effects in Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: video inpainting; diffusion transformer; video generation
Abstract: Video object removal has achieved advanced performance due to the recent success of video generative models. However, when addressing the side effects of objects, \textit{e.g.,} their shadows and reflections, existing works struggle to eliminate these effects for the scarcity of paired video data as supervision. This paper presents \method, termed \textbf{R}emove \textbf{O}bjects with \textbf{S}ide \textbf{E}ffects, a framework that systematically studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror. Given the challenges of curating paired videos exhibiting the aforementioned effects, we leverage a 3D rendering engine for synthetic data generation. We carefully construct a fully-automatic pipeline for data preparation, which simulates a large-scale paired dataset with diverse scenes, objects, shooting angles, and camera trajectories. ROSE is implemented as an video inpainting model built on diffusion transformer. To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing. Moreover, additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos. To fully investigate the model performance on various side effect removal, we presents a new benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five special side effects for comprehensive evaluation. Experimental results demonstrate that \method achieves superior performance compared to existing video object erasing models and generalizes well to real-world video scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRemoving objects in videos is challenging because not only the objects themselves but also their associated effects must be removed.\nThe authors address this issue by proposing the ROSE framework.\nSince real-world data containing both objects and their side effects is scarce, they construct a novel synthetic dataset using a rendering tool.\nThis dataset enables effective training of the ROSE framework.\nTo handle side effects, the method incorporates mask region guidance into a diffusion-based backbone.\nAdditionally, to account for potential human errors in object masking, they introduce mask augmentation techniques during training.\nWith these contributions, the proposed framework achieves effective and robust video object removal.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Novel dataset, which contains various inpainting scenarios with object mask and multi-view scenes.\n2. Effectively removing not only the target object but also its side effects reflects the natural human expectation of comprehensive object removal in video content.\n\nWeaknesses\n1. While the proposed method shows promising performance, an analysis of computational efficiency (e.g., number of parameters, FLOPs) is missing. A comparison of resource usage with baseline methods is necessary to demonstrate that the framework achieves its results without excessive computational overhead.\n2. Ablation analysis is insufficient. It is important to provide a detailed breakdown of how each module contributes to the overall performance improvement, based on the ablation results.\n\nQUESTIONS:\n1.The paper requires a comparative analysis of efficiency with baseline methods such as ProPainter and DiffuEraser. In particular, metrics such as FLOPS and per-frame runtime during inference should be reported. Additionally, as mentioned in the limitations, it would be valuable to show the difference in per-frame inference time between long and short clips.\n\n2. The motivation in lines 156–159 clearly justifies the necessity of MRG, and its effectiveness is well demonstrated through the \"+MRG\" results in Tab. 1. Given this, it would be interesting to see whether similarly dramatic improvements can be observed when mask region guidance is applied to other methods as well. Do baseline models like ProPainter or DiffuEraser exhibit the same performance gains when MRG is incorporated?\n\n3. Mask augmentation, despite being presented as a key component of the proposed training strategy, shows only marginal performance improvements in Tab. 1, even leads to decreased performance. What are the results when only MRG and DMP are applied without MA? As discussed in Section 4.3, the authors should provide concrete examples and analysis of specific scenarios where this augmentation brings actual benefits, along with visual or quantitative evidence to support its effectiveness.\n\n4. I’m curious whether masking only part of an object results in side effects being removed only for that region, or if the effect of mask augmentation is strong enough to eliminate all related side effects. For example, in a scene where a person and a horse are jumping over a fence, what would the result look like if only the person is masked out?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to create a video removal dataset that focuses on removing the objects and the side effects of the objects. Firstly, the authors investigate five types of side effects. Then, a 3D rendering engine is used to generate the samples and ensure the quality of side effects in the videos. In the training stage, an auxiliary branch is used to achieve additional supervision, which could help the model learn to recognize the region of interest. Also, the ROSE-Bench is used to evaluate the performance of video object erasing models.\n\nSTRENGTHS AND WEAKNESSES:\nS:\n1. The construction of a high-quality video removal dataset is necessary for the community. And the authors conclude six side-effect categories, which could guide following works.\n2. The 3D data successfully solves the problems of side effects in the previous datasets through the proposed construction pipeline.\nW:\n1. Fig. 1 is not clear when I zoom in, please use PDF images.\n2. The synthetic dataset only includes videos with camera motion and without object motion, which limits the capability of processing videos with moving objects. This point is only mentioned in Sec. 6, no visual results are shown in the paper.\n3. The side effects are divided into six types, but only three types are shown in Fig. 7.\n4. The lack of experiments on investigating the effectiveness of the mask-based filtering.\n5. Sec. 4.3 claims that the user-provided masks are often imperfect and thus proposes mask augmentation to mitigate this issue. But there is no such case in the experiment to show how ROSE could deal with the imperfect masks.\n6. The visual comparisons are superior in Fig. 7, but the PSNR value in Tab. 2 is lower than ProPainter.\n\nQUESTIONS:\n1. Does ROSE-Bench include similar imperfect masks as well to better support the function of mask augmentation?\n2. Does the low PSNR in Tab. 2 mean that ROSE cannot generalize well to the real data due to the synthetic training dataset?\n\nLIMITATIONS:\nThe suggestions are provided above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a video object removal framework that also removes associated side effects like shadows, reflections, and lighting. To address the lack of real paired data, the authors use a 3D rendering engine to generate synthetic video pairs. ROSE is a diffusion-based inpainting model that uses full video context and introduces a difference mask predictor for explicit supervision. A new benchmark, ROSE-Bench, is created to evaluate various side-effect scenarios. Experiments show that ROSE outperforms prior methods in both accuracy and realism.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.The paper addresses a practical  problem in video inpainting—removing not just objects but also their environmental side effects such as shadows and reflections. Its approach is well-motivated, with a clear design that combines a diffusion-based inpainting model, full video context input, and a difference mask predictor for explicit supervision\n\n2.The use of a 3D rendering pipeline to generate paired synthetic videos is effective and well-documented. Additionally, the introduction of ROSE-Bench provides a useful benchmark for evaluating side-effect removal. \n\n3. This paper is well-written and easy-to-follow.\n\n**Weakness**\n\n1. While the paper presents strong quantitative and qualitative comparisons, it lacks video-based results, which are crucial for evaluating temporal consistency and perceptual quality in video inpainting. The current comparisons rely mostly on metrics and static frame visualizations, which may not fully reflect performance on dynamic content. Including sample videos would strengthen the evaluation.\n\n2. Additionally, the paper does not report inference time or model size, making it difficult to assess the method’s efficiency and scalability compared to prior work.\n\n3. To better validate the effectiveness of the constructed dataset, the authors should consider training baseline video inpainting models (e.g., DiffuEraser or ProPainter) on the same synthetic data and comparing their improvements. Since ROSE builds on top of Wan, a powerful generative model, it's unclear how much of the performance gain comes from the dataset versus the backbone.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **ROSE (Remove Objects with Side Effects)**, a framework for video object removal that explicitly models and removes object-induced visual side effects such as shadows, reflections, lighting changes, and translucency. ROSE combines a novel synthetic dataset generation pipeline using Unreal Engine with a reference-based inpainting model that incorporates a difference mask predictor to guide side-effect localization. The authors also present **ROSE-Bench**, a benchmark tailored for evaluating side-effect-aware object removal, and demonstrate that ROSE significantly outperforms prior methods across synthetic and real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* **Timely Problem Formulation**: The paper targets a practical and underexplored problem (removing not only objects but their visible environmental effects) which is both novel and relevant.\n* **Robust Synthetic Dataset**: The large-scale, automatically generated dataset provides high-quality, temporally aligned video pairs, enabling strong supervision for training and evaluation.\n* **Architectural Design**: The use of full-reference video input and a difference mask predictor enhances temporal consistency and helps isolate object-related changes effectively.\n* **Comprehensive Evaluation**: Experiments span synthetic, realistic, and unpaired real-world videos, with consistent improvements across standard metrics and strong ablation studies that isolate component contributions.\n\n**Weaknesses:**\n\n* **Limited Real-World Validation**: The model’s performance on truly in-the-wild videos is insufficiently analyzed. Quantitative comparisons for real-world unpaired settings are lacking, and failure modes, particularly flickering under motion, are noted but not deeply explored.\n* **Computational Demands**: Both training and inference are resource-intensive due to the diffusion-based backbone and large-scale rendering pipeline. Practical deployment concerns are not addressed.\n* **Ablation Gaps**: While key components are evaluated, there is limited analysis of the difference mask predictor’s standalone effectiveness or its per-category contributions. The backbone model’s role is also not isolated.\n* **Unclear Sim-to-Real Transfer**: The domain gap between synthetic training data and complex real-world scenarios is acknowledged but not quantitatively assessed. The generalization capacity of ROSE remains uncertain.\n\nQUESTIONS:\n1. **Domain Gap**: How does ROSE's performance degrade when applied to uncurated, real-world videos with side effects outside the defined categories? Have you considered domain adaptation techniques to bridge this gap?\n2. **Failure Cases**: Can you provide quantitative metrics and visual examples of failure modes, such as flickering or artifacts in scenes with large motion or complex lighting?\n3. **Difference Mask Evaluation**: What is the standalone accuracy or effectiveness of the difference mask predictor across different types of side effects?\n4. **Inference Efficiency**: What are the model’s runtime characteristics (e.g., FPS, memory usage) on long videos? Are there strategies for reducing compute requirements?\n5. **Generalizability**: How transferable is ROSE to new video domains, such as mobile or drone footage, or to settings with unseen environmental side effects?\n\nLIMITATIONS:\nYes, the authors shortlist main limitations and societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}