Title: ZeroPatcher: Training-free Sampler for Video Inpainting and Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video editing

video inpainting

diffusion model
Abstract: Video inpainting and editing have long been challenging tasks in the video generation community, requiring extensive computational resources and large datasets to train models with satisfactory performance. Recent breakthroughs in large-scale video foundation models have greatly enhanced text-to-video generation capabilities. This naturally leads to the idea of leveraging the prior knowledge from these powerful generators to facilitate video inpainting and editing. In this work, we investigate the feasibility of employing pre-trained text-to-video foundation models for high-quality video inpainting and editing without additional training. Specifically, we introduce a model-agnostic denoising sampler that optimizes the trajectory by maximizing the log-likelihood expectation conditioned on the known video segments. To enable efficient dynamic object removal and replacement, we propose a latent mask fuser that performs accurate video masking directly in latent space, eliminating the need for explicit VAE decoding and encoding. We implement our approach in widely-used foundation generators such as CogVideoX and HunyuanVideo, demonstrating the model-agnostic nature of our sampler. Comprehensive quantitative and qualitative evaluations confirm that our method achieves outstanding video inpainting and editing performance in a plug-and-play fashion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a training-free video inpainting and editing method, named ZeroPatcher. The method is based on the back-projection operation of DDNM, and the authors claim that pure DDNM suffers from two key limitations: the estimation bias and not acquiring any information from previous time steps. Therefore, ZeroPatcher includes additional components such as CD-EM and LMF to reduce those side effects. The visual results reveal that ZeroPatcher outperforms recent methods, and can be extended to both flow-based and denoising-based architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n* The method has good visual results compared to other methods, and can perform both inpainting and editing.\n* The approach can also be extended to both flow-based and denoising-based approaches.\n\nWeaknesses:\n1. The paper has typographical and mathematical notation errors, both in main paper and in appendix. Please go over them. Some suggestions to improve:\n\n* Section 3.1 is too long; please shorten it and use the freed space for an EM section.\n* Notations are confusing across the paper. For instance, clarify that $ f_{\\theta}(x_t) $ in Line 100 is Tweedie’s estimation, mapping $ x_t $ to $ x_0 $. In Equation 8, $ \\mu $ looks like the denoising network output, so $ f_{\\theta} $ and $ \\mu $ should share a common symbol.\n* In Figure 2, consider merging the two denoising-step blocks and explicitly show where latent masking occurs.\n* Line 151 and Equation 10 state that EM comes from differentiating Equation 2 with respect to $ x_t $ and setting the result to zero. Please explain this derivation more clearly.\n* Figure 3 is puzzling because the horse is masked out and replaced with unrelated content. What do we mask out and replace it with? Show the intended outputs of the VVAE decoding stage to clarify.\n\n2. Even though the method looks promising at first, CD-EM and LMF is not justified adequately. CD-EM can be replaced with latent optimization, and LMF can be replaced with pixel-space masking, without a sub-network to train. Then, the overall method becomes a marginal improvement over DDNM. The authors should resolve the related issues in Questions part.\n\nQUESTIONS:\n1. RePaint and COPAINT use an analytical posterior + pixel-space masking that resembles ZeroPatcher’s EM + latent masking. Explain why EM is preferred.\n\n2. I would assume one of the major advantages of CD-EM over latent optimization would be converging to a unique fixed point, as Theorem 3.1 claims. However, no visuals prove that increasing K and P leaves converged results unchanged, visually. I would suggest adding inpainting outputs alongside Figure 6 to further prove your point. Otherwise, why prefer EM over latent sample optimization?\n\n3. Provide an explicit A pseudo-inverse example for a given masking matrix A and show how it is applied to $ x_0 $ via Equation 5.\n\n4. It seems that the majority of the improvements are due to BP (i.e. DDNM), and the added components improve the quantitative metrics marginally. Include a visual ablation that matches Table 3, it might be easier to observe the improvements.\n\n5. The latent spaces of such video models are well-aligned spatially with pixel domain (meaning that a pixel would be represented more or less on similar spatial regions in the latent space, across all latent channels, very unlike GANs' latent spaces). This means that resizing a pixel-space mask and applying it on the latent domain would more or less yield the desired results. However, the method indicates that performing latent masking would be better (Lines 186-187). Visually compare naive masking (pixel-space masking applied directly in latent space without LMF) with LMF.\n\n6. In Figure 4 the car-roundabout example still shows visible shadows. Does extending the mask remove these?\n\nThese questions are related to Weaknesses #2. I stand my rating as borderline reject, but I am going to increase my score 1 to 2 points if these questions are adequately answered and the novelty is justified.\n\nLIMITATIONS:\nThere are no fail cases visualized as the limitations of the method. Also, see the weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel, model-agnostic, training-free video inpainting and editing method that achieves strong performance and is supported by solid theoretical analysis. However, its reliance on multiple auxiliary models and limited comparison with other training-free methods raise concerns about pipeline robustness and experimental completeness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper introduces ZeroPatcher, a novel training-free method for video inpainting and editing, which leverages pre-trained text-to-video foundation models to produce high-quality results without any additional training.\n2. The proposed method is model-agnostic and can be flexibly applied to a range of diffusion-based video generators, including CogVideoX and HunyuanVideo.\n3. Extensive experiments demonstrate the effectiveness of ZeroPatcher in both video inpainting and editing tasks, showing superior performance over existing training-free baselines and even outperforming some trained models on certain metrics.\n4. The paper presents a thorough theoretical analysis of the proposed Conditional Denoising Expectation Maximization (CD-EM) algorithm, including proofs of solution uniqueness and convergence under well-defined conditions.\n\nWeaknesses\n1. Although the method is described as training-free, it still relies on several auxiliary base models. For instance, the use of CogVideoX involves external components such as LLaVA-NeXt and GPT-4 mini (o1) for caption generation, which introduces potential fragility into the overall pipeline. Moreover, the latent mask fuser appears to require additional training, which somewhat compromises the training-free claim.\n2. The paper lacks comparisons with several recent training-free video editing or inpainting methods, such as AnyV2V and ReVideo, which are relevant for a comprehensive evaluation.\n3. Certain formatting issues are present throughout the manuscript, including inconsistent line spacing and layout irregularities (e.g., Line 200 and the caption of Table 2), which may affect readability.\n\nQUESTIONS:\nQuestions\n1. Could the authors provide additional comparisons with recent training-free video editing methods, such as AnyV2V or ReVideo, to better contextualize the performance of ZeroPatcher?\n2. Given the reliance on accurate captions and object descriptions (e.g., from LLaVA-NeXt or GPT-4 mini), how robust is the method when such textual inputs are noisy or imprecise? Have the authors evaluated the model's sensitivity to caption quality?\n3. More questions please refer to the weakness.\n\nLIMITATIONS:\nYes. The authors have clearly acknowledged the limitations of their approach and briefly discussed potential negative societal impacts. Their transparency is appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a model-agnostic method for video inpainting and editing that can be integrated into existing generative video frameworks without requiring additional training. The authors propose a Latent Mask Fuser to enable accurate masking in the latent space, avoiding explicit VAE encoding/decoding. Additionally, they introduce Conditional Denoising Expectation Maximization (CD-EM), which formulates the generative process as an EM problem conditioned on the observed data. Supplementary materials demonstrate the method’s superiority over prior work across video inpainting and editing tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n- The proposed approach is designed to be compatible with a variety of diffusion and flow matching based text-to-video generation models, improving its versatility and practical applicability.\n- The method shows clear improvements in both video inpainting and editing tasks, as demonstrated by qualitative and quantitative comparisons with prior methods.\n\nCons:\n- Due to the EM formulation, the method introduces significant inference-time overhead, as reported in the appendix. Discussion on potential optimization or trade-offs would strengthen the paper.\n- While the proposed components appear effective, visual ablation studies would be helpful in further clarifying their individual contributions.\n- It would be valuable to evaluate the method on top of WAN 2.1, and compare against VACE [a], a state-of-the-art editing model built on WAN 2.1. Such a comparison would evaluate performance under similar architectures and clarify the benefits of the training-free method.\n\n[a] VACE: All-in-One Video Creation and Editing\n\nQUESTIONS:\nWhile the method’s increased inference cost is a concern, its architectural flexibility and empirical performance are compelling. Including additional qualitative ablation and more relevant baselines (e.g., VACE) would make the work even stronger.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors introduce ZeroPatcher, a training-free sampling algorithm that adapts video diffusion models for video inpainting and editing. Specifically, the authors introduce CD-EM to optimise the masked area to match the likelihood of the unmasked regions through a fixed-point EM strategy. The authors also introduce a latent mask fuser that allows them to directly mask at the latent space instead of the pixel space. The authors evaluate their training-free strategy across various models and datasets/tasks and find that their model performs on par with the state-of-the-art training-based model.

STRENGTHS AND WEAKNESSES:
### Strengths:

1. The paper is well written, and the content is easy to follow. 
2. The method appears to be theoretically sound and achieves good results.
3. Since it’s a training-free algorithm, it would allow for easy adoption and development due to its reduced requirements.

### Weaknesses:

1. The added steps of Expectation-Maximisation make the approach slower compared to the baselines.
2. Appendix D: “ZeroPathcer” vs “ZeroPatcher”
3. The notation in eq 8, eq 9, eq 10, eq 11 with $(x_1, x_2)$ seems to be unconventional and confusing.

QUESTIONS:
In the order of importance (not same as the order of ease): 

1. Would you be releasing the code? 
2. How does the model perform under different masking conditions, specifically extreme masking conditions? 
3. How does the inference speed change with the different values of $P, N, K$?
4. Can you provide additional details about the human evaluation performed?

LIMITATIONS:
1. Improved editing ability can lead to a potential negative societal impact. Follow-up works could explore solving EM with additional steering/alignment to prevent generating unwanted content.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03