ZeroPatcher: Training-free Sampler for Video Inpainting and Editing
Authors: Shaoshu Yang
Yingya Zhang
Ran He
Venue: NeurIPS 2025 poster ()
Keywords: Video editing
video inpainting
diffusion model
Primary area: applications

Abstract: Video inpainting and editing have long been challenging tasks in the video generation community, requiring extensive computational resources and large datasets to train models with satisfactory performance. Recent breakthroughs in large-scale video foundation models have greatly enhanced text-to-video generation capabilities. This naturally leads to the idea of leveraging the prior knowledge from these powerful generators to facilitate video inpainting and editing. In this work, we investigate the feasibility of employing pre-trained text-to-video foundation models for high-quality video inpainting and editing without additional training. Specifically, we introduce a model-agnostic denoising sampler that optimizes the trajectory by maximizing the log-likelihood expectation conditioned on the known video segments. To enable efficient dynamic object removal and replacement, we propose a latent mask fuser that performs accurate video masking directly in latent space, eliminating the need for explicit VAE decoding and encoding. We implement our approach in widely-used foundation generators such as CogVideoX and HunyuanVideo, demonstrating the model-agnostic nature of our sampler. Comprehensive quantitative and qualitative evaluations confirm that our method achieves outstanding video inpainting and editing performance in a plug-and-play fashion.

ZeroPatcher: Training-free Sampler for Video Inpainting and Editing
Shaoshu Yang!. Yingya Zhang?, and Ran Hel.
School Artificial Intelligence, University Chinese Academy of Sciences
Tongyi Lab
New Laboratory of Pattern Recognition (NLPR), CASIA
T
+
ZeroPatcher
Video
Inpainting Editing
Object removal
Input
Output
Video aditing:
video copturas
rhinoceros
robobic
Figure I: With ZeroPatcher, text-to-video generative model can achieve video inpainting and video editing without any finetuning.
Abstract
Video inpainting and editing have long been challenging tasks in the video genera- tion community, requiring extensive computational resources and large datasets train models with satisfactory performance. Recent breakthroughs in large-scale video foundation models have greatly enhanced text-to-viis video generation capa- bilities. This naturally leads to the idea of leveraging the prior knowledge from these powerful generators to facilitate video inpainting and editing. In this work, we investigate the feasibility of employing pre-trained text-to-video foundation models for high-quality video inpainting and editing without additional training- Specifically, we introduce a model-agnostic denoising sampler that optimizes the trajectory by maximizing the log-likelihood expectation conditioned on the known video segments. To enable efficient dynamic object removal and replacement, we propose latent mask fuser that performs accurate video masking directly latent space, eliminating the need for explicit VAE decoding and encoding. We imple- ment our approach in widely-used foundation generators such as Cog VideoX and Hunyuan Video, demonstrating the model-agnostic nature of our sampler. Compre- hensive quantitative and qualitative evaluations confirm that our method achieves outstanding video inpainting and editing performance in plug-and-play fashion.
Introduction
Given an input video and a dynamic mask, video inpainting or editing tasks require models to render the masked regions according user specifications. As a long-standing challenge in video research,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

this problem has been approached through various paradigms, including transformers 44 17) and diffusion models (3). Recent advances in foundational video generators [33 28), particularly through large diffusion transformers, have significantly improved video generation capabilities. This progress naturally suggests leveraging these powerful foundation models to advance video inpainting and editing. However, effectively utilizing their conditional generation abilities for these tasks would typically demand substantial computational resources for training, given their massive scale. Furthermore, as foundation models continue to evolve, traditional approaches relying on extensive fine-tuning will face increasing challenges adapting to new video generators. An alternative solution is to employ these video generators data priors, enabling task resolution in a training free manner.
Recent research has extensively explored methods for enabling conditional generation in image diffusion models. For inpainting tasks where significant portions of the image are unavailable, current approaches typically employ back-projection operations (32) to incorporate information from known regions. However, this process does not directly estimate the conditional denoising distribution and may fail when the generation trajectory substantially deviates from the known data (32)
In this work, we introduce ZeroPatcher, novel training-free approach that unlocks video inpainting capabilities in text-to-video foundation models. Our method is theoretically model-agnostic and com- patible with diffusion- based video generators. To approximate the conditional denoising distribution, we propose conditional denoising expectation maximization (CD-EM), which formulates an EM problem over the denoising trajectory conditioned on known data. The framework consists of an expectation step implemented through Monte-Carle sampling and maximization step solved via fixed-point iteration. We further establish the uniqueness of the fixed point in the maximization step through theoretical analysis. While existing methods inject known information by pixel replacement at each denoising step, this approach is incompatible with prevalent latent diffusion models that cannot perform precise masking in latent space. To overcome this limitation, we present mask fuser a lightweight convolutional architecture that enables accurate dynamic masking in latent space.
We perform extensive evaluations on the DAVIS 22) and YouTube- VOS (35) datasets to assess both the inpainting and editing capabilities of our method. By leveraging the generative power of modern video foundation models, our approach achieves performance competitive with trained inpainting models. Furthermore, we demonstrate ZeroPatcher's video editing potential, showing superior ability to modify object shapes while maintaining surrounding content compared to existing methods. Our key contributions are:
We introduce ZeroPatcher, training- frree framework that adapts video diffusion models for video inpainting and editing. The proposed latent mask fuser enables dynamic video masking directly in latent space.
We present conditional denoising expectation maximization (CD-MM) to optimize sampling trajectories using known video regions as guidance, accompanied by theoretical analysis proving solution uniqueness and convergence.
Comprehensive experiments demonstrate our method's model-agnostic nature and compet- itive performance against trained approaches across various video inpainting and editing tasks.
2
Related Works
2.1 Video generative models
Recent advances in generative learning methods, including autoregressive models 29). diffusion models (32) (2) 21), and flow matching models (19). have significantly enhanced video generation capabilities. Early approaches focused on learning video distributions in pixel space, with Video Diffusion Model (12) pioneering the application of diffusion denoising to this domain. Subsequent works like Make-A- Video (25), PYoCo (8), and Imagen Video (10) integrated large language models to enable text-to-vided generation. To address the high dimensionality of video data, recent research has shifted toward latent space learning. VideoGPT (37), combining VQ-VAE (6) with transformer- based next-token prediction, established an early benchmark for latent video modeling. He et al. further advanced this direction by employing diffusion models to approximate latent video distributions, giving rise to latent video diffusion models O 43 34, 31 3.4) VogVideoX (38)

---

introduced diffusion transformers and craporra-compresee video VAEs, substantially improving motion complexity. With growing datasets and computational resourees, large-scale video generators are now achieving unprecedented video generation quality.
28 33
2.2
Video inpainting and editing
Video inpainting and editing represent crucial downstream applications in video generation research. For video inpainting, E2FGVI (I6) employs optical flow to guide the inpainting process, while ProPainter (44) utilizes a transformer architecture to expand the perceptual field. More recently, Zhang et al. (42) introduced diffusion models to video inpainting by training a UNet for masked conditional denoising. The rapid advancement of foundation models has similarly propelled progress in video editing. Approaches like FateZero
(23) and VideoP2P (18) leverage diffusion inversion and attention map control for content swapping while preserving backgrounds, whereas VideoComposer (30) trains an editing-compatibe generator. However, these methods rely on architecture-specife properties. In this work, we investigate aggnosic approach that enables text-to- video generators to perform inpainting and editing through theoretical sampling analysis.
3 Method
3.1 Preliminatry
Diffusion model spatial dimensions and is the latent channel size. Generative models approximate this distribution through a series of Markovian diffusion and denoising transitions (12 26 21 27). The forward and backward diffusion processes are defined as:
Given a video p(xo), Xu E Rxxxxxxx where f, h, w are temporal and
(xxx xo) = (xxx Varso (1 I)
g(x |xe, xo) _N | xx)
where x, denotes the noisy data at timestep = 1, 2. ,T, and the mean (x,,, xo) is a linear combi- nation of x, and Xo. A diffusion model is trained iteratively denoise samples starting from Gaussian noise XT ~ (xT j0, I). The model learns to approximate the reverse diffusion process conditioned on y (e.g. textual prompts) through the distribution Pe y) (x, N N(xxx11 (X,, y). Eg (x,), y)- For notational simplicity, we will generally omit the explicit conditioning on in subsequent sections. The diffusion model is parameterized predict the clean data (x,, Xg- Using this trained model, we can approximate the denoising distribution by substituting the ground truth X9 with the model's prediction: po (x,x1 (x,)) q(x; 1 X,, fo (x.)). Specifically, the learned denoising distribution is
(a (x,) N (xx: l (x()))), o1)
Frree image inpainting The back- projectooettion method (32 has emerged as a prevalent approach for zero-shot image inpainting. This method employs a linear degradation operator A, where the degraded observation is given by x A Axp. Common degradation operators include masking, downsampling, and monochromatization. The complete image can be decomposed into orthogonal components:
Xg
A rangespace
+(I - (--AA)
4
mullspace
where A is the pseudo inverse A such that AA'A A. The range space component represents the preserved information in the degraded observation, while the null space component contains the missing information to be recovered. In training-free inpainting, we assume access to xij but not the original Xo. The key idea is to leverage pretrained diffusion model f-(-) to estimate the null space component during denoising. DDNM (32) proposes a back-projection step that enforces consistency with the observed data:
(x.) = (I- ()) + A Xx
(5)
This formulation ensures the diffusion model only predicts in the null space of A while perfectly preserving the range space component xi in the final output.
3

---

Denolsing step
XT
Xt
X
Back proj.
X
X4-
X0
CD-EM
Optimize trajectory
Denoising step
X
x
Fred et
X
X
Optimize trajectory
Σ los log
Add noke
Xt
b 8--
(xx: (
X
X
arg (xx,|xx,
X
Expectation og
Expectation step
Maximization stop
Figure 2: An illustration of the sampling process with conditional denoising expectation maximization (CD-EM) and fack-projection method. each sampling step, we first use CD-EM to optimize the sampling trajectory with the guidance of the known data. Then we inject known information through pack-projection method.
3.2 Formulation
Consider a data instance Xo = [x).x()> E IR composed of masked x. and unmasked components
xß. where the partitioning is defined by a binary mask m E Zd, m, E {0, 1), x3 = {xg (x/]mm In = 0}. x = {x/ mm = 1}. This mask-based partitioning extends naturally to noisy data x, at any timestep Given pretrained diffusion model f-(-) that estimates clean data from noisy inputs fxx),, our goal is to recover x through conditional denoising while preserving xi starting from
3.3 Conditional denoising expectation maximization
For inpainting tasks, we observe the ground truth denoising trajectory for known segments x?. Current proj-proeection methods sample from (o 1x x)) (32 12), where a represents noisy inpainting results. However, this approach suffers from two key limitations:
The estimation is biased from the true conditional denoising distribution (x X2. xfi. While diffusion models approximate the unconditional distribution po 1o (x.). learning the conditional variant requires additional training.
The method only leverages information from æ at noise level t, ignoring less noisy versions (e.g. x²_,) that contain more reliable information.
Since pretrained text-to-video diffusion models lack explicit conditional generation capabilities, we circumvent the need for po px(x-1|xx; x6) by reformulating the sampling process as maximum likelihood estimation problem. The pipeline of the method is presented in fig, 2. Our proposed conditional denoising expectation maximization (CD-EM) method optimizes the sampling trajectory to maximize the joint likelihood We aim to follow sampling trajectory that maximizes the likelihood po(x"_ x xx)) Within this likelihood, at-1 denotes the unknown masked component. While xp_ - + /1 04-16-11 represents the known unmasked ones. The expectation of the log-likelihood is defined as:
Q(xx|xxl)) = I
Po p log (o x/ Xx, xxx
x/_
We use xepld to denote the old noisy latent in the previous EM step. In eq. (6), sampling from pe(x; (xold) requires neural network forward propagation and thus the integration is intractable.

---

We can approximate it with Monte-Carie integration:
N
Q(x (x(
Σ
log pe (x,
bi X:
1x)) xx)),
(7)
bi where x sampled from pa (xx- (xpld) and N is the number of samples. Since the likelihood
term is Gaussian. We analytically express the approximated expectation and our objective:
n
Q(x:|x; xt ~ 20
Σ 1
x(-1) - u(xe, (x(x))
(8)
x arg min 1xx b.d
u(x,, (x:
Given this, we can optimize the conditional denoising trajectory by iteratively computing and maximizing the expectation. This method demonstrates two advantages:
We does not need to spend additional training to approximate po(x xß). Therefore, our method is training-free free and model-agnostic.
Instead of using x; as the condition for the denoising step. CD-EM utilizes x which is less noisy and therefore provides more reliable guidance.
3.4 Solving the maximization
In this section, we present our approach to maximize the expectation in eq. (9). The objective is to find a x, that satisfies:
(x where c/_ 1: 22 are constant coefficients. Since this equation cannot be solved analytically due to the neural network function f-).) In alternative, we propose to employ fixed point iteration to obtain an approximate solution. special case when N 1, the iteration formula becomes:
(x) (x + c- + c fo(x)
(10)
(x
2 (x,)
(11)
d-1
We find N = is able to get plausible results while maintaining computational efficiency. The complete deduction and the matrix formation when N > 1 are provided in the appendix. The effectiveness of our fixed point iteration method depends on solution uniqueness and convergence. We establish these properties the used diffusion model is well trained and N 1.
Theorem 3.1. Given diffusion model f-).), a noisy latent Xt, and a future step known data xi-1- the diffusion model is well trained with Lipschitz constant E. The score function s(.) is approximated through so sxx)) = s(x))xo). The approximation satisfies
[|]) so(x))llz C,llx] - x2112.
(12)
where x. E Rdd and x² E Rd are two arbitrary noisy samples. Then there is a noise schedule (oo, ... aT)) (0o, , and an ending timestep to- For 1. Z to, the iteration x, <-
x b.l + (o)()( /ct_1 converges unique fixed point.
The proof of theorem 3.1 is shown in the appendix. This theorem guarantees that CD-EM can effectively optimize the noisy latent x; through fixed point iteration. Let K denote the number of fixed point iterations and let P denote the number of EM iterations. While our method involves both Monte Carlo integration and fixed point iteration, experiments demonstrate that nighh-qulity, consistent results can be achieved with N = K = 1, P = 2 without significant computational overhead. Notably, when using step of fixed point iteration, no additional neural network forward passes are required.
3.5 Sampling with CD-EM
Our method operates independently or with back projection techniques (3). As shown in fig. back projection follows CD-EM optimization. Let x denote the CD-EM optimized latent. We then apply
5

---

Latent
Latent
encode
decode
Latent 2 m&m
Fused Infent X
Maskc Intertt
Figure 3: Sketched pipeline of latent mask fuser. We use VVAE to abbreviate video VAE. The model use two videos be fused and the pixel space dynamic mask the input. After encoding them with the encoder, the latent mask fuser concatenates the three inputs in channel dimension and perform video fusing in latent space.
denoising X po (X;-1 x For CD-EM only. optimize x;-1 x2-1- Repeat this process until we reach Xo- With back projection, we inject range space data during sampling. Mathematically, the process is
1x7 Use back proj.
X1-1
(13)
po -
Others
fx(x)) = (xi, 16 (x;)),
(14)
where (xt) {f) 1} is the masked portion of fx;)). CD-EM shows significant improvements in early denoising stages when visual layouts are determined, with subtle effects in later stages. We suspend CD-EM when < 7 to save computation (see appendix for algorithm).
3.6 Latent mask fuser
Pixel-space masks m E gd are straightforward to apply. and can be represented via DDNM's linear degeneration matrix A (32) However, modern generators using VAES {E, D) create compressed latent representations where direct masking becomes non-trivial trivial due to lost spatial correspondence. Simple approaches like resizing pixel masks latent space lose accuracy, while decoding to pixel space introduces computational overhead and potential information loss.
We propose latent mask fuser M(-) to achieve fast video masking in video latent space. The module leverages liight convolutional architecture. Let m E Z' denote the mask in pixel space with length = x H x x where the capital letters T, H, W are the video dimensions pixel space. Given two videos in the latent space X1 E Rd, X2 Rd the latent mask fuser computes the output via
x x= M
X2, (
(15)
where E(m) is the encoded mask. With this, and replacement in latent space. Let &, R E Red denote the videos in pixel space and X1 = (x)), X2 = E82).. We aim to approximate the masking operation in pixel space
achieve fast and accurate video masking
if m = m
x =
X2),
X
(16)
where we use the superscript to denote the element at the i-th entry. Then the training loss for the latent mask fuser is
L Ex k.
|||
x|| + A LPIPS(XP x)]
(17)
we use a series of simplex noise generators to produce dynamic random masks m.
4 Experiments
4.1 Details
We apply our method to a video diffusion models Cog VideoX 13 38) which uses a diffusion transformer architecture. We also show ZeroPatcher can be used in Hunyuan Video, please see it in

---

Imput
EXFOW
ProPainter
DOWM
Figure 4: A visual comparison in video inpainting.
The pertt it sveting probactivn sear leilyy joctey thit Ugghneter
partan walley on rodly beriale eserdty appuars huranal reter pancl beck
Figure 5: A visual comparison in video editing. We mark the changes in editing textual prompt using red (source) and green (editing).
the appendix. With Videox, we generate 49 720 480 videos using discrete euler sampler (L) To use dynamic video mask, we train the latent mask fuser on Cog Video VAE using small subset including 20,000 smaples from Panda-70MB) The video resolution used during the training is 17 x 256 x 256. The exact inference hyperparameters. the training configuration of the latent mask fuser, and comprehensive computation and memory cost analysis can be found in the appendix.
For video inpainting evaluation, conduct experiments DAVIS 22 (50 videos) and YouTube- VOS (35) (508 videos) using their original splits (44). As Cog VideoX requires text guidance, we use LLa Va-Nett (41) to generate precise original prompts identifying main objects, then modify them with GPT-4 mini ol by removing object descriptions to create inpainting prompts. Following (44), we evaluate using PSNR and SSIM for unmasked region similarity to source videos, VFID for visual realism, and Ewarp for temporal consistency, with all inputs center-cropped to model resolution.
For video editing evaluation, test 50 DAVIS videos following (42) 18), using GPT-4 mini ol to create editing prompts by modifying original video descriptions. We assess performance using CL.P scores for text alignment, PSNR/SSIM for background preservation, and Ewarp for temporal smoothness. Additionally, we conduct human evaluation collecting 1,310 responses from 171 participants who compared results DAVIS.
7

---

Youtube- VOS
DAVIS
Model
PSNRT
SSIM f VFIDI
Ewarp
PSNR
SSIM
VFID
Ewarp +
Trained
FuseFormer
33.32
0,9681
0.053
1.053
32.59
0.9701
0.137
1.349
ISVI
30.34
0,9458
0.077
1.008
32.17
0.9588
0.189
1.291
FGT
32.17
0,9599
0.054
1.025
32.86
0.9650
0.129
1.323
E2FGVI
33.71
0.9700
0.046
1.013
33.01
0.9721
0.116
1.289
ProPainter
34.43
0.9735
0.042
0.974
34.47
0.9776
0.098
1.187
Training free
SDEdit
26.41
0.8436
0.089
0.988
25.89
0.8266
0.203
1.266
DDNM
34.13
0,9705
0.067
1.069
33.97
0.9744
0.127
1.376
Ours
34.33
0.9715
0.039
0.961
34.13
0.9765
0.096
1.163
Table Quantitative comparisons of video inpainting on Youtube-VOS (36) and DAVIS (22). Note that we ensure the number of computation using the denoising network 100 for all training-free methods for a fair comparison. This indicates the computation cost for SDEdit, DDNM, and ours are almost the same in this experiment.
Model
Automatic metrics
Human evaluation
CLIP score
PSNR
SSIM
Ewarp
BC
TA
PF
0.3341
34.07
0.9684
1.513
30.6%
2.3 %
VideoComposer
0.3172
1.361
12.8 %
Video-P2P
0.3313
30.47
0.9433
1.457
9.3%
15.3 %
DDNM
0.3267
33.48
0.9649
1.264
29.2%
8.9 %
Ours
0.3543
34.23
0.9712
1.231
30.9%
60.7%
Table Quantitative comparisons editing performance. PF denotes per-frame video inpainting using stable diffusion inpainting (24). BC represents back ground preservation. TA denotes text-video alignment. We ignore PSNR, SSIM, and TA metrics in VideoComposer because does not have ability to maintain the background.
4.2 Comparisons
We evaluate ZeroPatcher on both video inpainting and editing tasks, comparing against training-free baselines and trained models.
Computation and memory cost We show the throughput and computation overhead of ZeroPatcher in the appendix.
Video inpainting We compare against trained methods (FuseFormer (17), ISVI 40), FGT (3)., E2FGVI (16), ProPainter (44)) and training-free baselines (SDEdit (20), DDNM (32). To use SDEdit in video inpainting. we fill the main object with the mean colour the video background. For a fair comparison, the number of function evaluations (NFE) is 100 for all training-free methods. The results are presented table ZeroPatcher achieves the state-of-the- maining-frre methods and is very competitive
performance within the compared with the training based counterparts. Note that we achieved the best performance in Ewarp in Youtube- VOS among all the methods. A visual comparison is provided in fig. 4. Our method has strong generation capability and can preserve good consistency between the removed parts and the background. The generation ability is very useful especially when the inpainting region is large and lack of references.
Video editing We compare ZeroPatcher with stable diffusion inpainting model (24), VideoCom- poser (30), Video-P2P (88 and DDNM (32). We rephrase our editing text prompts for Video-P2P, because it only allows users to change a word from the original description. The quantitative results are shown in table ZeroPather achieves the state-of-the-ar performance in quantitative metrics. Notebly, VideoComposer uses depth guide video editing, ignore the background similarity

---

Model
PSNR
SSIM f
VFID
Ewarp
Ours
34.13
0.9765
0.106
1.213
W.O. CD-EM
33.89
0.9711
0.123
1.355
W.o. LMF
33.61
0.9673
0.125
1.361
w.o. BP
17.41
0.5351
0.116
1.224
Table Ablation results on the technical components. We use BP to represent back projection and use LMF to represent the latent mask fuser.
0.000
0.009
0.097
0.003
0.053
0.083
-.-Ked, Ne1 2x2, N-1
K-2, N-2
K51, PC1
Kel, -M-M-2, P:2
N-2,
Figure 6: Ablation experiment on choice of P, N, and K parameters.
for VideoComposer. A visual comparison is shown in fig. 5. Our method achieves the best text-video alignment, video smoothness, and background similarity. 4.3 Ablation
We evaluated the individual technical components of ZeroPatcher study their effectiveness. Specif- ically, the ablation experiments are finished based on the pretrained CogVideoX (38) generator to produce 49 720 480 videos. We measure the impact of technical components including CD-EM, back projection, and latent mask fuser. The results are shown in table 3. Using CD-EM or back projection alone cannot achieve an ideal performance. Incorporating both methods can exploit the capability of the foundation model to the limit. When mask fuser improves lot in terms visual fidelity compared with using resized masks. Meanwhile, the visual comparison shows our method achieves accurate video masking with subtle reconstruction loss.
using dynamic video masks, the latent
A series of ablation experiments are conducted to validate the parameters in ZeroPatcher. We make a grid search over the sample number of Monte-Carle integration N, the fixed point iteration step number K, and the FM outer-loop iteration number P. The results are shown in fig. 6. We perform video inpainting on DAVIS and compute VFID) The experiment suggests when N, K, and P become larger, the performance of our method increases monotonically. Compared to the case N 1, K 1, P = 2, the largest computation setting achieves approximately 3% improvement on VFID. The iteration speed is shown in the appendix. Our method is able to achieve convincing performance without considerably more computation.
Conclusion and limitation
In this paper, we present ZeroPatcher, a training-free plugin to introduce video inpainting and editing capability for pre-trained video foundation models. Our method optimizes denoising trajectory given the unmasked video through conditional denoising expectation maximization. To achieve the video masking in video VAE features, we propose latent mask fuser to achieve fast and accurate video masked fusing. ZeroPatcher widely applicable to prevalent diffusion generators and demonstrates satisfactory performance in inpainting and editing tasks. ZeroPatcher relies on the prior knowledge of pretrained text-to-video generation model to perform video inpainting and editing. As a result, the input data or textual prompts are beyond the domain of the pretrained model, our method may not be able to correctly render the video.
Acknowledgement
This work supported by Alibaba Innovative Research Program.

---

References
[1] Blattmann, A.. Dockhom, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi. Y., English, Z.. Voleti, V., Letts, A., Jampani, V., Rombach, R.: Stable video diffusion: Sealing latent video diffusion models to large datasets (2023)
[2] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models, In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
[3] Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang. S., Xing, J., Liu, Y., Chen, Q. Wang, X., Weng, C., Shan, Y:: Videocrafterl: Open diffusion models for high-quality video generation (2023)
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C.,, Shan, Y:: Videocrafter2: Overcoming data limitations for high-quality video diffusion models (2024)
[5] Chen, T.S., Starohin, A., Menapace, W., Deyneka, E., Chao, H.w., Jeon, B.E., Fang. Y., Lee, H.Y., Ren, J., Yang, M.H., Tulyakov, S.: Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)
[6] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 12873- 12883 (2021)
Garber, T., Tirer, T:: Image restoration by denoising diffusion models with iteratively preconditioned guidance. In: Proceedings the IEEE/CVF conference on computer vision and pattern recognition (2024)
[8] Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang. J.B., Liu, M.Y., Balaji, Y.: Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305. 10474 (2023)
[9] He, Y., Yang, T., Zhang, Y., Shan, Y., Chen, Q:: Latent vidco diffusion models for high-fidelity vidco generation with arbitrary lengths. arXiv preprint #xXv:2211.13221 (2022)
[10] Ho, J., Chan, W., Saharia, C., Whang, Gao, R., Gritsenko, A.,, Kingma, D.P. Poole, B., Norouzi, M., Fleet, D.J., al.: Imagen video: High definition video generation with diffusion models, arXiv preprint arXiv:2210003303 (2022)
[11] Ho, J., Jain, A., Abbeel, P.: Denoising diffuusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840-6851 (2020)
[12] Ho. J.. Salimans. T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, DJJ.: Video diffusion models. arXiv preprint arXiv:2204.03458 (2022)
[13] Hong, W., Ding, M., Zheng, W., Liu, X., Tang, J.: Cogvideo: Large-scale pretraining for text-to-vidco generation via transformers, arXiv preprint arXiv:2205. 15868 (2022)
[14] Huang, X., Belongie, S.: Arbitrary style transfer real-time with adaptive instance normalization. In: Proceedings of the IEFF/CVF International Conference on Computer Vision (2017)
[15] Li, X., Xue, H., Ren, P. Bo, L.: Diffueraser: A diffusion model for video inpainting (2025), https //axivv. wwwiiv.org/abbb www./oggg//b/o/oooooooooooe /50///ab////..... 2501 10018
[16] Li, 7., Lu, C.Z., Qin, J., Guo, C.I.,, Cheng, M.M.: Towards an end-to-end framework for flow-guided video inpainting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[17] Liu, R., Deng, H., Huang, Y., Shi, X., Lu, L., Sun. W., Wang, X., Dai. J.. Li, H.: Fuseformer: Fusing gneinee information transformers for video inpainting. In: International Conference Computer Vision (ICCV) (2021)
[18] Liu, S., Zhang. Y.. Li, W., Lin. Z., Jia, J.: Video-p2p: Video editing with cross- cooss-atenttion control (2023)
[19] Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow (2023), /tttps:///araet
org/ abs /2099. 03003
[20] Meng, C., He, Y., Song. Y., Song, J.. Wu. J., Zhu, J.Y., Ermon, S.: SDEdit: Guided image synthesis and editing with stochastic differential equations. In: International Conference on Learning Representations (2022)
10

---

[21] Nichol, A.Q.. Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International Conference Machine Learning pp. 8162-8171. PMLR (2021)
[22] Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L.,, Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2016)
[23] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.0993 09533 (2023)
[24] Rombach, R.. Blattmann, A.. Lorenz, D., Esser, P., Ommer. B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684-10695 (June 2022)
[25] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q. Yang. H., Ashual, O., Gafni, O., al.: Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022)
[26] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010002502 (2020)
[27] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1.13455 (2020)
[28] Team, G.: Mochi
(2024)
[29] Tian, K., Jiang, Y., Yuan, Z., Peng, B., Wang, L.: Visual autoregressive modeling: Scalable image generation via next-scale prediction (2024)
[30] Wang, X., Yuan, H., Zhang. S., Chen, D., Wang. J., Zhang, Y., Shen, Y., Zhao, D., Zhou, J.: Videocomposer: Compositional video synthesis with motion controllability (2023)
[31] Wang, Y., Chen, X., Ma, X., Zhou, Huang, Z., Wang, Y., Yang, Hc, Y., Yu, J., Yang, P. Guo, Y., Wu,
T., Si, C., Jiang, Y., Chen, C., Loy, C.C., Dai, B., Lin, D., Qiao, Y., Liu, Z.: Lavie: High-quality vidco generation with cascaded latent diffusion models (2023)
[32] Wang. Y.. Yu, J.. Zhang. J.: Zero-shot image restoration using denoising diffiusion null-space model. The Eleventh International Conference on Learning Representations (2023)
[33] Weijie Kong. Qi Tian, Z.Z.R.................. W.J.K. A.Q. WWW.LLDEEEHHHHW..B........M Jie Jiang, a.w. :. Hunyuanvideo: A systematic framework for large video generative models (2024), /taps: org/ /abb//2412. 03603
W.M.L.P..
[34] Xing, J., Xia, M., Liu, Y., Zhang, Y., Zhang, Y., Hc, Y., Liu, H., Chen, H., Cun, X., Wang, X., et al.: Make-your-video: Customized video generation using textual and structural guidance, arXiv preprint arXiv:2306.00943 (2023)
[35] Xu. N.. Yang, L.. Fan, Y., Yang, J.. Yue, D., Liang, Y.. Price, B., Cohen, S., Huang. T.: Youtube-vos: Scquence-to-sequeee video object segmentation (2018)
[36] Xu, N., Yang. L., Fan, Y., Yue, D., Liang, Y.. Yang. J., Huang. T.S.: Youtube-vos: A large-scale video object segmentation benchmark. CoRR abs/1809.03322 003277 (ttp))..ttt://aax// org/ hts///bs//1000 0tt//8899.033 03327
[37] Yan, W., Zhang, Y., Abbeel, P. Srinivas, A.: Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2 104::2004.11157 (2021)
[38] Yang, Z., Teng, J.. Zheng, W., Ding. M., Huang, S., Xu, J., Yang. Y., Hong. W., Zhang. X., Feng, G., al.: Cogvideox: Text-to-vides diffusion models with an expert transformer. arXiv preprint arXiv:2408.0600 06072 (2024)
[39] Zhang, K., Fu, J., Liu, D.: Flow-guided transformer for vidco inpainting. In: European Conference on Computer Vision. pp. 74 90. Springer (2022)
[40] Zhang, K., Fu, J., Liu, D.: Inertia-guided flow completion and style fusion video inpainting, In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5982-5991 (June 2022)
[41] Zhang, Y. Li. B., Liu, h. Lee, Y.j. Gui, L., Fu, D., Feng. J. Liu, Z. Li, C.: Llava-next: strong zero-shot video understanding model (April 2024), httla://llava- v1, vl.github. 10/ blog// 2024 04- 30- lllva-nezeee newt- veieo/ideo/

---

[42] Zhang, Z., Wu, B., Wang. X., Luo, Y., Zhang. L., Zhao, Y., Vajda. P., Metaxas, D., Yu, L.: Avid: Any-length video inpainting with diffusion model. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (2024)
[43] Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., Feng, J: Magicvidco: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211 22t:22111111188 (2022)
[44] Zhou, S., Li, Chan, K.C., Loy. C.C.: ProPainter: Improving propagation and transformer for video inpainting. In: Proceedings of LEEE International Conference on Computer Vision (ICCV) (2023)
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims are closely related to the contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification:
We show the limitations of the method in main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
12

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
All theoretical result is supported by the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We show all the details to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
13

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification:
Code and checkpoints will be released after the review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blcc/gudeeeeeeeeeeieeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show all the details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We use the metrics in this research field.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
14

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Obey the rules.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are in the script.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
15

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Related works properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
16

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
17

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We use LLM to produce textual prompts for evaluation only.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur// htt.://p//.///ofeeeeeee///////// for what should or should not be described.
18

---

A Detailed deduction of the fixed point method
Below, we show the matrix formation for the fixed point iteration that soles CD-EM. To simplify the equation, we use two constants c/_ c- to represent the coefficients. d_1 and di1 are mathematically
1-11- 04-1 oi-1
aa- 04)
01-1 - -0 Von
2--1=
(18)
04
04
We aim to make the equation holds:
(x x (x(x), = (xx_1, x + c X + (o()).
(19)
We aim to find a X; that ensures
x x + c + 2 (f
(20)
in order to maximize the expectation log-likelihodd. Unluckily, this equation can not be solved analytically since f-(:) is neural function. In alternative, we propose fixed point iteration to get an approximated solution. We rewrite eq. (20) to matrix formation. Mathmatically that is
((x 1s
b,1 fo ((xf-1,
x x + (0)0)((
6.2
I
(21)
((xf [(x]_1,>> 4 + fo
where we use ";" to denote concatenation in column, and X, is at the previous fixed point iteration step. We use I denote the Nd x matrix I [I I formation of least square solution. Specifically, the solution is
I" The fixed point iteration follows
x x-1 + c7 fx()) ((x
6.1
X1
(1"T)-11 -I
(xxf X2 1-1
+ 17 fx()))
(22)
(xx (. + 2 fx(()))
B Proof of theorem 3.1
We show the proof in this section.
Proof. We first study what we need fg (-) to be like in order to ensure an unique fixed point. We use g(-) denote the iteration step
- g(x)), 9(x:) = (x
- 1.1 (x-))x
+ e fo(x))
(23)
c/1
According to Banach's 's fixed point theorem, we need g(.) to guarantee smaller than Lipschitz constant
(1x();) - g(x))ll2 < ||x] x ll
(24)
for arbitrary x! x². Expanding g(-) to the fo(.) formation, we derive
(xf-1:x/1) b.l x ot (ox()
(xf_1, x
b.1 fo
< [|x]
x2112
(25)
c/
2
c 1 [(fo()))
fx(()))
(x22)
(26)
c/1
[x+()
fx(x3))l
<
|x] x2112.
(27)
19

---

Therefore, we require the Lipschitz constant of f-) denoted by C, to meet Cy < c of
Note
that d-1 > 0 and 2 > 0. The former one is obvious since we pre-define 0r-1 to satisfy 1 1- 0 0/-1 > O. In a diffusion model, ai < 0 L E (I...,,T)) Therefore we also derive
a1 at V 11 at-1 oل-1 which makes the denominator of c-_ positive. We then relate the Lipschitz constant L, to that of the real score function L. The clean data estimant can be expressed
by
fx(x)) = X v1 - (x) X2 + (x1)
(28)
Va,
Vaz
We construct an inequality that can be the sufficient condition for to find the Lipschitz constant of f-)) Assume the following inequality is satisfied
4-IVa
1
1+ 1+6.
(29)
G-1
c-1vaa
|x| - x 1l2 + [x] x;, 112 <
||x] - -x,11 112
(30)
sq. (12) ||x xill2 ||=o(x²) so sx )22
||= - xill2 + cx
x2lla
G--va
|x
xilla
(31)
triangular, inequality
[[(x) x,) + (so(x)) so(x)))11z <
si-VVor || - x22ll2
| x²ll2 1112 + [1sp - so(x))lla <
(32)
9-1
x + (xx)
x2 + sx()))
[x+]) fo (x
1
|
x² 222-
(33)
Va
Vai
Therefore, if 1 + L. Lipschitz constant of the real score function s(.) is computed below. The score function is defined as
is satisfied, we ensure an unique fixed point can be found. Then, the
sxx Vx, logp(x;(xo) Xt Varxo
(34)
1 at
We derive the Lipschitz constant through
(x (xo) - s(x; (x))l22 =
|| x /
VaPx
X
(35)
1 a6
x
x 1|2
< L. x
x² 11
(36)
1 at
Therefore, the Lipsethiz constant is L. = 11((1 a²). Replacing this and the definition of c o to eq. (29) we have
2 Ot
(1- a- - 1)a
(1-a1-1-
<
(37)
at
V Vat-1(1 - -wt) 1(1 at-1 - 2 -0
rearrange
(2 ar)
(Va--1(1- - 0z0) V(1 a:-1 - 02-1)a0 <1 -aaiaa/(1 (1 01-1 - o²_1)a
(38)
(2-a))aa-1(1- < (3 /a a0--1 -1.0--1---12
(39)
Knowing whether this inequality holds not obvious. Instead of seeking what kind of noise schedule makes the condition holds, we study the commonly used noise schedules. We use m(t)
20

---

Algorithm 1: Conditional denoising expectation maximization for diffusion model
Known video segment z. pre-trained diffusion model stopping criteria T, inference timesteps T, EM iteration number P, Monte-Carlo sample number N, fixed point iteration step number K
Input:
Output: Rendered video segment zg
3 2T ~ 10,
for t T to do
for p = 1 to do
Ftt -- 8t
for i = to N do
_;_1 ~ Po (a) x
end
(O,
11
Fi-1 1 v 04- + vI 16
12
for k = 1 to K do
13
Apply eq, (22) to get Cz.
14
end
15
end
2t-1 Po kt,)
end
18
return 20
003000
4b-03
eta-1.0
-.200.0
-00-03
-00-00
-.0
0.35
0.01
0.3
0.25
0.2
ml0
-0.01
Timesbaps
Timestaps
Figure 7: The curve of m(t) under different settings of 17. The right curve zoom in the regions that curves hit zero line.
0.-1 (3 2a.) 2 and uniqueness hold when m(t) > O. We plot m(t) curve for series of diffusion schedules. The results are shown fig. 7. We plot situation Cog VideoX scheduler with different choices r). In most sampling timestep, we can ensure uniqueness and convergency.
a;)) for simplification. Then the convergency
C CD-EM on flow matching generators
Our method is not only applicable to diffusion based generators. Instead, CD-EM is also applicable to flow matching generative models. Consider a model that uses the following flows and velocity field in t [0, T").
2 (xx)) 0t
3o((). åt
(xo) o())x (1 (()),, (x))((x))
(xo
c),
(40)
where (-)) is the noise schedule, and a random gaussian noise. The velocity field ue(-) will support probability path
(xx:(xo) (Nxx/lo(t)xx (1 II)
(41)
21

---

Solving the ODE in Eq.(4.1.1) produces the sampling result. Euler sampler is commonly used in which the iteration is
X = x + (s (xx), 0 S 8 <IST
(42)
To optimize the model throught flow matching, one usually use a neural network va(-) to approximate the scaled velocity field (x(),) = e Xu, uy (xx) (t) vo(x.). Replace the velocity in Eq.(4.1.3) with the model estimation, we derive
x, x + (a(() o(s))
(43)
To optimize the trajectory with expectation maximization method, we need know the sampling posterior distribution g(x,/Xx, ((xx/xx,xx)) Xu), 0 I < t T of the flow model. Applying bayes formula, we get
(x, x1, = g(xe|x,, (00))xx.xx))(x,b))))) xx)x(xx|xx)
(44)
((x1(x0))
The distribution (xo)) and qxx (X0) are already known. However, since a flow model is deter- ministic, the term in the nominator g(x,|x,, Xo) is specifically Dirac delta. Therefore, the sampling distribution will also be Dirac delta that
1 1 ((s) (x, Xb) = 8 X,
X + a(t) o(s) xo)
(45)
1- I()
o()
Mcanwhile, also derive the the modeled sampling distribution px (x,) 8 (x, x; (t) - ((s))vo (x:)).
(46)
Given this, we are able to maximize the expectation in (2.2.3)
Σ log po (x x xx )
(47)
which requires po (x x xx) to be maximized. In this case, expect the following equation set to hold
(x,, x h.1) = x, + (I) a o (x))
(x 2 + (a(s) (x)))
(x,, x = X4 + (a(t) - (x))
(x 2) 2) (a(s) (t)) vx))v
I
X
(x,, x NN + (a(s) - v()))v (xpld)
( boN = x, + (o(1) (+))
(48)
I E R=*x is an identity matrix. We solve this equation set with least square and produces the fixed point iteration step
(x] x b.1) + (a(s) a (()) ); x + ot))voo (x( gold)
X (1'1)-1
(49)
[xf, x ) - (a(s) (tt) (xpid) where we use II to denote the Nd x d matrix I [ I I T Similarly, when N 1, the iteration step is implified to
(x..x") (a(() o vo
(50)
To uniqueness and convergency of the fixed point iteration can be found using similar way in appendix B. We try this method on flow matching video generator named Hunyuan Video 33 to perform video editing. Please see our supplementary material.
D Computation and memory cost
Ablation experiment The computation and memory cost of ZeroPathcer shown table The theoretical complexity considers the computation required for each denoising step. We let the complexity for every function evaluation be O(1). P is the outer loop step number of the expectation maximization. N denotes how many xŞ_1 are sampled in order to compute the likelihood expectation. K is the step number of the fixed point iteration. With reasonable set of hyperparameters (i.e. P 2, N 1, K = 1), ZeroPathcer will not introduce enormous computation compared to the direct text-to-video inference. Meanwhile, we show that the ZeroPathcer does not use considerable addition memory which is usually required for attention manipulation methods like Video-P2P (8) and FateZero (23).
22

---

Model
Theoretical complexity
Memory
Speed
Inference params
2, 1
(2v model
O(1)
22,489MB
2.61 s/it
BP
0(1)
22MB12MB
2.63 s/it
CD-EM
0(1 + PN P(K - 1))
22,567MB
7.41 s/it
LMF
O(1 + PN -(K-1) 1))
22.893MB
7.63 s/it
Table Theoretical and practical computation cost analysis of our method. Theoretical complexity
and speed show the computation required for each denoising step. We use LMF to represent the latent mask fuser. The speed is computed under seconds per iteration.
Method
Resolution
Memory
NFE
Speed
Text-to-video
49 x 720 x 480
22.488MB
30
78.3 s/sample
Inpainting
SDEdit
49 x 720 x 480
22.503MB
100
240.3 s/sample
DDNM
49 x 720 x 480
22.51
100
241.7 s/sample
Ours
49 x 720 x 480
22,567MB
100
250.9 s/sample
Editing
PF
512 x 512
4,279 MB
50
81.2 s/sample
VideoComposer
16 x 256 x 256
7,394MB
50
101.4
s/sample
VideoP2P
512 512
19,453MB
100
137.8 s/sample
DDNM
49 x 720 x 480
22.522MB
100
241.5 s/sample
Ours
x 720 x 480
22,569MB
100
249.4 s/sample
Table 5: A comparison over computation and memory usage.
Comparison with other methods We compare the memory and throughput among the methods. The result is shown table 5. Our method achieves remarkable improvements over the training- free methods under similar computation cost. To achieve faster inference, one can choose to trade performance speed by letting N 1, K P
E
Algorithmic description of CD-EM
We show an algorithmic description of CD-EM in Algo. provides the clean CD-EM without back projection and latent mask fuser.
F Details of latent mask fuser
In this sector, we briefly introduce the architecture of our latent mask fuser. is consists of 6 residual blocks, a convolutional input layer, and a convolutional output layer. After the 3-th block, we use an adaptive instance normalization layer (14) to perform mask fusing. Each residual block is made of two convolution blocks concatenation of GroupNorm layer, SiLU layer, and a 3D convolution layer) and skip connection layer achieved by a x 1 x 3D convolution. During inference, concatenate three latent features from two source video and the mask in channel dimension, and then the feature is feed into the input layer. No upsampling and downsampling modules are used in latent mask fuser. The model uses a channel size of 1536.
G Inference details and hyperparameters
The inference hyperparameters are shown table We use DDIM diffusion sampler with sampling stochasticity factor 7 1.0. Our experiment shows using 1.0 greatly increase video
23

---

Hyperparameter
CogVideoX
EM outer loop steps P
2
Number of samples in expectation N
1
Fixed point iteration steps K
Sampler
DDIM
Inference steps
50
CD-EM stop step
25
NFEs
100
Classifier-free guidance scale
6
cta
1.0
Table 6: Hyperparameters use in inference.
consistency since the model will not follow the deterministic sampling trajectory. CD-EM is not always required in all denoising timesteps. We find using CD-EM early sampling stages can already produce plausible results. Therefore, add stoping step for CD-EM at 25 to save computation without losing noticeable performance. With CD-EM hypermarameters set to = 2, N = 1, K we ensure every denoising step with CD-EM will only cost 3 NFEs. With the remaining 25 steps
using only 1 NFE, sampling through ZeroPatcher requires 100 NFEs in total. We use 8 NVIDIA A100 80G GPUs to run in parallel during inference get results faster.
H Additional visual results
We attach result videos in our supplementary material. We would appreciate it if you check them.
I Broader impact
The integration of pre-trained text-to-video foundation models into video inpainting and editing workflows presents transformative opportunities alongside critical ethical challenges, By enabling high-quality video manipulation without additional training, this approach democratizes access to advanced editing tools, empowering independent creators and small studios to achieve professional- grade results potentillll revitalizing archival restoration efforts and lowering costs for cultural heritage preservation. However, the same capabilities raise significant concerns: the efficiency of dynamic object removal and latent-space masking could streamline the creation of convincing deepfakes, exacerbating misinformation risks in era already plagued by synthetic media distrust. The model-agnostic nature of the method further amplifies these risks, as could be applied to any foundation model, including those with fewer ethical safeguards.
24


Reviews:
[{"review_id": "iZgzkDZ6sj", "reviewer": "Reviewer_U6KC", "summary": "The authors introduce ZeroPatcher, a training-free sampling algorithm that adapts video diffusion models for video inpainting and editing. Specifically, the authors introduce CD-EM to optimise the masked area to match the likelihood of the unmasked regions through a fixed-point EM strategy. The authors also introduce a latent mask fuser that allows them to directly mask at the latent space instead of the pixel space. The authors evaluate their training-free strategy across various models and datasets/tasks and find that their model performs on par with the state-of-the-art training-based model.", "questions": "In the order of importance (not same as the order of ease): \n\n1. Would you be releasing the code? \n2. How does the model perform under different masking conditions, specifically extreme masking conditions? \n3. How does the inference speed change with the different values of $P, N, K$?\n4. Can you provide additional details about the human evaluation performed?", "limitations": "1. Improved editing ability can lead to a potential negative societal impact. Follow-up works could explore solving EM with additional steering/alignment to prevent generating unwanted content.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n\n1. The paper is well written, and the content is easy to follow. \n2. The method appears to be theoretically sound and achieves good results.\n3. Since it’s a training-free algorithm, it would allow for easy adoption and development due to its reduced requirements.\n\n### Weaknesses:\n\n1. The added steps of Expectation-Maximisation make the approach slower compared to the baselines.\n2. Appendix D: “ZeroPathcer” vs “ZeroPatcher”\n3. The notation in eq 8, eq 9, eq 10, eq 11 with $(x_1, x_2)$ seems to be unconventional and confusing.", "was_revised": true, "final_justification": "The rebuttal has sufficiently addressed my queries with the paper. With that done, the paper is suitable for acceptance. Therfore I will retain my score as 5.", "rebuttal": "Thanks for your professional review. The considerations are very valuable in making our paper better. The typos are now corrected in our script. Our response is shown below.\n\n> W 1: Inference speed compared with baselines.\n\nCompared to the baseline DDNM, both DDNM (using 100 inference timesteps) and ZeroPatcher (50 timesteps, for the first 25 steps, use CD-EM with $N=1, K=1, P=3$) uses 100 NFEs. DDNM consumes 241.7 seconds, and ZeroPatcher 250.9 seconds for video inpainting. Our method is not using significantly more time to run compared to the training-free inpainting baseline. \n\nFor video editing, VideoComposer and VideoP2P are running on smaller resolution (VideoComposer: 16x256x256, VideoP2P: 8x512x512) compared to ours (49x720x480). If we simply use them to perform 49-frame editing clip by clip, our method is in fact not using more time than them.\n\nMeanwhile, it would be unfair to compare ZeroPatcher (250.9 seconds) with the text-to-video foundation model (78.3 seconds) on inference speed, since the foundation model do not have video inpainting and editing capability. \n\n> W 3: notation in Eqn.8, Eqn.9, Eqn.10, and Eqn.11\n\nThanks for the advice. We are pleased to improve our notation to make it easier to follow. We replace the notation $(\\boldsymbol{x}_ {t - 1} ^ a, \\boldsymbol{x}_ {t-1} ^ {b, i})$ with $\\tilde{\\boldsymbol{x}}_ {t-1} = \\\\{\\boldsymbol{x}_ {t - 1} ^ a, \\boldsymbol{x}_  {t-1} ^ {b, i}\\\\}$. We will explain the bracket means masked fusing as shown in Eqn.16. It is also consistent to our notation in Eqn.16. \n\n> Q 1: Release code.\n\nYes, we are now working on it. The project will open-source soon. \n\n> Q 2: Performance on extreme masking conditions.\n\nIt is a valueable to test our method on extreme masking conditions. We are working on the outpainting and large region inpainting test. The visual results will be added to the appendix.\n\n> Q 3: How inference speed change according to P, K, N\n\nIn Tab.1 in the appendix. We show a theoretical boundary of computation complexity given $P, K, N$. We use $T$ timesteps to sample a video among them, the first $\\tau$ steps will use CD-EM. The NFEs of ZeroPatcher can be computed using the following formula.\n$$\\mathrm{NFEs} = \\tau (PN + P(K-1)) + (T - \\tau), \\qquad P \\ge 1, N \\ge 1, K \\ge 1, \\tau \\le T$$ Below, we show the measured inference speed ($T = 50, \\tau = 25$). \n\n|P|K|N|NFEs|Time|\n|-|-|-|-|-|\n|3|1|1|100|259.9 seconds|\n|3|1|2|175|454.8 seconds|\n|3|2|1|175|457.1 seconds|\n|5|1|1|150|388.6 seconds|\n|5|1|2|275|718.3 seconds|\n|5|2|1|275|720.1 seconds|\n\n> Q 4: Human evaluation details.\n\nWe create a questionaire containing 50 questions using all video editing samples computed by all methods on DAVIS (But skip VideoP2P on background consistency, because it do not preserve background, comparing it is meaningless). Each question shows the editing results from all 5 methods along with the editing input. The question ask the respondent to answer which one achieves the best background preservation, and which one matches the editing prompt most. The editing results are labeled anonymously (By A, B, C, D, E). \nDuring the human evaluation, the participant will randomly receive 1 of the 50. The labeling and display order of the editing results are shuffled each time we show the participant the question. We ensure the participant will not receive repeated questions. the participant can answer up to 50 questions. We collect 1,310 answers from 171 different participants."}, {"review_id": "YEhtvuZUzD", "reviewer": "Reviewer_PvBK", "summary": "This paper presents a model-agnostic method for video inpainting and editing that can be integrated into existing generative video frameworks without requiring additional training. The authors propose a Latent Mask Fuser to enable accurate masking in the latent space, avoiding explicit VAE encoding/decoding. Additionally, they introduce Conditional Denoising Expectation Maximization (CD-EM), which formulates the generative process as an EM problem conditioned on the observed data. Supplementary materials demonstrate the method’s superiority over prior work across video inpainting and editing tasks.", "questions": "While the method’s increased inference cost is a concern, its architectural flexibility and empirical performance are compelling. Including additional qualitative ablation and more relevant baselines (e.g., VACE) would make the work even stronger.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n- The proposed approach is designed to be compatible with a variety of diffusion and flow matching based text-to-video generation models, improving its versatility and practical applicability.\n- The method shows clear improvements in both video inpainting and editing tasks, as demonstrated by qualitative and quantitative comparisons with prior methods.\n\nCons:\n- Due to the EM formulation, the method introduces significant inference-time overhead, as reported in the appendix. Discussion on potential optimization or trade-offs would strengthen the paper.\n- While the proposed components appear effective, visual ablation studies would be helpful in further clarifying their individual contributions.\n- It would be valuable to evaluate the method on top of WAN 2.1, and compare against VACE [a], a state-of-the-art editing model built on WAN 2.1. Such a comparison would evaluate performance under similar architectures and clarify the benefits of the training-free method.\n\n[a] VACE: All-in-One Video Creation and Editing", "was_revised": true, "final_justification": "- The proposed method presents a practical trade-off between computational cost and performance.\n\n- The use of the EM-based approach offers a promising direction that could inspire other training-free methods, especially in the era of large models.\n\n- Open-sourcing the WAN-based version would be valuable for the research community, particularly for advancing video editing and generation tasks.", "rebuttal": "> W 1: Inference speed overhead, possile trade-offs or optimization. \n\n**Inference speed overhead**\nPlease refer to our response to W 1 for Reviewer U6KC. Our method is not considerably slower than the editing and inpainting baselines. For your convenience, we copy it here\n\n> Compared to the baseline DDNM, both DDNM (using 100 inference timesteps) and ZeroPatcher (50 timesteps, for the first 25 steps, use CD-EM with $N=1, K=1, P=3$) uses 100 NFEs. DDNM consumes 241.7 seconds, and ZeroPatcher 250.9 seconds for video inpainting. Our method is not using significantly more time to run compared to the training-free inpainting baseline. \nFor video editing, VideoComposer and VideoP2P are running on smaller resolution (VideoComposer: 16x256x256, VideoP2P: 8x512x512) compared to ours (49x720x480). If we simply use them to perform 49-frame editing clip by clip, our method is in fact not using more time than them.\nMeanwhile, it would be unfair to compare ZeroPatcher (250.9 seconds) with the text-to-video foundation model (78.3 seconds) on inference speed, since the foundation model do not have video inpainting and editing capability. \n\n**Possible trade-offs or optimization**\n\nOne can choose to use smaller $P, K, N$ to achieve faster sampling. An extreme situation is when $P = 1, K = 1, N = 1$, we reduce the NFEs to 75. The inference time is reduced to 191.4 seconds (22% speed up). As Fig.6 shows, RFID will degrade to 0.102. \n\n> W 2: Visual ablation studies.\n\nDue to the text-only rebuttal regulation this year, we are sorry the visual ablation cannot be shown to you now. We are additing it to the appendix. In alternative, we provide some evidences on how each tecnical component contributes to the final result. \n\n**Contribution of CD-EM**\nWhen the model uses PF only, it is identical to DDNM. In Fig.4, we show a visual comparison between DDNM and ZeroPatcher. It is showing what the results look like without CD-EM.\n\n**Contribution of LMF**\nWhen not using LMF, the model adopts direct latent space mask fusing. Given a dynamical video mask in the pixel space. We first downsample it using the scale 4x8x8. Then, we apply masked replacing in latent space directly. Below, we show the performance comparison on LMF and the naive method. We randomly sample 50 video pairs from DAVIS, each pair use a randomly sampled video mask. The methods are used to fuse the video pairs. The ground truth is obtained by using masked replacing in pixel space. It shows LMF achieves much better fusing performance. \n\nWe cannot show visual comparison due to the full-text rebuttal rule this year. But if you look into the results, latent space mask fusing is doing okay in the centeral area of the mask. But very severe artifacts are observed in mask boundaries.\n\n|Method|PSNR $\\uparrow$|SSIM $\\uparrow$|\n|-|-|-|\n|LMF|34.49|0.9774|\n|Latent space mask fusing|31.64|0.9578|\n\n> W 3: ZeroPatcher on Wan 2.1. \n\nThis is very good advice to adopt ZeroPatcher on Wan 2.1. We are now working on it and ZeroPatcher + Wan 2.1 will be available when we open-source our project. Then, we can add a comparison between ZeroPatcher and VACE."}, {"review_id": "PN0oxKrKF7", "reviewer": "Reviewer_vt1q", "summary": "The paper presents a novel, model-agnostic, training-free video inpainting and editing method that achieves strong performance and is supported by solid theoretical analysis. However, its reliance on multiple auxiliary models and limited comparison with other training-free methods raise concerns about pipeline robustness and experimental completeness.", "questions": "Questions\n1. Could the authors provide additional comparisons with recent training-free video editing methods, such as AnyV2V or ReVideo, to better contextualize the performance of ZeroPatcher?\n2. Given the reliance on accurate captions and object descriptions (e.g., from LLaVA-NeXt or GPT-4 mini), how robust is the method when such textual inputs are noisy or imprecise? Have the authors evaluated the model's sensitivity to caption quality?\n3. More questions please refer to the weakness.", "limitations": "Yes. The authors have clearly acknowledged the limitations of their approach and briefly discussed potential negative societal impacts. Their transparency is appreciated.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper introduces ZeroPatcher, a novel training-free method for video inpainting and editing, which leverages pre-trained text-to-video foundation models to produce high-quality results without any additional training.\n2. The proposed method is model-agnostic and can be flexibly applied to a range of diffusion-based video generators, including CogVideoX and HunyuanVideo.\n3. Extensive experiments demonstrate the effectiveness of ZeroPatcher in both video inpainting and editing tasks, showing superior performance over existing training-free baselines and even outperforming some trained models on certain metrics.\n4. The paper presents a thorough theoretical analysis of the proposed Conditional Denoising Expectation Maximization (CD-EM) algorithm, including proofs of solution uniqueness and convergence under well-defined conditions.\n\nWeaknesses\n1. Although the method is described as training-free, it still relies on several auxiliary base models. For instance, the use of CogVideoX involves external components such as LLaVA-NeXt and GPT-4 mini (o1) for caption generation, which introduces potential fragility into the overall pipeline. Moreover, the latent mask fuser appears to require additional training, which somewhat compromises the training-free claim.\n2. The paper lacks comparisons with several recent training-free video editing or inpainting methods, such as AnyV2V and ReVideo, which are relevant for a comprehensive evaluation.\n3. Certain formatting issues are present throughout the manuscript, including inconsistent line spacing and layout irregularities (e.g., Line 200 and the caption of Table 2), which may affect readability.", "was_revised": true, "final_justification": "The rebuttal has addressed the main issues I raised in my initial review. No further critical concerns remain. I therefore keep my evaluation unchanged at borderline accept.", "rebuttal": "> W 1: Is our method fragile for using the captioner? Is our method training-free with LMF?\n\n**Usage of captioner**\n\nWe are giving text-to-video generators the ability for video inpainting and video editing. As the feature of a text-to-video generators, the model depends a lot on the input prompts. Luckily, the LLavaNext + GPT4-o captioning strategy works well for our experiments. We first use LLavaNext to caption the video and extract the main objective within the mask.  \n\nFor example, in the car-roundabout video, the original prompt is: \"The video captures a sequence of images showing a dark-colored Mini Cooper car in motion on a city street. The vehicle is positioned at different angles, indicating it is turning or navigating through the traffic. The backdrop includes buildings with classical architecture, parked cars, and blue parking signs. The sky is clear, suggesting fair weather conditions.\"\n\nWe use GPT4-o to image a video without the masked object to produce the inpainting prompt. \"The video shows a city street with buildings, parked cars, and a blue traffic sign. The sky is clear, suggesting fair weather conditions.\" The performance for removing the main objective from video caption is satisfying for DAVIS and Youtube-VOS. We believe during the practical use of ZeroPatcher, a real user will not be troubled by this prompt dependency.\n\n**Is our method strictly training-free**\n\nThe usage of LMF will not compromise our training-free claim. The main reasons are: \n1. LMF is our computation optimization patch for latent video diffusion models. If ZeroPatcher is applied to pixel space video diffusion models. There is no need for LMF. \n2. For latent video diffusion models. Our method can achieve the same performance at the cost of additional computation for mask fusing operation. LMF is the way to erase the additional cost.\n\nThere are three ways to achieve mask replacing opeartion for latent video diffusion models:\n+ Method 1: Directly downsample pixel space mask with the scale 4x8x8. Then, we apply the downscaled mask for direct latent space masked replacing.\n+ Method 2: First, decode the latent to pixel videos using VAE decoder. Then, apply pixel space masked replacing. Finally, encode the replaced result to latent space using VAE encoder. \n+ Method 3: Train and use LMF. \n\nTo compare these three methods. We randomly sample 50 video pairs from DAVIS, each pair use a randomly sampled video mask. The methods are used to fuse the video pairs. The ground truth is obtained by using masked replacing in pixel space. \n\n|Method|Time|PSNR $\\uparrow$|SSIM $\\uparrow$|\n|-|-|-|-|\n|1. Latent space mask fusing|0.01s|31.64|0.9578|\n|2. Pixel space mask fusing with VAE|5.67s|34.67|0.9789|\n|3. LMF|0.07s|34.49|0.9774|\n\nAs shown by the table. Method 2 achieves very good masked replacing performance. It requires no training, but will use a lot additional computation. One can replace LMF with method 2 to achieve strict training-free.\n\n> W 2: Comparison with training-free video editting method such as AnyV2V and ReVideo.\n\nAnyV2V is a training-free editing method that improves over the framework VideoP2P. ReVideo, however, is a requires the training of a ControlNet-like architecture to inject content and motion. We compare ZeroPatcher with them on DAVIS using automatic metrics for a comprehensive evaluation. The result is shown below:\n\n|Method|Trainiing-free|CLIP score $\\uparrow$|PSNR $\\uparrow$|SSIM $\\uparrow$|$E_{warp}$ $\\downarrow$|\n|-|-|-|-|-|-|\n|AnyV2V|Y|0.3396|31.07|0.9467|1.354|\n|ReVideo|N|0.3494|34.12|0.9703|1.259|\n|Ours|Y|0.3543|34.23|0.9712|1.231|\n\n> W 3: Formating problem.\n\nWe update our script for improved readability. Thanks for you advise. \n\n> Q 2: Sensitivity to caption quality.\n\nIt is very valuable to introduce a caption quality sensitive analysis in our paper. We are working on it."}, {"review_id": "ikVpkSwZ1P", "reviewer": "Reviewer_4gUU", "summary": "The authors propose a training-free video inpainting and editing method, named ZeroPatcher. The method is based on the back-projection operation of DDNM, and the authors claim that pure DDNM suffers from two key limitations: the estimation bias and not acquiring any information from previous time steps. Therefore, ZeroPatcher includes additional components such as CD-EM and LMF to reduce those side effects. The visual results reveal that ZeroPatcher outperforms recent methods, and can be extended to both flow-based and denoising-based architectures.", "questions": "1. RePaint and COPAINT use an analytical posterior + pixel-space masking that resembles ZeroPatcher’s EM + latent masking. Explain why EM is preferred.\n\n2. I would assume one of the major advantages of CD-EM over latent optimization would be converging to a unique fixed point, as Theorem 3.1 claims. However, no visuals prove that increasing K and P leaves converged results unchanged, visually. I would suggest adding inpainting outputs alongside Figure 6 to further prove your point. Otherwise, why prefer EM over latent sample optimization?\n\n3. Provide an explicit A pseudo-inverse example for a given masking matrix A and show how it is applied to $ x_0 $ via Equation 5.\n\n4. It seems that the majority of the improvements are due to BP (i.e. DDNM), and the added components improve the quantitative metrics marginally. Include a visual ablation that matches Table 3, it might be easier to observe the improvements.\n\n5. The latent spaces of such video models are well-aligned spatially with pixel domain (meaning that a pixel would be represented more or less on similar spatial regions in the latent space, across all latent channels, very unlike GANs' latent spaces). This means that resizing a pixel-space mask and applying it on the latent domain would more or less yield the desired results. However, the method indicates that performing latent masking would be better (Lines 186-187). Visually compare naive masking (pixel-space masking applied directly in latent space without LMF) with LMF.\n\n6. In Figure 4 the car-roundabout example still shows visible shadows. Does extending the mask remove these?\n\nThese questions are related to Weaknesses #2. I stand my rating as borderline reject, but I am going to increase my score 1 to 2 points if these questions are adequately answered and the novelty is justified.", "limitations": "There are no fail cases visualized as the limitations of the method. Also, see the weaknesses part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n* The method has good visual results compared to other methods, and can perform both inpainting and editing.\n* The approach can also be extended to both flow-based and denoising-based approaches.\n\nWeaknesses:\n1. The paper has typographical and mathematical notation errors, both in main paper and in appendix. Please go over them. Some suggestions to improve:\n\n* Section 3.1 is too long; please shorten it and use the freed space for an EM section.\n* Notations are confusing across the paper. For instance, clarify that $ f_{\\theta}(x_t) $ in Line 100 is Tweedie’s estimation, mapping $ x_t $ to $ x_0 $. In Equation 8, $ \\mu $ looks like the denoising network output, so $ f_{\\theta} $ and $ \\mu $ should share a common symbol.\n* In Figure 2, consider merging the two denoising-step blocks and explicitly show where latent masking occurs.\n* Line 151 and Equation 10 state that EM comes from differentiating Equation 2 with respect to $ x_t $ and setting the result to zero. Please explain this derivation more clearly.\n* Figure 3 is puzzling because the horse is masked out and replaced with unrelated content. What do we mask out and replace it with? Show the intended outputs of the VVAE decoding stage to clarify.\n\n2. Even though the method looks promising at first, CD-EM and LMF is not justified adequately. CD-EM can be replaced with latent optimization, and LMF can be replaced with pixel-space masking, without a sub-network to train. Then, the overall method becomes a marginal improvement over DDNM. The authors should resolve the related issues in Questions part.", "was_revised": true, "final_justification": "I find the rebuttal adequate, hence I increase my final score a point.\n\nHowever, I expect to see the visual results of Q3-Q4-Q5, written explanations for Q1-Q2, and the tables provided in the rebuttal in the final paper & supplementary material. I believe that these are important for the sake of completeness.", "rebuttal": "> Weaknesses 1.a: Shorten the preliminary. Show where latent masking occurs in pipeline figure.\n\nThanks for you advice. We are so pleased to update the paper to make it easier to read.\n\n> Weaknesses 1.b: Is there a confusion notation on $\\boldsymbol{f}_\\theta(\\boldsymbol{x}_t)$ and $\\boldsymbol{\\mu} (\\mathbf{x}_t, \\boldsymbol{f} _\\theta  (\\boldsymbol{x}_t))$? \n\nIn fact, there is no confusion. The notation of our paper follows the style of DDIM paper [3]. $\\boldsymbol{f}_ \\theta (\\boldsymbol{x}_ t)$ estimates $\\boldsymbol{x}_ 0 $. However, $\\boldsymbol{u}(\\boldsymbol{x}_ t , \\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t))$ is the mean of $\\boldsymbol{x}_ {t-1} $. As is introduced in line 95, it is a linear combination between $\\boldsymbol{x}_ t$ and $\\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t)$. The exact formulation is shown in Eqn.7 in DDIM paper. It is:\n\n$$\\boldsymbol{\\mu}(\\boldsymbol{x}_ t, \\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t)) = \\sqrt{\\alpha_ {t-1}} \\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t) + \\sqrt{1 - \\alpha_ {t-1} - \\sigma_t^2} \\cdot \\frac{\\boldsymbol{x}_ t - \\boldsymbol{f}_  \\theta(\\boldsymbol{x}_ t)}{\\sqrt{1 - \\alpha_ t}}$$\n \nWe find readers may be confused with the $\\boldsymbol{\\mu}_ \\theta(\\boldsymbol{x}_ t)$ in Eqn.10. As is introduced in line 98, we define $\\boldsymbol{\\mu}_ \\theta(\\boldsymbol{x}_ t) = \\boldsymbol{\\mu}(\\boldsymbol{x}_ t, \\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t))$. We will update this part to make our notation more clear.\n\n> Weaknesses 1.c: Explain how the expectation in Eqn.8 is maximized. Is Eqn.10 differentiating Eqn.8 w.r.t. $\\boldsymbol{x}_ t$?\n\nEqn.8 defines a least square problem. Since $\\|(\\boldsymbol{x}_ {t-1} ^ a, \\boldsymbol{x}_ {t-1}^ {b, i}) - \\boldsymbol{\\mu}(\\boldsymbol{x}_ t, \\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t))\\|_ 2 ^ 2 \\ge 0$, the term is minimized if and only if $(\\boldsymbol{x}_ {t-1}^a, \\boldsymbol{x}_ {t-1}^{b, i}) - \\boldsymbol{\\mu}(\\boldsymbol{x}_ t, \\boldsymbol{f}_ \\theta(\\boldsymbol{x}_ t))=0$. With this, we can already define a fixed point iteration problem. In our appendix, we use the analytical solution for least square problem. The solution is shown in Eqn.4 in the appendix. We do not compute the derivative w.r.t. $\\boldsymbol{x}_ t$ in our paper. \n\n> W 1.d: Latent mask fuser illustration is using two irrelevant videos.\n\nThis is very good advice! In practice, we use mask to fuse the model generated masked region with the known background. We will update it for better readability.\n\n> W 2.a: Why not use latent optimization? \n\nWe tried to optimize the latent throught back-prop with the objective Eqn.9. Below, we show a quantitative comparison on DAVIS video inpainting. We use an adaptive learning rate $\\eta_t = 0.02 \\sqrt{\\alpha_t}$ suggested by COPAINT. The experiment shows, fixed point iteration achieves better performance while using considerable lower cost. \n\n**Computation and memory cost** \nOne objective of our method is to run it without using more graphic memory than normally sampling videos from the model. As the text-to-video generators becomes larger and video resolution becomes higher, this problem becomes more severe. Latent optimization method requires one or multiple back-propagation of the diffusion model (with gradient checkpointing, in most cases, which leads to even more cost), this introduces significantly more computation and memory usage. As shown below. \n\n|Method|NFEs|Back-props|PSNR $\\uparrow$|SSIM $\\uparrow$|VFID $\\downarrow$|$E_{\\mathrm{warp}} \\downarrow$|Time|Memory|\n|-|-|-|-|-|-|-|-|-|\n|Latent optimization|50|25|34.03|0.9756|0.115|1.217|292.6 s|76,354 MB|\n|Latent optimization|50|50|34.11|0.9759|0.107|1.203|457.7 s|76,362 MB|\n|Latent optimization|50|75|34.13|0.9761|0.101|1.185|619.3 s|76,364 MB|\n|Fixed point iteration (ours)|100|0|34.13|0.9765|0.096|1.163|249,4 s|22,569 MB|\n\n> Weaknesses 2.b: Why not use pixel space masking? Comparison between LMF and direct masked replacing in latent space.\n\nThis is a pity we cannot show you the visual comparison due to the full-text rebuttal this year. A quantitative comparison on DAVIS is shown below. Pixel space masking is prohibitively expensive for latent video diffusion models, as it requires decoding to pixels, applying the mask, and re-encoding to latent space at every denoising iteration. Video VAE encoding/decoding alone takes ~6 seconds per iteration, making this approach impractical.\n\nOur Latent Mask Fuser (LMF) eliminates this overhead, reducing latency to 0.07s versus 5.67s for pixel-space masking while maintaining near-identical quality (PSNR: 34.49 vs. 34.67; SSIM: 0.9774 vs. 0.9789). Direct latent-space mask replacement (0.01s) causes severe boundary artifacts due to spatial-temporal compression (e.g., 4×8×8 downsampling in CogVideoX), and the video latent is somewhat delicate. It degrade PSNR to 31.64 and SSIM to 0.9578. \n\n|Method|Time|PSNR $\\uparrow$|SSIM $\\uparrow$|\n|-|-|-|-|\n|LMF|0.07s|34.49|0.9774|\n|Latent space mask fusing|0.01s|31.64|0.9578|\n|Pixel space mask fusing|5.67s|34.67|0.9789|\n> Question 1: Compared to RePaint and COPAINT theoretically\n\nIn the paper of RePaint. The key algorithm is described in Eqn.(8a), Eqn.(8b) and Eqn.(8c). This is identical to back projection method described in DDNM [1]. Its limitation is shown in line 126 in our paper. \n\nCOPAINT proposes to optimize latent $\\boldsymbol{x}_t$ by maximizing the conditional posterior in each denoising step. As Eqn.10 in COPAINT paper says, we translate the notation to the style of our paper:\n\n$$\\boldsymbol{x}_ t ^ * = \\arg \\min_{\\boldsymbol{x}_ t} \\|\\boldsymbol{x}_ 0 ^ a - \\boldsymbol{f}_ \\theta ^ a (\\boldsymbol{x}_ t)\\|_ 2 ^ 2$$ where $\\boldsymbol{f}_ \\theta ^ a (\\boldsymbol{x}_ t)$ is the unmasked part of model prediction. A foundamental difference between COPAINT objective and the EM objective is that, our objective includes the term $\\boldsymbol{x}_ t$ instead of only $\\boldsymbol{f}_ \\theta ^ a (\\boldsymbol{x}_ t)$. Only then, we can write the solution in this formation $\\boldsymbol{x}_ t ^ * = g(\\boldsymbol{x}_ t ^ *)$, and fixed point iteration can be applied.\n\nNote that we also extend the objective for conditional flow matching generative models, as shown in our appendix. Crucially, as reported by COPAINT in Fig.3, the latent optimization method requires siginicantly more time to compute than DDNM (about 5 to 20 times more depending on hyparameter choice). Please see W.2a to check how latent optimization is working in the video model. Our fixed-point iteration strategy is essential for efficient sampling.\n\n> Question 2: Visual illustration on convergency of the fixed point.\n\nWe show how CD-EM reaches a fixed point by evaluating how much $\\boldsymbol{f}_ \\theta (\\boldsymbol{x}_t)$ is different from the previous iteration. The similarity is evaluated using PSNR on 10 sampling videos. In this experiment, we use $N=1, P=1$. Note that, each row show the convergency of a certain denosing timstep. As K (iteration steps) becomes larger, the fixed point iteration will tend to not to change the content, and reaches a fixed point.\n\n|Steps (K)|t|1|2|3|5|10|15|20|30|\n|-|-|-|-|-|-|-|-|-|-|\n|PSNR $\\uparrow$|5|17.54|21.36|24.84|27.84|28.64|29.31|29.68|29.81|\n|PSNR $\\uparrow$|15|22.61|24.46|26.44|28.03|29.34|30.27|30.94|31.26|\n|PSNR $\\uparrow$|25|26.41|28.81|29.69|30.54|31.21|31.85|32.24|32.41|\n\n> Question 3: A pseudo-inverse example for Eqn.5. \n\nTo apply Eqn.5. One must flatten the image to a 1d vector (e.g, flatten using raster order). Now, I show the case where A is a 2x2 mean pooling operator on a 4x4 image. We have:\n\n$$\\mathbf{A} = \\left[\\begin{matrix}\n0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\\\\\\n0    & 0    & 0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0    & 0    & 0.25 & 0.25 & 0 & 0 & 0.25 & 0.25\n\\end{matrix}\\right].$$ Then the pseudo-inverse of it is a upsampling operator\n\n$$\\mathbf{A}^ \\dagger = \\left[\\begin{matrix}\n1 & 1 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0  \\\\\\\\\n0    & 0    & 1 & 1 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 1 & 1 & 0 & 0 \\\\\\\\\n0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0    & 0    & 1 & 1 & 0 & 0 & 1 & 1\n\\end{matrix}\\right]^ T.$$\n\n> Question 4: Visual comparison with BP. CD-EM's improvements over BP. \n\n**Visual comparison with BP**\nWe will add a visual ablation comparison to our paper. Due to the full-text rebuttal this year. We recommend looking to Fig.4 for a qualitative comparison for now (you are right only using BP is equal to DDNM). \n\n**CD-EM is the major contributor to video quality**\nIn Tab.3, the four metrics are evaluating the performance from very different perspectives. In PSNR, and SSIM, the majority improvements are brought by BP because they are evaluating whether the background region of the inpainting results looks similar to the ground truth. BP is directly replacing the known background to the generated samples. This makes it naturally the most significant contributor to PSNR and SSIM. In contrast, CD-EM is mainly working on the masked region, which is not considered by PSNR and SSIM. As shown in Tab.3, CD-EM contributed the most on VFID and $E_{\\mathrm{warp}}$, the video quality metrics. Therefore, CD-EM is in fact the major contributor to video quality.\n\n> Question 6: Shadow in the car-roundabout example. \n\nAs shown in the first row of Fig.4, the official mask is not including the car shadow. Then all examples in the comparison is not removing the shadow. We rework it by inflating the car mask by 64 pixels to get an extended mask covering the shadow. Then our method is able to render that region. We are sorry for not being able to show you the visual result due to the regulation this year. \n\n[1] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model\n[2] Video diffusion models\n[3] Denoising Diffusion Implicit Models"}]

Author rebuttal:

Meta-review: The authors introduce ZeroPatcher, a novel training-free model-agnostic sampler for video inpainting and editing using pre-trained text-to-video foundation models.

The sampler optimizes the missing/masked parts of the denoising trajectory by maximising the (marginal) log-likelihood expectation conditioned on unmasked video segments. The inner-loop marginalization is handled by applying the Expectation-Maximization (EM) problem over the denoising trajectory, involving an expectation step via Monte-Carlo sampling and a maximization step solved by fixed-point iteration, with theoretical analysis showing solution uniqueness and convergence. CD-EM conditions on the unmasked part of x_{t-1} as guidance, which is more informative than using the unmasked x_t as is done by traditional back-projection methods.
To enable inpanting and editing with latent-space video models without repeatedly converting between the latent and pixel spaces, the authors introduce a lightweight convolutional architecture called the Latent Mask Fuser. This module performs video masking directly in latent space, thereby eliminating the need for VAE decoding and encoding, which are computationally expensive.
ZeroPatcher demonstrates impressive video inpainting performance using CogVideoX and HunyuanVideo as foundation models. It outperforms the training-free inpaining baselines and is even competitive with trained inpainting models.

Strengths: 
- The proposed approach is novel, sensible, well-motivated, and clearly described.
- It performs well at both video inpainting and editing, and its training-free and model-agnostic nature makes it widely applicable and well positioned to benefit from improvements in the underlying foundation models.
- Additional results provided by the authors in response to the reviewers strengthen the paper further.

Weaknesses:
- While some reviewers pointed out that the fact that the Latent Mask Fuser (LFM) needed to be trained contradicted the training-free label of the method, the authors clarified that the LFM was used to improve efficiency of the approach by eliminating the repeated conversion between the latent and pixels spaces, and that the sampler itself is truly training-free and can be used without LFM and provided empirical evidence of this claim.
- As ZeroPatcher uses EM in its sampling loop, it is going to require more computation than vanilla sampling from the underlying foundation model. However, hyperparameters (N,P, and K) provide a way of trading off the quality of the results against the additional computation and the authors show that good results can be obtained at a relatively low additional computations cost.
- One of the reviewers had concerns about the notation, but I found nothing nothing wrong with it.
- Some of the more mathematical parts of the paper (e.g. Theorem 3.1) are harder to follow than necessary because of choppy writing. Please merge consecutive short sentences into longer sentences when they clearly belong together.

This paper makes a substantial contribution to video generation and editing, and clearly meets the bar for acceptance.
Decision: Accept (poster)