Semantically Consistent Video Inpainting with Conditional Diffusion Models

Authors: Dylan Green, William Harvey, Saeid Naderiparizi, Matthew Niedoba, Yunpeng Liu, Xiaoxuan Liang, Jonathan Wilder Lavington, Ke Zhang, Vasileios Lioutas, Setareh Dabiri, Adam Scibior, Berend Zwartsenberg, Frank Wood
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: diffusion models, video inpainting, conditional generative modeling

Abstract:
Current state-of-the-art methods for video inpainting typically rely on optical flow or attention-based approaches to inpaint masked regions by propagating visual information across frames. While such approaches have led to significant progress on standard benchmarks, they struggle with tasks that require the synthesis of novel content that is not present in other frames. In this paper, we reframe video inpainting as a conditional generative modeling problem and present a framework for solving such problems with conditional video diffusion models. We introduce inpainting-specific sampling schemes which capture crucial long-range dependencies in the context, and devise a novel method for conditioning on the known pixels in incomplete frames. We highlight the advantages of using a generative approach for this task, showing that our method is capable of generating diverse, high-quality inpaintings and synthesizing new content that is spatially, temporally, and semantically consistent with the provided context.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730062979450

thread_type:
official_review

<thread_object>
creator: reviewer_i15w

<content>
summary:
This paper introduces a novel video inpainting framework based on conditional diffusion models, which can generate semantically consistent content that remains realistic over extended occlusions and diverse scene complexities. The authors present sampling schemes tailored for inpainting that capture essential long-range dependencies within the context and develop a new method to condition on known pixels in incomplete frames. Results show that the proposed method consistently outperforms methods based on optical flow or attention.

strengths:
- The motivation is clear and the techniques sound reasonable.
- The writing and organization are easy to follow.
- The experiment result validates the effectiveness of this approach, especially for complex scenarios where occlusions (input information is scarce) prevent conventional methods from performing well.

weaknesses:
1. The author needs to highlight this paper’s innovative contributions in the INTRODUCTION part and present a completed method graph rather than just showing model inputs.
2. This work relies heavily on the inpainting model to handle object occlusions and interactions. However, the authors do not address how the model performs in highly cluttered or dense environments with multiple possible object placements.
3. Additional experiments are needed to demonstrate the effectiveness of the sampling schemes and the model's generalization to non-traffic datasets, along with further comparisons/discussions with recent works, such as [1] [FFF-VDI](https://arxiv.org/pdf/2408.11402) [2] [FGT++](https://arxiv.org/pdf/2301.10048) [3] [SViT](https://openaccess.thecvf.com/content/ICCV2023/papers/Lee_Semantic-Aware_Dynamic_Parameter_for_Video_Inpainting_Transformer_ICCV_2023_paper.pdf).

questions:
- Could you also provide quantitative comparisons of your method on the YouTube-VOS and DAVIS datasets? Additional results for object removal tasks would be more convincing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730090336587

thread_type:
official_review

<thread_object>
creator: reviewer_ca5i

<content>
summary:
State-of-the-art video inpainting methods focus on propagating visual information but struggle with generating novel content. This paper redefines video inpainting as a conditional generative task using video diffusion models, enabling synthesis of inpaintings with long-range temporal consistency. The approach leverages inpainting-specific sampling schemes to handle incomplete frames and is effective even when objects are partially visible or absent in the initial context.

strengths:
1. The authors invest significant effort in building a large-scale video dataset to train the diffusion model.
2. Compared to previous video inpainting methods, the proposed model emphasizes filling in unseen content rather than relying solely on propagation.

weaknesses:
1. Practicality concerns: The proposed dataset is heavily focused on scenarios involving cars, which restricts its generalizability to broader real-world applications.

2. Lack of control over inpainting content: The proposed model does not offer mechanisms to specify what should fill the missing regions.

3. Limited diversity in demonstrations: All examples shown in the paper involve cars in the inpainted regions. Are there demonstrations with objects other than cars?

4. Comparison with recent methods: Some recent mask-guided video editing techniques can also fill gaps with cars, offering options to specify the car model and color via text prompts. Why is there no comparison with these methods?

5. The claim of “strong experimental results” is potentially biased. The authors only test their model on BDD-Inpainting and Traffic-Scenes (datasets used for model training), while baselines are only trained on Youtube-VOS and DAVIS. This discrepancy in training data limits fair comparison.

questions:
Inference time: How long does the proposed model take to process a typical 2-second video clip?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730142001323

thread_type:
official_review

<thread_object>
creator: reviewer_jgsq

<content>
summary:
The authors extend the idea of Flexible Diffusion Model (FDM) to the video inpainting domain. A different sampling and training scheme is also developed to accomodate the new task.

strengths:
- The paper is easy to follow
- The idea is intuitive and works also practically well
- The authors showcased that their model is better than flow- and attention-based video inpainting methods both qualitatively and quantitatively
- Different sampling schemes are also evaluated extensively.

weaknesses:
Given that the authors proposed a variety of inference techniques, it would be interesting to investigate whether different sampling schemes during **training** makes a difference. It seems from Equation 3 that the $\mathcal{X}$ and $\mathcal{Y}$ are randomly sampled; would a benefit be gained e.g., when training the model in a similar way to how you sample at inference?

questions:
Is the model trained from scratch? How well does it generalize to other, more complicated open video domains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708990844

thread_type:
official_review

<thread_object>
creator: reviewer_uk6y

<content>
summary:
This paper presents a conditional generative framework for video inpainting that leverages diffusion models to synthesize new, semantically coherent content for large occlusions. Unlike traditional methods dependent on optical flow, this approach achieves realistic object behavior and maintains temporal consistency even when entire content regions must be generated. Evaluated on newly constructed datasets, the framework demonstrates superior quality and consistency in inpainted sequences compared to existing methods.

strengths:
1. The method effectively captures temporal and spatial dependencies, enabling high-quality synthesis for occluded scenes and objects.
2. Quantitative results show clear improvements over state-of-the-art methods, especially in challenging video inpainting scenarios.

weaknesses:
1. Please provide inference costs (e.g., time and peak GPU memory usage) for a specified input size and number of frames, and compare these with other SOTA methods.

2. A dedicated ablation study comparing the method trained on standard datasets used by other methods versus the newly introduced datasets is suggested. This would help isolate the effects of the method itself from those of the datasets, as data alone can often address significant aspects of the problem.

3. The framework lacks clear illustration, making it difficult to grasp the main concepts. For instance, it is unclear how conditional inputs are processed in the forward pass, the roles of the trainable and frozen modules, or if any new layers for controllable generation are introduced.

4. Please also discuss the potential limitations or failure cases of the proposed method. Beyond inpainting cars, can this method inpaint other objects in a zero-shot manner?

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732316029096

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We thank all of the reviewers for their thoughtful comments and valuable feedback. We will soon post a revised manuscript addressing many of the points raised here, and believe our paper will be much stronger as a result. In particular, the following changes will be made:

- Adding run time and peak GPU memory usage for our method and those we compare to, as requested by reviewer Uk6y.
- Adding a diagram illustrating our method’s inpainting process, as requested by reviewers Uk6y and i15w, along with detailed training and sampling algorithms to remove any ambiguity about our approach.
- An updated introduction which itemizes the core contributions of our work, as suggested by reviewer i15w.

If we missed anything that reviewers feel should be changed in the paper, or if any questions are not addressed satisfactorily, please send us a reply and we will be happy to make further changes or expand upon our answers. 

## Regarding the standard YouTube-VOS and DAVIS benchmarks

Three out of four reviewers asked about the omission of these data sets from our paper, and so we felt it best to address this in a general comment and address the finer points of these questions in our responses to individual reviewers. 

As video generative models attempt to model the distribution from which their training data was generated, they require large scale datasets which provide reasonable coverage of this distribution in order to produce visually compelling results. We note that both the number of videos in the dataset as well as the number of frames per video contribute to the meaning of “large scale” here, as such models typically subsample frames due to memory constraints. Further, as the content of the videos becomes more semantically diverse, we expect that more training examples are needed in order to cover the support of the distribution. Even on restricted and relatively simple domains, common training datasets for video generation such as [UCF-101](https://arxiv.org/abs/1212.0402), [BAIR Robot Pushing](https://arxiv.org/abs/1710.05268) and [Kinetics-600](https://arxiv.org/abs/1808.01340v1) contain at minimum tens of thousands of examples, while datasets used to train models on open domains such as [WebVid-10M](https://arxiv.org/abs/2104.00650) and [HD-VILA-10M](https://arxiv.org/abs/2111.10337) typically contain millions of videos and require extensive computational resources to train on. As such, the YouTube-VOS training set (note that DAVIS has no training set and is used only as a test set for models trained on YouTube-VOS) is problematic for training generative models, as the videos are semantically diverse and the scale of the dataset is small relative to datasets typically used to train video generative models (3471 videos containing approximately 200 frames each). As a result, we made the decision early on to curate our own large scale datasets on restricted domains, which we hope will enable more research groups working in resource constrained settings to be able to pursue research making use of video generative models in the video inpainting context.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732778722270

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Posted

comment:
We thank all of the reviewers for engaging in the discussion period. This is just a quick note to announce that we have now posted the revised manuscript with the previously mentioned changes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This paper introduces a conditional diffusion model for video inpainting over a long temporal context. Unlike traditional approaches, it avoids reliance on optical flow and presents an inpainting-specific sampling scheme.
Strengths: The proposed approach is intuitive and clearly motivated. It demonstrates advantages over flow- and attention-based methods, as shown through both qualitative and quantitative evaluations. 
Weaknesses: As noted by three reviewers, the paper lacks key experimental results and ablations. In particular, there is no ablation study comparing performance on standard datasets versus the newly introduced ones, which would help isolate the method’s contributions from the dataset’s impact. Furthermore, the methodology’s scope is primarily focused on traffic-related scenes, limiting its broader applicability. Reviewer i15w also highlighted the absence of ablations on the sampling scheme's generalization to non-traffic datasets, an issue that remains unaddressed by the authors. The rebuttal has not sufficiently alleviated these concerns to justify overriding the reviewers’ assessments.