HERO: Harnessing Temporal Modeling for Diffusion-Based Video Outpainting

Authors: Li Li, Yaohua Wang, Aixi Zhang, Weiming Lu, Senzhang Wang
Primary area: generative models
Keywords: video outpainting, diffusion model

Abstract:
Video outpainting expands the spatial perspective of a video, enabling it to adapt to various display devices with different aspect ratios.
Current diffusion-based approaches for video outpainting often suffer from quality issues such as blurred details, local distortion, and temporal instability, significantly impacting the user experience.
The root cause is the insufficient temporal modeling in video
outpainting, which inadequately represents the relationships between frames over time.
To address this issue, a novel approach called HERO~(Harnessing the tEmpoRal modeling for diffusion-based Outpainting) is proposed to effectively tackles these generated video quality problems.
HERO employs two critical components to enhance temporal modeling: the Temporal Reference Module, which provides reference features that extend beyond spatial dimensions; and the Interpolation-based Motion Modelling Module, designed to stabilize generated frames.
By integrating these modules, these quality issues in video outpainting are effectively addressed.
Extensive experiments on multiple benchmarks demonstrate that HERO outperforms existing methods qualitatively and quantitatively.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730103963961

thread_type:
official_review

<thread_object>
creator: reviewer_mnkm

<content>
summary:
This paper proposes a novel approach for high-quality video outpainting. The temporal reference module provides a new perspective besides the spatial perspective, alleviating the inadequate temporal modeling problem. The proposed interpolation-based motion modeling module utilizes adjacent frame relations, enabling more stable results. Qualitative and quantitative experiments show that the proposed approach outperforms previous methods.

strengths:
1. In addition to the spatial perspective, this paper gives insight into diffusion-based video outpainting from a temporal perspective, proposing corresponding solutions to remove temporal limitations from prior studies.

2. This paper innovatively introduces optical flow to enrich temporal information and strengthen the relationship between adjacent frames. Besides, the proposed $IM^3$ improves the stability of results in a low learning cost.

weaknesses:
1. The parameters and computation for the network introduced in this paper are notable. Although its impacts on GPU memory and inference time are discussed in Lines 508-510, quantitative evaluation of parameters and computations should be demonstrated and compared with related methods.

2. The ablation results in Table 4 and Table 5 are insufficient to support that $IM^3$ is necessary for both 3D-RefNet and 3D-UNet (Line 471-472). Ablation experiments for 3D-UNet w/o $IM^3$ should be conducted.

3. Some minor errors: Check the spelling of “INTRODUTION” which leaves out the letter C (Line 047). The sentence on line 159 misses a period. There is an extra space on line 161. Check the citation format of the sentence on line 186. Missing a space in the sentence on line 782. The positions of Binary Mask and Optical Flow in Figure 3 are reversed.

questions:
1. As stated in lines 155-156, the padded video is transformed into optical flow maps and then concatenated with binary masks, while on lines 209-211, it claimed that the dense optical flow of the input video is first estimated and the unknown regions will be filled with zero. It is confusing whether the input video is padded or not when calculating optical flow. Besides, if follow the statement on lines 155-156 that the input video is padded, the padding area will affect the generation of optical flow in known regions, making an influenced input.

2. Considering that the first and last frames only have one adjacent frame rather than two, does the lack of enhancement for these frames in the $IM^3$ module potentially result in poorer restoration quality for them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730216034575

thread_type:
official_review

<thread_object>
creator: reviewer_kzw5

<content>
summary:
This paper describes a method called HERO, which solves the problems of blurred details, local distortion, and temporal instability common in existing methods by introducing two key components: a temporal reference module and an interpolation-based motion modeling module. The former provides reference features beyond the spatial dimension, stabilising the generated frames through the relationship between neighboring frames.

strengths:
HERO has shown qualitative and quantitative results in several benchmark tests that outperform existing methods.
The HERO design consists of two independent but complementary modules that can be flexibly applied to different video generation tasks.

weaknesses:
The paper points out issues with existing diffusion model-based video episodic processing, like detail blurring, local distortions, and temporal instability. However, it doesn't dive deep into what’s causing these problems. It claims inadequate temporal modeling is the main issue but doesn’t explain why current methods can’t tackle these challenges effectively.
There’s a noticeable lack of innovation in the paper. The time-referenced module and the interpolation-based motion modeling module in HERO mainly combine existing models like VAE, 3D-RefNet, 3D-UNet, and other diffusion models that are already well-known in this field.
While HERO shows promising results in various benchmarks, the experimental section doesn’t properly validate important aspects like model parameters and speed. These factors are crucial for a full evaluation of how well the model performs.

questions:
The author’s description of the model structure in section 2.2 doesn’t align with what’s shown in Figure 3. This is particularly evident in the Temporal Reference Perspective section, where it’s unclear why the results from Optical Flow Enc are handed off to the second half of 3D-UNet.
In Figure 3, it’s unclear why the Spatial Reference Perspective shows CLIP pointing to both the front and back parts of 3D-UNet. This could use some clarification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720555843

thread_type:
official_review

<thread_object>
creator: reviewer_ddae

<content>
summary:
# Summary

The paper proposes a diffusion-based video outpainting framework. The proposed framework designs a temporal reference module to better model the spatiotemporal representations and designs a interpolation-based motion modelling module for better temporal consistency. The proposed framework outperforms the selected video outpainting baselines

strengths:
# Strengths

- The proposed modules both seems reasonable and technically sound
- Extensive experiments demonstrate the effectiveness of the proposed framework

weaknesses:
# Weaknesses

- In Sec 2.1, the statements about denoising UNet seem incorrect; also DDPM is not a deterministic sampling process
- Considering there are no ground truth results for video outpainting, the usage of pixel-based metrics like MSE/PSNR/SSIM does not necessarily reflect the actual performance
- Diffusion models can generate diverse results with the same input conditions and different random seeds. The paper does not mention how results are selected are comparison. It might be better to repeat the experiments across different seeds to report mean+std and also report the success rate for each generation
- The paper does not mention if prompts were used in the evaluations
- The selected evaluation metrics do not necessarily reflect the actual visual quality and temporal consistency of generated videos and human evaluations remain the most reliable measure. User study should be used for comparisons
- We can still observe local jitting artifacts/object distortions in the outpainted regions (in supplementary videos)
- The paper does not evaluate the video outpainting on complicated scenes like occlusions, object re-appearance, fast motions. 

# Other comments (not weaknesses)

- In Fig 3, the captions for "Binary Mask" and "Optical Flow" are incorrectly positioned
- The overview (Fig 3) lacks clarity, which might need a better visualization of the shared UNet

questions:
Please refer to the weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730762112536

thread_type:
official_review

<thread_object>
creator: reviewer_c5r4

<content>
summary:
This paper presents HERO (Harnessing the tEmpoRal modeling for diffusion-based Outpainting), a novel video outpainting method that addresses quality issues such as blurred details, local distortion, and temporal instability.HERO uses two key components: the temporal reference module ( TRM) and an interpolation-based motion modeling module (IMM).TRM provides reference features beyond the spatial dimension, while IMM stabilizes the generated frames. The authors have conducted extensive experiments on several benchmarks to demonstrate that HERO outperforms existing methods both qualitatively and quantitatively.

strengths:
- Innovative Approach: HERO introduces a novel approach to video outpainting by focusing on temporal modeling, which is a significant advancement in the field.
- Temporal Reference Module (TRM): The TRM effectively provides reference features that enhance the spatial-temporal context, leading to improved outpainting quality.
- Interpolation-based Motion Modeling Module (IMM): The IMM stabilizes generated frames, reducing temporal instability and improving the overall quality of the outpainted video.
- Comprehensive Experiments: The paper includes extensive experiments on multiple benchmarks, providing strong empirical validation of the method's effectiveness.

weaknesses:
- Limited Dataset Diversity: The experiments are primarily conducted on the DAVIS and YouTube-VOS datasets, which may not fully capture the diversity of real-world video content. I am curious about the model’s performance on ultra-high-resolution videos or other datasets with large and complex motion patterns. I would like to see more comprehensive experiments. Adding qualitative comparisons on real-world videos would be a big plus—for example, using the proposed algorithm to directly outpaint videos captured on a mobile phone. Specifically, can HERO perform well for 1080p videos, or higher resolution videos (2K or 4K), which are part of our daily lives? What is the runtime and computational complexity like on these scenarios? Also how well does the HERO perform for real-life scenarios in sports? For example, basketball and soccer in sports, or even badminton and table tennis in small resolution.

- Insufficient Error Analysis: The paper lacks a detailed error analysis that could provide insights into the method's limitations and failure cases. For example, will HERO have temporal discontinuous results? What are the reasons for these failure cases?

- Computational Efficiency: The computational efficiency of the method, especially in comparison to existing methods, is not thoroughly analyzed. I suggest that the authors provide a comparison of computational complexity, including metrics such as parameter count, FLOPs, runtime, and energy consumption.

- Hyperparameter Sensitivity: The sensitivity of the model to hyperparameters is not extensively discussed, which could impact its practical usability. For instance, is the diffusion model sensitive to hyperparameter choices?

- User Studies: I suggest that the authors include user studies. I would suggest that the authors could conduct a user study by finding a group of people (e.g., the size of 20 people) and counting the scores that these people give to the results obtained by the different methods.

- Comparison to Other Methods: The comparison to other methods is limited to a few benchmarks, and a more comprehensive comparison could strengthen the paper.

questions:
Could you elaborate further on the significance of video outpainting. In my view, this area seems quite narrow, with arguably less value compared to video inpainting (minor point).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732417334961

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate all reviewers for their valuable comments and suggestions. Your insights have significantly contributed to improving the quality of the paper.
Building on this constructive feedback, we emphasize that:
-  HERO is the first work to hit the issue of insufficient temporal modeling in existing diffusion methods and explicitly harness temporal modeling to improve video outpainting performance.
- HERO reflects our innovation in extending the remarkable performance of diffusion models from image generation to the challenging domain video editing.


We have carefully addressed the comments and highlighted the key improvements as follows:
- We have compared our method with related approaches in terms of parameters and computational complexity, filling a gap in the community’s benchmarking. **This provides a foundation for future research**, with this comparison table serving as a key reference.
- We conducted a **human evaluation with 12 participants** to assess HERO and baseline methods. This makes HERO’s experimental results more robust.
- We have added the **mean and standard deviation** to the quantitative evaluation on the DAVIS dataset, enhancing the reliability of the results.
- We have added video demos of HERO in **extreme scenarios** (*e.g.*, occlusions, object re-appearance, and fast motions) to the supplementary materials (in the ‘Newly_Add’ folder) for your review, 

For each of the reviewers’ specific comments, please refer to the corresponding responses.

Minor changes have already been directly incorporated into the revised manuscript, while the newly added tables and sections will be included in the appropriate sections of the final version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper was reviewed by four experts providing initially unanimous negative ratings (<6).

The authors provided responses and three experts kept their initial ratings while one improved to "6: marginally above".

Most of the reviewers and the AC agree that the paper is technically sound, but at the same time the contributions are rather limited and despite the extra information and new results provided by the authors three out of four reviewers remained on the negative with their ratings (<6).

Given the above, the AC finds the paper below the acceptance bar an invites the authors to benefit from the received feedback and further improve their work.