VEnhancer: Generative Space-Time Enhancement for Video Generation

Authors: Jingwen He, Tianfan Xue, Dongyang Liu, Xinqi Lin, Peng Gao, Dahua Lin, Yu Qiao, Wanli Ouyang, Ziwei Liu
Primary area: generative models
Keywords: Diffusion models, Video Generation, Generative Video enhancement, video super-resolution, frame interpolation, space-time super-resolution

Abstract:
We present \emph{VEnhancer}, a generative space-time enhancement method that can improve the existing AI-generated videos spatially and temporally through one video diffusion model. Given a generated low-quality video, our approach can increase its spatial and temporal resolution simultaneously with arbitrary up-sampling space and time scales by adding more details in spatial domain and synthesize detailed motion in temporal domain. Furthermore, VEnhancer is able to remove generated spatial artifacts and temporal flickering of generated videos.  
To achieve this, basing on a pretrained generative video prior, we train a \textbf{S}pace-\textbf{T}ime Controller and inject it to the prior as a condition on low-frame-rate and low-resolution videos. To effectively train this ST-Controller, we design \textit{space-time data augmentation} to create diversified video training pairs as well as \textit{video-aware conditioning} for realizing different augmentation parameters in both spatial and temporal dimensions.
Benefiting from the above designs, VEnhancer can be end-to-end trained to enable multi-function in one single model. 
Extensive experiments show that VEnhancer
surpasses existing state-of-the-art video super-resolution and space-time super-resolution methods in enhancing AI-generated videos. Moreover, VEnhancer is able to greatly improve the performance of open-source state-of-the-art text-to-video methods on video generation benchmark, VBench.

Decision: reject
Reviewer scores: 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730033619174

thread_type:
official_review

<thread_object>
creator: reviewer_sk5p

<content>
summary:
This paper proposes a video super-resolution algorithm called VEnhancer, which enhances video resolution in both spatial and temporal dimensions. The method is based on a pretrained video generation model, with a Spatio-Temporal (ST) Controller trained specifically for the task. The video to be enhanced is provided as a condition input to the ST-Controller. The authors constructed a test dataset, AIGC2023, to validate the effectiveness of the algorithm.

strengths:
1. From the visual results provided by the authors, the proposed method enhances the video's spatial resolution and FPS. The details of objects are richer compared to the pre-enhancement version, and the smoothness of the video has also improved to some extent.
2. This method can achieve both temporal and spatial video super-resolution simultaneously.

weaknesses:
1. The proposed method's overall idea is quite similar to ControlNet: both introduce a control branch by copying the encoder of the UNet model to the control branch and initializing the output layer to zero. The difference lies in their applications—this method uses the framework for video super-resolution, while ControlNet is designed for controllable generation. In this paper, the condition is the video to be super-resolved, whereas in ControlNet, conditions include modalities like depth, edge, and mask. The authors should explicitly discuss how their approach differs from or improves upon ControlNet for the specific task of video enhancement. 
2. For the video-aware conditioning (section 4.3), the noise augmentation technique proposed by the authors is a common trick that was first introduced by *Cascaded Diffusion Models* [Jonathan Ho et al., 2021] for multi-stage high-resolution image generation. Essentially, it incrementally upscales from low to high resolution, which is quite similar to this paper's application. Additionally, using the time embedding as a condition is a standard approach in diffusion models. Conditioning on the downscale factor is somewhat analogous to using FPS as a condition in video generation [Make-A-Video, Uriel Singer et al., 2022], allowing the generation process to better incorporate additional conditioning information. This approach isn't particularly novel in itself. The authors should clarify what specific innovations or improvements their approach offers over these existing techniques. 
3. The proposed method is for video super-resolution and, theoretically, should be applicable to both AI-generated videos and real videos. However, the authors only conducted experiments on AI-generated videos (section 5.3), making this comparison less comprehensive. I suggest that the authors include experiments on real-world videos or explain why their method might not be suitable for such videos if that's the case.

questions:
1. In terms of experimental results, the proposed method shows only minor improvements in Aesthetic Quality Dynamic, Degree of Motion, and Smoothness, while achieving relatively significant improvements in MUSIQ and DOVER. What causes this discrepancy?
2. As shown in Table 3, the improvement of the proposed method on CogVideoX-5B is relatively minor and not as significant as on VideoCrafter-2. Could the authors explain the reasons for this?
3. How many videos are there in the AIGC2023 dataset? What are the approximate resolutions and lengths of these videos? Is there any statistical information available? Will the authors release this test set in the future?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730124222357

thread_type:
official_review

<thread_object>
creator: reviewer_lblj

<content>
summary:
The paper presents a diffusion-based video space-time enhancement model. Specifically, the proposed VEnhancer leverages a control branch to integrate the low-resolution and low-frame-rate video prior into diffusion. Besides, a new data augmentation strategy is proposed to carry out the video super-resolution and enhancement with different functions, e.g., enhancing video with details in different levels. Comparisons with both video super-resolution methods and cascaded super-resolution methods verify the superiority of the proposal.

strengths:
1.	The proposed approach unifies the space-time video super-resolution through re-sampling operation over time-space axes, achieving multi-function in a single video diffusion model.
2.	Both of the objective video quality evaluation and subjective human evaluation show the effectiveness of the proposed approach in video enhancement. 
3.	The paper is well-written and technical description is clear.

weaknesses:
1.	One of my key concerns about the technical novelty is about the involving of the UNet-based control branch whose weights are copied from the original I2VGen-XL. The novelty seems limited since the controling approach has been proposed by ControlNet and the key frame information integration via feature summation is also intuitive. 
2.	The technical design of space-time data augmentation should be investigated quantitatively in the main paper. Besides, the motivation of the position embedding encoded from the noise augmentation is unclear. What are the effects by exploiting this parameter $\sigma$ in video enhancement? 
3.	There should be more in-depth analysis or discussions with other SOTA approaches to demonstrate the technical contributions of the proposal. Only describing the visual results cannot give readers many insights, but only leads them think that the proposed approach is more like engineering rather than research.  
4.	The ablation study should be conducted in the main paper instead of the appendix. 
5.	Some format issue and typos: a)	The "ramdom" in line 259; b)	Remove period symbol in the section title “Conclusion and Limitation.”

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730408465222

thread_type:
official_review

<thread_object>
creator: reviewer_vxkg

<content>
summary:
The paper introduces VEnhancer, a novel generative method designed to enhance both the spatial and temporal quality of AI-generated videos. VEnhancer increases resolution, adds detail, and reduces artifacts and flickering through a single model, leveraging a space-time controller for conditioning on low-quality inputs.
### Contributions

1. **Unified Model:** VEnhancer combines spatial and temporal super-resolution and video refinement capabilities in one model.
2. **Space-Time Controller:** Introduces a mechanism for injecting multi-frame conditions based on a pre-trained generative video prior.
3. **Performance:** Demonstrates superiority over some of the existing state-of-the-art methods in video super-resolution through extensive experiments.

strengths:
- Handles multiple up-scaling in both space and time dimensions, allowing for versatile application scenarios.
- Improves the fidelity of generated videos while maintaining or enhancing detail, demonstrated through rigorous testing against current top methods.

weaknesses:
- As with most diffusion models, the complexity of the model could lead to longer inference times, which may limit its applicability in real-time or low-resource scenarios.
- The model's performance heavily relies on the availability of high-quality training data, which might not always be available or feasible to collect in certain domains.
- For each different text-to-video model, a new VEnhancer need to be trained to accommodated the different architecture, limiting the use case of the proposed method. It would be more practical if the proposed method could be a standalone video super-resolution toolkit.
- Other diffusion based video super-resolution works is not compared in the paper, e.g. Upscale-a-video.

questions:
- It would be helpful if the authors could discuss more about how they collect the data, i.e., filtering out 350K data from Panda 70M.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730532146965

thread_type:
official_review

<thread_object>
creator: reviewer_dkly

<content>
summary:
The paper titled "VEnhancer: Generative Space-Time Enhancement for Video Generation" introduces a novel method for enhancing AI-generated videos both spatially and temporally using a single model. VEnhancer is capable of up-sampling the resolution and frame rate of low-quality videos with many scales, while adding spatial details and synthesizing detailed motion. It also removes artifacts and flickering, improving upon existing methods by integrating a Space-Time Controller (ST-Controller) trained with space-time data augmentation and video-aware conditioning. The method aims to be end-to-end trainable, enabling multi-function enhancement within one model, and claims to surpass current state-of-the-art in video super-resolution and space-time super-resolution for AI-generated content.

strengths:
1. Novelty: The paper presents a unified approach for generative spatial and temporal super-resolution, which is novel in the field of video generation. The integration of a pretrained generative video prior with a ST-Controller for conditioning is a creative solution that addresses the limitations of cascaded models. The concept of space-time data augmentation and video-aware conditioning is innovative and contributes to the training of the ST-Controller in an end-to-end manner.

2. Quality: The paper is well-structured, with a comprehensive presentation of the methodology, experiments, and results. The visual results and quantitative metrics provided are convincing and demonstrate the effectiveness of VEnhancer.

3. Significance: The work is relatively significant since it's an important complementary to current open-sourced video diffusion models.The ability to handle many up-sampling scales and to refine videos while maintaining content fidelity is a substantial contribution to the field.

weaknesses:
1. Some expressions of the paper are not clear and rigorous enough, and there are certain ambiguities, ambiguities or even errors. Below are the instances:
- Wrong notations, instead of 𝐼^(1:𝑚:𝑓), 𝐼^(1:𝑚:𝑓) is typically used to denote a sequence starting from 1, ending at 𝑓, with a step size of 𝑚. Similar cases for z, t, \sigma, and s. Besides, in Fig.3 z^{1:m}_t, t^{1:m} should be z^{1:f}_t, t^{1:f} 
- Wrong illustrations. In Fig.2 Space-Time Data Augmentation part, both the noised videos (with noise strength t and \sigma) should be in latent space rather than still being natural videos. The noises are directly added to the latent videos.

2.  Some critical details are lacking such as how to do inference specifically like the initialization of z_t and z_{s, \sigma} (it should be more clear if you have a pseudocode algorithm), and the inference time and performance (like Vbench Results or other metrics) across different SSR and TSR scales.

questions:
Are the shapes of inputs to ST-Controller different because of different interval m? How does the model handle inputs with different m?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received mixed scores: two borderline accept, one borderline reject, and one reject. Most reviewers acknowledged the simple and effective solution proposed in the paper. However, some reviewers raised concerns, including the high similarity to existing methods like ControlNet, limited original contributions (as many components for video-aware conditioning were borrowed from previous works), and the limited applicability of the proposed methods. The AC has reviewed the paper, the reviews, and the rebuttal. While super-resolution for AI-generated content is an intriguing and promising idea, the current version lacks a clear, significant contribution that could be recognized as a major advancement. Although two reviewers gave acceptance recommendations, their arguments lacked specific reasons strong enough to overturn the rejection opinions. Therefore, AC agrees that the current version is not ready for acceptance. Nevertheless, the AC strongly encourages the authors to address all the reviewers' concerns and resubmit to a future venue.