VEnhancer: Generative Space-Time Enhancement for Video Generation

## Abstract

Abstract We present VEnhancer , a generative space-time enhancement framework that improves the existing text-to-video results by adding more details in spatial domain and synthetic detailed motion in temporal domain. Given a generated low-quality video, our approach can increase its spatial and temporal resolution simultaneously with arbitrary up-sampling space and time scales through a unified video diffusion model. Furthermore, VEnhancer effectively removes generated spatial artifacts and temporal flickering of generated videos.
To achieve this, basing on a pretrained video diffusion model, we train a video ControlNet and inject it to the diffusion model as a condition on low frame-rate and low-resolution videos. To effectively train this video ControlNet, we design space-time data augmentation as well as video-aware conditioning .
Benefiting from the above designs, VEnhancer yields to be stable during training and shares an elegant end-to-end training manner.
Extensive experiments show that VEnhancer
surpasses existing state-of-the-art video super-resolution and space-time super-resolution methods in enhancing AI-generated videos. Moreover, with VEnhancer, exisiting open-source state-of-the-art text-to-video method, VideoCrafter-2, reaches the top one in video generation benchmark – VBench.

## 1 Introduction

With the advances of text-to-image (T2I) generation [, , , ] and large-scale text-video paired datasets [], there has been a surge of progress in the field of text-to-video (T2V) generative models [, , , , , , , , , ]. These developments enable users to generate compelling videos through textual descriptions of the desired content.
One common solution [, , , , ] to obtain high-quality videos is to adopt cascaded pipelines, which stacks several video diffusion models (DM), including T2V, temporal super-resolution (T-SR) and spatial super-resolution (S-SR) DMs. Such a pipeline significantly reduces computation cost when generating high-resolution and high-frame-rate videos. However, it poses several issues. First, it might be redundant and time-consuming to enhance videos in spatial and temporal axes separately with different models, as spatial and temporal super-resolution are strongly correlated tasks.
Second, the proposed diffusion-based T-SR/S-SR in [, , , , ] could only handle fixed interpolation ratio (i.e., predicting 3 frames between two consecutive frames) or fixed upscaling factors (i.e., $4\times$), thus the flexibility and functionality are limited.
Third, training T-SR/S-SR using synthesized video pairs might result in inferior generalization ability as it could only generate low-level details without fundamentally understanding the semantics and structures of video contents. Therefore, such models only improve the spatial or temporal resolution with high fidelity, but struggle in modifying (i.e., generative enhancing) the original generated videos, such as eliminating video artifacts and flickering.

Another common approach mainly focuses on removing video artifacts and refining distorted content of generated videos.
One example is I2VGEN-XL [], which follows the idea of SDXL [] and uses a refinement model to remove visual artifacts and regenerate video content through a noising-denoising process [].
I2VGEN-XL trains a diffusion model for refinement with large-scale high-quality video datasets captioned by short texts, leading to powerful regeneration ability.
Although this method has shown incredible performance in improving the stability of generated videos, it cannot increase the spatial and temporal resolution. More importantly, naive noising-denoising process will change the original video content significantly (i.e., sacrificing fidelity), which cannot always be acceptable in real-world applications.

To conclude, generative video enhancement methods have several limitations. First, cascaded temporal and spatial super-resolution diffusion models requires independent training, but their corresponding training datasets are basically the same (high-quality video datasets).
Thus, such design is both sub-optimal and inefficient during the inference.
Second, only fixed upscaling factors are supported for both spatial and temporal super-resolution, which limits their practicality.
Third, it struggles in obtaining a good balance between quality and fidelity for
generative enhancing (i.e., removing artifacts or flickering).

To this end, we propose *VEnhancer*, a unified generative space-time enhancement framework which supports both spatial and temporal super-resolution with flexible space and time up-sampling scales.
Additionally, it also remove visual artifacts and video flickering without severe drop in fidelity. To achieve this, we adopt a pretrained video diffusion model(^†^†https://modelscope.cn/models/iic/Video-to-Video/files) as the fixed generative video prior, supplying the generative ability for video enhancement.
To condition the generation on low frame-rate and low-resolution videos, we follow ControlNet [] and copy the weights of multi-frame encoder and middle block from the generative video prior as the trainable condition network.

Furthermore, to handle different up-sampling scales and reduce artifacts and flickering, we propose space-time data augmentation to construct the training data.
In particular, at traininig stage, we randomly sample different downsampling factors (ranging from $1\times$ to $8\times$), different number skipped frames (ranging from 0 0 to $7$), as well as different noise levels in noise augmentation [, ].
To ensure condition network be aware of the associated data augmentation applied to each input video, we also propose video-aware conditioning that can realize different conditions across frames. To inject this condition, for key frame, the condition network takes additional input, including the multi-frame condition latents, the embeddings of the associated downscaling factor $s$ and noise level $\sigma$ by noise augmentation.

Equipped with the above designs, VEnhancer yields stable and efficient end-to-end training. Extensive experiments have demonstrated VEnhancer’s ability in enhancing generated videos (see visual results in Figure 1, 6). Besides,
it outperforms state-of-the-art real-world and generative video super-resolution methods for spatial super-resolution only. Regarding space-time super-resolution for generated videos, VEnhancer also surpasses state-of-the-art methods as well as two-stage approach (frame interpolation + video super-resolution).
Moreover, on the public video generation benchmark, VBench [], combined with VideoCrafter-2, our approach reaches the first place.

Our contributions can be summarized as below:

- 1.
We propose VEnhancer, an efficient approach for generative space-time super-resolution and refinement in a unified diffusion model for the first time. Inspired by ControlNet, we devise a video ControlNet for multi-frame condition injection. Besides, effective data augmentation and conditioning schemes are proposed to assist the end-to-end model training.
- 2.
VEnhancer is flexible to adapt to different upsampling factors for either spatial or temporal super-resolution, exceeding the limits of existing diffusion-based spatial or temporal super-resolution methods. Besides, it provides flexible control to modify the refinement strength for handling diversified video artifacts.
- 3.
VEnhancer surpasses exisitng state-of-the-art video super-resolution methods and space-time super-resolution methods in enhancing generated videos.
With VEnhancer, existing text-to-video method VideoCrafter-2 [] achieves the top one in VBench [] in both semantic and quality, outperforming professional video generation products, Gen-2 (^†^†https://runwayml.com/ai-tools/gen-2-text-to-video/) and Pika (^†^†https://pika.art/).

## 2 Related Work

### 2.1 Video Generation

Recently, there have been substantial efforts in training large-scale T2V [, , , , , , , ] models on large
scale datasets.
Some works [, , ] inflate a pre-trained text-to-image (T2I) model by inserting temporal layers and fine-tuning them or all parameters on video data, or adopts a joint image-video training strategy.
In order to achieve high-quality video generation, [, , ] adopts multi-stage pipelines. In particular, cascaded video diffusion models are designed: One T2V base model that is followed by one or more frame interpolation and video super-resolution models. VideoLDM [], LaVie [], and Upscale-A-Video [] all develop the video super-resolution model based on $4\times$ sd (StableDiffusion []) upscaler, which has an additional downsampled image for conditioning the generation.
One drawback of this base model is losing quite a lot generative ability compared with T2I base models.
On the contrary, I2VGEN-XL follows SDXL [] and uses noising-denoising process [] to refine the generated artifacts. However, this strategy could improve stability but cannot increase the space-time resolution. VEnhancer is based on a generative video prior, and could address temporal/spatial super-resolution and refinement in a unified model.

### 2.2 Video Enhancement

Video Super-Resolution. Video Super-Resolution (VSR) is proposed to enhance video quality by upsampling low-resolution (LR) frames into high-resolution (HR) ones. Traditional VSR approaches[, , , , , , , , , ] often rely on fixed degradation models to synthesize training data pairs, which leads to a noticeable performance drop in real-world scenarios. To bridge this gap, recent advances[, ] in VSR have embraced more diversified degradation models to better simulate real-world low-resolution videos. To achieve photo-realistic reconstruction, Upscale-A-Video[] integrates diffusion prior to produce detailed textures, upgrading VSR performance into next level. Space-Time Super-Resolution. Space Time Video Super-Resolution (STVSR) aims to simultaneously increase the resolutions of video frames in both spatial and temporal dimensions. Deep-learning based approaches[, , , ] have achieved remarkable results on STVSR. STARNet[] increases the spatial resolution and frame rate by leveraging the mutual information between space and time. FISR[] propose a joint framework with a multi-scale temporal loss to upscale the spatial-temporal resolution of videos. [] proposes a one-stage STVSR framework, which incorporates different sub-modules for LR frame features interpolation, temporal information aggregation and HR reconstruction. VideoINR[] utilize the continuous video representation to achieve STVSR at arbitrary spatial resolution and frame rate. Although these methods obtain smooth and high-resolution output videos, but they fail in generating realistic texture details.

## 3 Methodology

### 3.1 Preliminaries: Video Diffusion Models

Our method is built on a video diffusion model  [], which is developed based on one of the latest text-to-image diffusion models, Stable Diffusion 2.1(^†^†https://huggingface.co/stabilityai/stable-diffusion-2-1) []. Given an video $\textbf{x}\in\mathbb{R}^{F\times H\times W\times 3}$, the encoder $\mathcal{E}$ first encodes it into latent representation $\textbf{z}=\mathcal{E}(\textbf{x})$ frame-by-frame, where $\textbf{z}\in\mathbb{R}^{F\times H^{\prime}\times W^{\prime}\times C}$. Then, the forward diffusion and reverse denoising are conducted in the latent space. In the forward process, the noise is gradually added to the latent vector z in total $T$ steps. And for each time-step $t$, the diffusion process is formulated as follows:

$$ $\textbf{z}_{t}=\alpha_{t}\textbf{z}+\sigma_{t}\bm{\epsilon},$ (1) $$

where $\bm{\epsilon}\in\mathcal{N}(\textbf{0},\textbf{I})$,
and $\alpha_{t}$, $\sigma_{t}$ specify the noise schedule in which the corresponding log signal-to-noise-ratio ($\log[\alpha_{t}^{2}/\sigma_{t}^{2}]$) decreases monotonically with $t$. And at time-step $T$, $q(\textbf{z}_{T})=\mathcal{N}(\textbf{0},\textbf{I})$.
As for backward pass, a diffusion model is used for iteratively denoising under the guidance of the text prompt $c_{text}$.
By adopting v-prediction parameterization [], the U-Net denoiser $f_{\theta}$ learns to make predictions of $\textbf{v}_{t}\equiv\alpha_{t}\bm{\epsilon}-\sigma_{t}\textbf{z}$. The optimization objective is simply formulated as:

$$ $\mathcal{L}_{LDM}=\mathbb{E}_{\textbf{z},c_{text},\bm{\epsilon}\sim\mathcal{N} (\textbf{0},\textbf{I}),t}\Big{[}\|\textbf{v}-f_{\theta}(\textbf{z}_{t},t,c_{ text})\|_{2}^{2}\Big{]}.$ (2) $$

At the end, the generated videos are obtained through the VAE decoder: $\hat{\textbf{x}}=\mathcal{D}(\textbf{z})$.

### 3.2 Architecture Design

The architecture is built upon a video diffusion model.
This video diffusion model is able
to generate temporal-coherent content and high-quality texture details during iterative denoising. To upsample and refine a low-frame-rate and low-resolution videos in both spatial and temporal dimensions, the visual information should be incorporated into the video diffusion model carefully in order to obtain high-quality results without sacrificing fidelity significantly. High-quality generated videos stem from powerful generative models, while fidelity requires the algorithm to preserve the visual information in the input. Balancing the visual quality and fidelity is always challenging in generative model research.
In this work, we follow ControlNet [] to keep the pretrained video diffusion model untouched for preserving generative capability, but create a trainable copied network for effective condition injection. The architecture is illustrated in Figure 2. We will elaborate on our design carefully in subsequent paragraphs.

The pretrained video diffusion model follows the design of stacking a sequence of interleaved spatial and temporal layers within the 3D-UNet [, ] architecture (blue blocks in Figure 2). Specifically, each spatial convolution layer (or attention layer) is followed by a temporal convolution layer (or attention layer). The spatial layers are the same as those in Stable Diffusion 2.1, including ResBlocks [], self-attention [] layers, and cross-attention layers.
The temporal convolution and attention layers are incorporated with their output layers initialized to zero and finetuned with video datasets. Specifically, the temporal convolution is one-dimensional convolution layer with a kernel size of 3, and the temporal attention is one-dimensional attention layer [].
In this 3D-UNet, the video features that aligned by temporal layers in encoder will be skipped to the decoder, in which concatenation operation will be performed to combine skipped features with decoder features.

Figure: Figure 2: The architecture of VEnhancer. It follows ControlNet [] and copies the architecures and weights of multi-frame encoder and middle block of a pretrained video diffusion model to build a trainable condition network. This “video ControlNet" accepts low-resolution key frames as well as full frames of noisy latents as inputs. Also, the noise level $\sigma$ regarding noise augmentation and downscaling factor $s$ serve as additional network conditioning apart from timestep $t$ and prompt $c_{text}$.
Refer to caption: x2.png

To build the condition network, we make a copy (both the architectures and weights) of the multi-frame encoder and the middle block in 3D-UNet (orange blocks in Fig. 2). This condition network also takes full frames of noisy latents as input, and outputs multi-scale temporal-coherent video features. These features will be injected into the original 3D-UNet through newly added zero convolutions (yellow blocks in Fig. 2). The output features of the middle block in condition network will be added back to the features of the middle block in 3D-UNet. While for output features of encoder blocks in condition network, their features will be added to the skipped video features in 3D-UNet, which are also produced by encoder blocks. Such architecture design is consistent with the original ControlNet. The main difference is that video ControlNet has temporal convolution or attention layer positioned after spatial convolution or attention layer, and the multi-frame control signals are aligned by temporal layers.

### 3.3 Space-Time Data Augmentation

In this section, we discuss about how to achieve unified space-time super-resolution with arbitrary up-sampling space and time scales, as well as refinement with varying degrees. To this end, we propose a novel data augmentation strategy for both space and time axes. Details are discussed below.

Time axis. Given a sequence of high-frame-rate and high-resolution video frames $I^{1:f}=[I^{1},I^{2},...,I^{f}]$ with frame length $f$, we use a sliding window across time axis to select frames. The frame sliding window size $m$ is randomly sampled from a predefined set, ranging from $1$ to $8$. This corresponds to time scales from $1\times$ to $8\times$. Note that $1\times$ time scale requires no frame interpolation, thus the multi-task problem downgrades to video super-resolution. After the frame skipping, we obtain a sequence of key frames $I^{1:f:m}=[I^{1},I^{1+m},I^{1+2\times m},...,I^{f}]$.

Space axis. Then, we perform spatial downsampling for these obtained key frames. Specifically, the downscaling factor $s$ is ramdomly sampled from $[1,8]$, which represents $1\times\sim 8\times$ space super-resolution. When $s=1$, there is no need to perform spatial super-resolution.
All frames in one sequence are downsampled with the same downscaling factor $s$. Thus, we arrive at low-frame-rate and low-resolution video frames: $I_{\downarrow s}^{1:f:m}$. In practice, we should upsample them back to the original spatial sizes by bilinear interpolation before being passed to the networks, so we obtain $I_{\downarrow s,\uparrow s}^{1:f:m}$.
Note that each space or time scale corresponds to different difficulty level, and thus the sampling is not uniform. Particularly, we set sampling probabilities of scales $4\times$ and $8\times$ based on a ratio of $1:2$, which is determined by their associated scale values.

Then, we use the encoder part of a pretrained variational autoencoder (VAE) $\mathcal{E}$ to project the input sequence to the latent space frame-wisely:

$$ $z_{s}^{1:f:m}=[\mathcal{E}(I_{\downarrow s,\uparrow s}^{1}),\mathcal{E}(I_{ \downarrow s,\uparrow s}^{1+m}),\mathcal{E}(I_{\downarrow s,\uparrow s}^{1+2 \times m}),...,\mathcal{E}(I_{\downarrow s,\uparrow s}^{f})].$ (3) $$

Noise augmentation in latent space. At this stage, we conduct noise augmentation to noise the latent condition information in varying degrees in order to achieve controllable refinement. This noise augmentation process is the same as the diffusion process (1) used in the video diffusion model. Specifically, the condition latent sequence is corrupted by:

$$ $z_{s,t^{\prime}}^{1:f:m}=\alpha_{t^{\prime}}z_{s}^{1:f:m}+\sigma_{t^{\prime}} \epsilon^{1:f:m},$ (4) $$

where $\alpha_{t^{\prime}}$, $\sigma_{t^{\prime}}$ determine the signal-to-noise-ratio at time-step $t^{\prime}$, and $t^{\prime}\in\{1,...,T^{\prime}\}$. Note that the pretrained video diffusion model adopts 1,000 steps ($T=1000$ in Eq. (1)). While the noise augmentation only needs to corrupt the low-level information, $T^{\prime}$ is set to $300$ empirically.
For more intuitive denotation, we use $\sigma$ instead of $t^{\prime}$. Finally, we arrive at $z_{s,\sigma}^{1:f:m}=\mathcal{E}(I_{\downarrow s,\uparrow s})_{\sigma}^{1:f:m}$.

The whole process of space-time data augmentation is summarized as follows:

$$ $I^{1:f}\rightarrow I^{1:f:m}\rightarrow I_{\downarrow s}^{1:f:m}\rightarrow I_ {\downarrow s,\uparrow s}^{1:f:m}\rightarrow\mathcal{E}(I_{\downarrow s, \uparrow s})^{1:f:m}\rightarrow\mathcal{E}(I_{\downarrow s,\uparrow s})_{ \sigma}^{1:f:m}.$ (5) $$

### 3.4 Video-Aware Conditioning

Figure: Figure 3: Video-aware conditioning. For frame that has condition image as input (key frame), we add it to the first layer of video ControlNet. Besides, the embeddings of noise level $\sigma$ and downscaling factor $s$ are added to the existing $t$ embedding, which will be broadcast to all spatial layers.
Refer to caption: x3.png

Besides data augmentation, the corresponding conditioning mechanism should also be designed in order to boost the model training and avoid averaging performance for different space or time scales and noise augmentation. In practice, the condition latent sequence $z_{s,\sigma}^{1:f:m}$, the corresponding downscaling factor $s$, and augmented noises $\sigma$ are all considered as for conditioning. Please refer to Fig. 3 for more intuitive demonstration.

Given the synthesized condition latent sequence $z_{s,\sigma}^{1:f:m}$, we use one convolution layer with zero-initialization $\texttt{Conv}_{\texttt{zero}}$ for connecting it to video ControlNet. Specifically, we have:

$$ $\displaystyle f_{out}^{1:f}=\texttt{Conv}(z_{t}^{1:f}),$ (6) $\displaystyle f_{out}^{1:f:m}=\texttt{Conv}(z_{t}^{1:f:m})+\texttt{Conv}_{ \texttt{zero}}(z_{s,\sigma}^{1:f:m}),$ (7) $$

where Conv is the first convolution layer in video ControlNet, $z_{t}^{1:f}$ and $z_{t}^{1:f:m}$ denote the full frames and key frames of noisy latents at timestep $t$, respectively.
Note that Conv and $\texttt{Conv}_{\texttt{zero}}$ share the same hyper-parameter configuration (i.e.,kernel size, padding, et.al.),
As it is shown, only key-frame features in video ControlNet will be added with the condition features, while others remain unchanged. This strategy enables progressive condition injection as the weights of $\texttt{Conv}_{\texttt{zero}}$ grows from zero starting point.

For conditioning regarding downscaling factor $s$ and noise augmentation $\sigma$, we incorporate them to the existing time embedding in video ControlNet. Specifically, for timestep $t$, sinusoidal encoding [, , ] is used to provide the model with a
positional encoding for time. Then, one MLP (two linear layers with a SiLU [] activation layer in between) is applied. Specifically, we have:

$$ $\displaystyle t_{pos}=\texttt{Sinusoidal}(t),$ (8) $\displaystyle t_{emb}=\texttt{Linear}_{\texttt{t}}^{(2)}(\texttt{SiLU}(\texttt {Linear}_{\texttt{t}}^{(1)}(t_{pos}))),$ (9) $\displaystyle t_{emb}^{1:f}=\texttt{Repeat}(t_{emb},f),$ (10) $$

where $t_{emb}^{1:f}$ is obtained by Repeat $t_{emb}$ by $f$ times in the frame axis. This time embedding sequence will be broadcast to all ResBlocks in video ControlNet for timestep injection.

Also, we elucidate the conditioning for noise augmentation. As mentioned in Eq.(4), noise augmentation shares the same way as diffusion process, but with much smaller maximum timestep (i.e., $T^{\prime}=300$). Therefore, we reuse the encoding and mapping for timestep $t$ in diffusion process. After this, we add a linear layer with zero initialization (denoted as $\texttt{Linear}_{\texttt{zero}}$).
To conclude, we have:

$$ $\displaystyle\sigma_{pos}=\texttt{Sinusoidal}(\sigma),$ (11) $\displaystyle\sigma_{emb}=\texttt{Linear}_{\texttt{zero},\sigma}(\texttt{ Linear}_{\texttt{t}}^{(2)}(\texttt{SiLU}(\texttt{Linear}_{\texttt{t}}^{(1)}( \sigma_{pos})))).$ (12) $$

To achieve video-aware conditioning, we add $\sigma_{emb}$ only to the key frames. So $\sigma_{emb}$ is repeated $k$ times, where $k$ is the number of key frames. The video-aware controlling is presented as follows:

$$ $\displaystyle t_{emb}^{1:f:m}=t_{emb}^{1:f:m}+\sigma_{emb}^{1:k},$ (13) $$

where the addition operation is performed frame-wisely for key frames.

Regarding downscaling factor $s$, the corresponding encoding, mapping and controlling are similar as above.
In particular, we newly introduce one MLP, in which the output layer is zero-initialized. The video-aware conditioning is performed as:

$$ $\displaystyle s_{pos}=\texttt{Sinusoidal}(s),$ (14) $\displaystyle s_{emb}=\texttt{Linear}_{\texttt{zero},\texttt{s}}(\texttt{SiLU} (\texttt{Linear}_{\texttt{s}}(s_{pos}))),$ (15) $\displaystyle s_{emb}^{1:k}=\texttt{Repeat}(s_{emb},k),$ (16) $\displaystyle t_{emb}^{1:f:m}=t_{emb}^{1:f:m}+s_{emb}^{1:k}.$ (17) $$

With our proposed space-time data augmentation and video-aware conditioning, VEnhancer can be well-trained in an end-to-end manner, and yields great performance for generative enhancement.

## 4 Experiments

Datasets.
We collect around 350k high-quality and high-resolution video clips from the Internet to constitute our training set.
We train VEnhancer on resolution $720\times 1280$ with center cropping, and the target FPS is fixed to $24$ by frame skipping.
Regarding test dataset, we collect comprehensive generated videos from state-of-the-art text-to-video methods [, , , ]. Practically, we select videos with large motions and diverse contents. This test dataset is denoted as AIGC2023, which is used to evaluate VEnhancer and baselines for video super-resolution and space-time super-resolution tasks.
For evaluation on VBench, all generated videos based on the provided prompt suite are considered, resulting in more than 5k videos.

Implementation Details.
The batch size is set to 256. AdamW
[] is used as the optimizer, and the learning rate is set to $10^{-5}$.
During training, we dropout the text prompt with a probability of 10%.
The training process lasts about four days with 16 NVIDIA A100 GPUs. During inference, we use 50 DDIM [] sampling steps and classifier-free guidance (cfg) [].

Metrics.
Regarding evaluation for video super-resolution and space-time super-resolution on AIGC2023 test dataset, we use both image quality assessment (IQA) and video quality assessment (VQA) metrics. As there is no ground-truth available, we can only use non-reference metrics. Specifically, MUSIQ [] and DOVER [] are adopted. Moreover, we refer to video generation benchmark,
VBench [], for more comprehensive evaluation.
Specifically, we choose Subject Consistency (i.e., whether the subject remains consistent), Motion smoothness (i.e., how smooth the video is), Aesthetic Quality, and Imaging Quality for evaluation. Regarding evaluation for video generation, we consider all 16 evaluation dimensions from VBench.

### 4.1 Comparison with Video Super-Resolution Methods

For video super-resolution,
VEnhancer is compared with the state-of-the-art real-world video super-resolution method, RealBasicVSR [], and the state-of-the-art generative video super-resolution method, LaVie-SR [] (super-resolution).

**Table 1: Quantitative comparison for video super-resolution ($4\times$) on AIGC2023 test dataset. Red and blue indicate the best and second best performance. The top 3 results are marked as gray.**
|  | DOVER↑ | MUSIQ↑ | Imaging<br>Quality | Aesthetic<br>Quality | Subject<br>Consistency | Motion<br>Smoothness |
| --- | --- | --- | --- | --- | --- | --- |
| LaVie-SR [] | 0.8427 | 55.8428 | 0.5481 | 0.6692 | 0.9562 | \cellcolor[HTML]F2F2F20.9710 |
| RealBasicVSR [] | \cellcolor[HTML]F2F2F20.8252 | \cellcolor[HTML]F2F2F250.5978 | \cellcolor[HTML]F2F2F20.5401 | \cellcolor[HTML]F2F2F20.6622 | \cellcolor[HTML]F2F2F20.9555 | 0.9729 |
| Ours | 0.8498 | 56.6113 | 0.5872 | 0.6728 | 0.9624 | 0.9787 |

Figure: Figure 4: Visual comparison for video super-resolution ($4\times$) on AIGC2023 test dataset. Prompt: Iron Man flying in the sky. Zoom in for best view.
Refer to caption: x4.png

As shown in Table 1, VEnhancer outperforms both generative video super-resolution method (LaVie-SR []) and real-world video super-resolution method (RealBasicVSR []) in all metrics, suggesting its outstanding enhancement ability for videos.
Note that LaVie-SR surpasses RealBasicVSR in image/video quality, as LaVie-SR obtains higher scores in MUSIQ, DOVER, Imaging Quality, and Aesthetic Quality. This indicates that LaVie-SR could produce sharper results than RealBasicVSR. It is because that LaVie-SR is based on diffusion models while RealBasicVSR is trained with GAN loss. Besides, RealBasicVSR shows better performance in Motion Smoothness compared with LaVie-SR, since smooth results are more likely to achieve higher score in this metric. Practically, there is a trade off between
video smoothness and video quality [].
Nevertheless, VEnhancer achieves a great balance between these two aspects.

The visual comparison is presented in Fig. 4. The prompt is “Iron man flying in the sky".
The input video is already consistent with the prompt, but lacks details on the iron man suit. RealBasicVSR could
remove some noises or artifacts of the generated videos as it incorporates complex degradation for model training. However, it fails in generating realistic details but produces over-smoothed results, since its generative ability is limited.
On the other hand, the results of LaVie-SR contains more artifacts than input.
Without successfully removing artifacts, the generative super-resolution model will enlarge the existing defects.
In contrast, VEnhancer could first remove unpleasing artifacts and refine the distorted content (e.g., head region), and then generate faithfuls details (e.g., helmet and armor) that are consistent with the text prompt. Also, the whole video is with high-quality and high-resolution, as well as smooth dynamics.

### 4.2 Comparison with Space-Time Super-Resolution Methods

For space-time super-resolution task, we compare two state-of-the-art space-time super-resolution methods: VideoINR [] and Zooming-Slow-Mo [] (Zoom for short).
We also consider LaVie’s two-stage pipeline: LaVie-FI (frame interpolation) + LaVie-SR for more thorough comparison.

**Table 2: Quantitative comparison for space-time super-resolution ($4\times$) on AIGC2023 test dataset. Red and blue indicate the best and second best performance. The top 3 results are marked as gray.**
|  | DOVER↑ | MUSIQ↑ | Imaging<br>Quality | Aesthetic<br>Quality | Subject<br>Consistency | Motion<br>Smoothness |
| --- | --- | --- | --- | --- | --- | --- |
| LaVie-FI + LaVie-SR [] | 0.8159 | 53.2128 | 0.5299 | 0.6566 | \cellcolor[HTML]F2F2F20.9603 | 0.9857 |
| VideoINR [] | \cellcolor[HTML]F2F2F20.7608 | \cellcolor[HTML]F2F2F234.1060 | 0.4778 | \cellcolor[HTML]F2F2F20.6624 | 0.9615 | 0.9933 |
| Zooming Slow-Mo [] | 0.7328 | 33.8470 | \cellcolor[HTML]F2F2F20.4925 | 0.6624 | 0.9524 | 0.9908 |
| Ours | 0.8487 | 50.3659 | 0.5648 | 0.6665 | 0.9666 | \cellcolor[HTML]F2F2F20.9898 |

Figure: Figure 5: Visual comparison for space-time super-resolution ($4\times$) on AIGC2023 test dataset. Prompt: A cute raccoon playing guitar in a boat on the ocean. Zoom in for best view.
Refer to caption: x5.png

As shown in Table 2, we observe that our method surpasses all baselines in DOVER, Imaging Quality, and Aesthetic Quality, showing its superior capability in generating sharp and realistic video content. Besides, it obtains highest score in Subject Consistency, indicating VEnhancer’s ability in maintaining the subject consistent when performing joint frame interpolation and spatial super-resolution.
We notice that state-of-the-art space-time super-resolution methods (VideoINR and Zooming Slow-Mo) stand out in Motion Smoothness. As both of them are optimized with reconstruction loss, the produced results are very smooth across frames.
At a cost, they perform poorly in metrics regarding quality, such as DOVER, MUSIQ, and Imaging Quality.
The two-stage approach (LaVie-FI + LaVie-SR) obtains good scores in DOVER, MUSIQ and Imaging Quality, demonstrating DM-based methods’ advantage in generation. However, its performance in video stability and subject consistency is unsatisfactory due to its inferior capability in video refinement.

The visual comparison is illustrated in Fig. 5.
The first and third columns present the low-resolution key frames.
Note that the input frames are not consistent especially in the region of guitar strings. The two-stage DM-based approach, LaVie-FI + LaVie-SR [], can produce very sharp results for all frames (key and predicted ones). However, it generates messy contents which are not semantically aligned with prompt. Moreover, the generated details are changing across time, indicating severe flickering.
For reconstruction-based methods (VideoINR [] and Zoom []), the produced results are similar: lacking details and fail in improving the consistency of the original input frames.
On the contrary, VEnhancer is not only able to achieve unified space-time super-resolution, but can also improve the temporal consistency of the generated videos by refinement (i.e., guitar strings and raccoon hands plucking the strings).

### 4.3 Evaluation on Improving Video Generation

In this section, we evaluate VEnhancer’s ability in improving the existing state-of-the-art T2V method (i.e., VideoCrafter2 []). The baselines includes open-source T2V methods: VideoCrafter-2 (VC-2 for short), Show-1 [], Lavie [], Open-Sora (^†^†https://github.com/hpcaitech/Open-Sora), as well as professional video generation products, Pika and Gen-2. All dimensions of VBench are considered for evaluation.

The quantitative results are organized in Table 3.
In general, VideoCrafter-2 is already the best in overall Semantic compared with other baselines, demonstrating its superiority in generating high-fidelity contents to the VBench’s prompt suite. Regarding the overall Quality, it lags behind Pika and Gen-2.
But with VEnhancer, VideoCrafter-2 is able to achieve the highest scores in both overall Quality and Semantic. This indicates that VEnhancer can improve the semantic content and video quality at the same time, showing a powerful unified enhancement (i.e., refinement and space-time super-resolution) ability.

Regarding Quality aspect, we note that Pika and Gen-2 could obtain good scores in various dimensions, but perform very poorly in dynamic degree. This dimension assesses whether a video has large motions, and higher score corresponds to larger motion. It is suggested that Gen-2 and Pika
obtain good temporal consistency by sacrificing versertile motions.
This demonstrates the advantage of adopting a two-stage pipeline: the first T2V model focuses on generating semantic content and motions with good fidelity to the prompts, while the following enhancement model can improve the semantic in low-level and image quality, as well as temporal consistency. We also provide visual comparison in Fig. 6. It is obvious that VideoCrafter-2 yields the best video content but with low-resolution. And VEnhancer significantly improves its quality (with 2k resolution). Gen-2 and Pika could produce results with around 1k resolution, but their generated contents have some apparent flaws (weird robe in Gen-2 and unnatural rabbit’face in Pika). Open-Sora and Lavie both generate videos with square size, leading to incomplete generation for backgrounds.

**Table 3: VBench Evaluation Results per Dimension. This table compares the performance of four open-source video generation models (LaVie [], Show-1 [], Open-Sora, VideoCrafter-2 []) and two professional video generation products (Pika and Gen-2) across each of the 16 VBench dimensions regarding two aspects (Quality and Semantic). A higher score indicates relatively better performance for a particular dimension. Red and blue indicate the best and second best performance. The top 3 results are marked as gray.**
|  | Dimensions | Show-1 [] | LaVie [] | Open-Sora | Pika | Gen-2 | VC-2 [] | VC-2+VEnhancer |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Subject Consistency | 95.53% | 91.41% | 92.09% | 96.76% | \cellcolor[HTML]F2F2F297.61% | \cellcolor[HTML]F2F2F296.85% | \cellcolor[HTML]F2F2F297.17% |
|  | Background Consistency | 98.02% | 97.47% | 97.39% | \cellcolor[HTML]F2F2F298.95% | 97.61% | \cellcolor[HTML]F2F2F298.22% | \cellcolor[HTML]F2F2F298.54% |
|  | Temporal Flickering | \cellcolor[HTML]F2F2F299.12% | 98.30% | 98.41% | \cellcolor[HTML]F2F2F299.77% | \cellcolor[HTML]F2F2F299.56% | 98.41% | 98.46% |
|  | Motion Smoothness | \cellcolor[HTML]F2F2F298.24% | 96.38% | 95.61% | \cellcolor[HTML]F2F2F299.51% | \cellcolor[HTML]F2F2F299.58% | 97.73% | 97.75% |
|  | Aesthetic Quality | 57.35% | 54.94% | 57.76% | \cellcolor[HTML]F2F2F263.15% | \cellcolor[HTML]F2F2F266.96% | 63.13% | \cellcolor[HTML]F2F2F265.89% |
|  | Dynamic Degree | \cellcolor[HTML]F2F2F244.44% | \cellcolor[HTML]F2F2F249.72% | \cellcolor[HTML]F2F2F248.61% | 37.22% | 18.89% | 42.50% | 42.50% |
| Quality | Imaging Quality | 58.66% | 61.90% | 61.51% | 62.33% | \cellcolor[HTML]F2F2F267.42% | \cellcolor[HTML]F2F2F267.22% | \cellcolor[HTML]F2F2F270.45% |
|  | Object Class | \cellcolor[HTML]F2F2F293.07% | 91.82% | 74.98% | 87.45% | 90.92% | \cellcolor[HTML]F2F2F292.55% | \cellcolor[HTML]F2F2F293.39% |
|  | Multiple Objects | 45.47% | 33.32% | 33.64% | \cellcolor[HTML]F2F2F246.69% | \cellcolor[HTML]F2F2F255.47% | 40.66% | \cellcolor[HTML]F2F2F249.83% |
|  | Human Action | \cellcolor[HTML]F2F2F295.60% | \cellcolor[HTML]F2F2F296.80% | 85.00% | 88.00% | 89.20% | \cellcolor[HTML]F2F2F295.00% | \cellcolor[HTML]F2F2F295.00% |
|  | Color | 86.35% | 86.39% | 78.15% | 85.31% | \cellcolor[HTML]F2F2F289.49% | \cellcolor[HTML]F2F2F292.92% | \cellcolor[HTML]F2F2F294.41% |
|  | Spatial Relationship | 53.50% | 34.09% | 43.95% | \cellcolor[HTML]F2F2F265.65% | \cellcolor[HTML]F2F2F266.91% | 35.86% | \cellcolor[HTML]F2F2F264.88% |
|  | Scene | 47.03% | \cellcolor[HTML]F2F2F252.69% | 37.33% | 44.80% | 48.91% | \cellcolor[HTML]F2F2F255.29% | \cellcolor[HTML]F2F2F251.82% |
|  | Appearance Style | 23.06% | \cellcolor[HTML]F2F2F223.56% | 21.58% | 21.89% | 19.34% | \cellcolor[HTML]F2F2F225.13% | \cellcolor[HTML]F2F2F224.32% |
|  | Temporal Style | 25.28% | \cellcolor[HTML]F2F2F225.93% | \cellcolor[HTML]F2F2F225.46% | 24.44% | 24.12% | \cellcolor[HTML]F2F2F225.84% | 25.17% |
| Semantic | Overall Consistency | \cellcolor[HTML]F2F2F227.46% | 26.41% | 26.18% | 25.47% | 26.17% | \cellcolor[HTML]F2F2F228.23% | \cellcolor[HTML]F2F2F227.57% |
|  | Quality | 80.42% | 78.78% | 78.82% | \cellcolor[HTML]F2F2F2 82.68% | \cellcolor[HTML]F2F2F282.46% | 82.20% | \cellcolor[HTML]F2F2F2 83.28% |
| Overall | Semantic | 72.98% | 70.31% | 64.28% | 71.26% | \cellcolor[HTML]F2F2F273.03% | \cellcolor[HTML]F2F2F2 73.42% | \cellcolor[HTML]F2F2F2 76.73% |

Figure: Figure 6: Visual comparison of screenshots obtained from VideoCrafter-2+VEnhancer and other T2V models. The layout is arranged based on their original resolutions. Zoom in for best view.
Refer to caption: x6.png

### 4.4 Ablation Studies

Please refer to Appendix A for arbitrary up-sampling scales for space/time super-resolution, as well as different noise levels in noise augmentation.

## 5 Conclusion and Limitation.

In this work, we propose a generative space-time enhancement method, VEnhancer, for video generation. It can achieve spatial super-resolution, temporal super-resolution and video refinement in a unified framework. We base on a pretrained video diffusion model and build a trainable video ControlNet for effective condition injection. Space-time data augmentation and video-aware conditioning are proposed to train video ControlNet in an end-to-end manner. Extensive experiments have demonstrated the superiority over state-of-the-art video super-resolution and space-time
super-resolution methods in enhancing AI-generated videos. Moreover, VEnhancer is able to improve the results of existing state-of-the-art T2V method, lifting VideoCrater-2’s ranking to top one. However, our work has several limitations. First, as it is based on diffusion models, the inference takes more time than one-step methods, for instance, reconstruction-based space-time super-resolution methods. Second, it may face challenges in handling AI-generated long videos, since the long-term (over 10s) consistency has not been addressed in this work.

## Appendix A Ablation Studies

### A.1 The Effectiveness of Noise Augmentation.

During training, the noise level regarding noise augmentation is randomly sampled within a predefined range. While during inference, one can change the noise level to achieve refinement with different strengths. In general, higher noise corresponds to stronger refinement and regeneration.
We present the visual comparison among different noise levels in Figure 7. The first frame of one AI-generated video is presented in the left. It is of low-resolution and lacks details. Also, the original video has very obvious flickering. If we set $\sigma=0$, VEnhancer will generate unpleasing noises in the background. As there is domain mismatch between the training data and testing data, the enhancement fails in handling unseen and challenging scenarios. Fortunately, we can mitigate this by adding noise in the condition latents for corrupting the noisy and unknown low-level details. As we increase the noise level, the artifacts are gradually vanishing. When $\sigma=250$, the result is noise-clean, and has abundant semantic details.

Figure: Figure 7: Visual comparison of setting different noise levels in noise augmentation during testing.
Refer to caption: x7.png

### A.2 Arbitrary Up-sampling Scales for Spatial Super-Resolution.

In this section, we show that VEnhancer is able to up-sample videos with arbitrary scales.
From Figure 8, we observe that VEnhancer could produce satisfactory results on different scales ($2.5\times$, $3\times$, $3.5\times$, $4\times$, and $4.5\times$), suggesting its flexibility and generalization in adapting to different tasks.
In particular, given one frame of the generated video ($312\times 512$), VEnhancer could improve the generated details when the up-sampling scale grows up. When $s=2.5\sim 3.5$, the panda’s hand is less realistic. But it becomes better when $s=4$ or $s=4.5$. It is also noticed that the panda’ fur is becoming more realistic as $s$ grows.

Figure: Figure 8: Visual results of different up-sampling scales ($2.5\times$, $3\times$, $3.5\times$, $4\times$, and $4.5\times$) for spatial super-resolution during testing.
Refer to caption: x8.png

### A.3 Arbitrary Up-sampling Scales for Temporal Super-Resolution.

Figure: Figure 9: Visual results of different up-sampling scales ($3\times$, $4\times$, and $5\times$) for temporal super-resolution during testing.
Refer to caption: x9.png

In this part, we show VEnhancer is able to achieve arbitrary up-sampling in time axis. Given two low-resolution key frames, we aim to up-sample them to high-resolution ones, and also interpolate several frames (ranging from $2$ to $4$) between them. As shown in Figure 9, the results are consistent across frames, showing not flicking or distortions. Besides, the spatial quality has also been significantly improved. As shown in the last row, $5\times$ frame interpolation yields smooth frames with generated contents: the shadow in the right leg is changing, showing a very natural transition. This indicates that diffusion-based frame interpolation has great capability in both motion and content generation.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Are the shapes of inputs to ST-Controller different because of different interval m? How does the model handle inputs with different m?
soundness: 3
strengths: 1. Novelty: The paper presents a unified approach for generative spatial and temporal super-resolution, which is novel in the field of video generation. The integration of a pretrained generative video prior with a ST-Controller for conditioning is a creative solution that addresses the limitations of cascaded models. The concept of space-time data augmentation and video-aware conditioning is innovative and contributes to the training of the ST-Controller in an end-to-end manner.

2. Quality: The paper is well-structured, with a comprehensive presentation of the methodology, experiments, and results. The visual results and quantitative metrics provided are convincing and demonstrate the effectiveness of VEnhancer.

3. Significance: The work is relatively significant since it's an important complementary to current open-sourced video diffusion models.The ability to handle many up-sampling scales and to refine videos while maintaining content fidelity is a substantial contribution to the field.
summary: The paper titled "VEnhancer: Generative Space-Time Enhancement for Video Generation" introduces a novel method for enhancing AI-generated videos both spatially and temporally using a single model. VEnhancer is capable of up-sampling the resolution and frame rate of low-quality videos with many scales, while adding spatial details and synthesizing detailed motion. It also removes artifacts and flickering, improving upon existing methods by integrating a Space-Time Controller (ST-Controller) trained with space-time data augmentation and video-aware conditioning. The method aims to be end-to-end trainable, enabling multi-function enhancement within one model, and claims to surpass current state-of-the-art in video super-resolution and space-time super-resolution for AI-generated content.
weaknesses: 1. Some expressions of the paper are not clear and rigorous enough, and there are certain ambiguities, ambiguities or even errors. Below are the instances:
- Wrong notations, instead of 𝐼^(1:𝑚:𝑓), 𝐼^(1:𝑚:𝑓) is typically used to denote a sequence starting from 1, ending at 𝑓, with a step size of 𝑚. Similar cases for z, t, \sigma, and s. Besides, in Fig.3 z^{1:m}_t, t^{1:m} should be z^{1:f}_t, t^{1:f} 
- Wrong illustrations. In Fig.2 Space-Time Data Augmentation part, both the noised videos (with noise strength t and \sigma) should be in latent space rather than still being natural videos. The noises are directly added to the latent videos.

2.  Some critical details are lacking such as how to do inference specifically like the initialization of z_t and z_{s, \sigma} (it should be more clear if you have a pseudocode algorithm), and the inference time and performance (like Vbench Results or other metrics) across different SSR and TSR scales.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - It would be helpful if the authors could discuss more about how they collect the data, i.e., filtering out 350K data from Panda 70M.
soundness: 3
strengths: - Handles multiple up-scaling in both space and time dimensions, allowing for versatile application scenarios.
- Improves the fidelity of generated videos while maintaining or enhancing detail, demonstrated through rigorous testing against current top methods.
summary: The paper introduces VEnhancer, a novel generative method designed to enhance both the spatial and temporal quality of AI-generated videos. VEnhancer increases resolution, adds detail, and reduces artifacts and flickering through a single model, leveraging a space-time controller for conditioning on low-quality inputs.
### Contributions

1. **Unified Model:** VEnhancer combines spatial and temporal super-resolution and video refinement capabilities in one model.
2. **Space-Time Controller:** Introduces a mechanism for injecting multi-frame conditions based on a pre-trained generative video prior.
3. **Performance:** Demonstrates superiority over some of the existing state-of-the-art methods in video super-resolution through extensive experiments.
weaknesses: - As with most diffusion models, the complexity of the model could lead to longer inference times, which may limit its applicability in real-time or low-resource scenarios.
- The model's performance heavily relies on the availability of high-quality training data, which might not always be available or feasible to collect in certain domains.
- For each different text-to-video model, a new VEnhancer need to be trained to accommodated the different architecture, limiting the use case of the proposed method. It would be more practical if the proposed method could be a standalone video super-resolution toolkit.
- Other diffusion based video super-resolution works is not compared in the paper, e.g. Upscale-a-video.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 3
strengths: 1.	The proposed approach unifies the space-time video super-resolution through re-sampling operation over time-space axes, achieving multi-function in a single video diffusion model.
2.	Both of the objective video quality evaluation and subjective human evaluation show the effectiveness of the proposed approach in video enhancement. 
3.	The paper is well-written and technical description is clear.
summary: The paper presents a diffusion-based video space-time enhancement model. Specifically, the proposed VEnhancer leverages a control branch to integrate the low-resolution and low-frame-rate video prior into diffusion. Besides, a new data augmentation strategy is proposed to carry out the video super-resolution and enhancement with different functions, e.g., enhancing video with details in different levels. Comparisons with both video super-resolution methods and cascaded super-resolution methods verify the superiority of the proposal.
weaknesses: 1.	One of my key concerns about the technical novelty is about the involving of the UNet-based control branch whose weights are copied from the original I2VGen-XL. The novelty seems limited since the controling approach has been proposed by ControlNet and the key frame information integration via feature summation is also intuitive. 
2.	The technical design of space-time data augmentation should be investigated quantitatively in the main paper. Besides, the motivation of the position embedding encoded from the noise augmentation is unclear. What are the effects by exploiting this parameter $\sigma$ in video enhancement? 
3.	There should be more in-depth analysis or discussions with other SOTA approaches to demonstrate the technical contributions of the proposal. Only describing the visual results cannot give readers many insights, but only leads them think that the proposed approach is more like engineering rather than research.  
4.	The ablation study should be conducted in the main paper instead of the appendix. 
5.	Some format issue and typos: a)	The "ramdom" in line 259; b)	Remove period symbol in the section title “Conclusion and Limitation.”

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In terms of experimental results, the proposed method shows only minor improvements in Aesthetic Quality Dynamic, Degree of Motion, and Smoothness, while achieving relatively significant improvements in MUSIQ and DOVER. What causes this discrepancy?
2. As shown in Table 3, the improvement of the proposed method on CogVideoX-5B is relatively minor and not as significant as on VideoCrafter-2. Could the authors explain the reasons for this?
3. How many videos are there in the AIGC2023 dataset? What are the approximate resolutions and lengths of these videos? Is there any statistical information available? Will the authors release this test set in the future?
soundness: 2
strengths: 1. From the visual results provided by the authors, the proposed method enhances the video's spatial resolution and FPS. The details of objects are richer compared to the pre-enhancement version, and the smoothness of the video has also improved to some extent.
2. This method can achieve both temporal and spatial video super-resolution simultaneously.
summary: This paper proposes a video super-resolution algorithm called VEnhancer, which enhances video resolution in both spatial and temporal dimensions. The method is based on a pretrained video generation model, with a Spatio-Temporal (ST) Controller trained specifically for the task. The video to be enhanced is provided as a condition input to the ST-Controller. The authors constructed a test dataset, AIGC2023, to validate the effectiveness of the algorithm.
weaknesses: 1. The proposed method's overall idea is quite similar to ControlNet: both introduce a control branch by copying the encoder of the UNet model to the control branch and initializing the output layer to zero. The difference lies in their applications—this method uses the framework for video super-resolution, while ControlNet is designed for controllable generation. In this paper, the condition is the video to be super-resolved, whereas in ControlNet, conditions include modalities like depth, edge, and mask. The authors should explicitly discuss how their approach differs from or improves upon ControlNet for the specific task of video enhancement. 
2. For the video-aware conditioning (section 4.3), the noise augmentation technique proposed by the authors is a common trick that was first introduced by *Cascaded Diffusion Models* [Jonathan Ho et al., 2021] for multi-stage high-resolution image generation. Essentially, it incrementally upscales from low to high resolution, which is quite similar to this paper's application. Additionally, using the time embedding as a condition is a standard approach in diffusion models. Conditioning on the downscale factor is somewhat analogous to using FPS as a condition in video generation [Make-A-Video, Uriel Singer et al., 2022], allowing the generation process to better incorporate additional conditioning information. This approach isn't particularly novel in itself. The authors should clarify what specific innovations or improvements their approach offers over these existing techniques. 
3. The proposed method is for video super-resolution and, theoretically, should be applicable to both AI-generated videos and real videos. However, the authors only conducted experiments on AI-generated videos (section 5.3), making this comparison less comprehensive. I suggest that the authors include experiments on real-world videos or explain why their method might not be suitable for such videos if that's the case.

Meta-review: This paper received mixed scores: two borderline accept, one borderline reject, and one reject. Most reviewers acknowledged the simple and effective solution proposed in the paper. However, some reviewers raised concerns, including the high similarity to existing methods like ControlNet, limited original contributions (as many components for video-aware conditioning were borrowed from previous works), and the limited applicability of the proposed methods. The AC has reviewed the paper, the reviews, and the rebuttal. While super-resolution for AI-generated content is an intriguing and promising idea, the current version lacks a clear, significant contribution that could be recognized as a major advancement. Although two reviewers gave acceptance recommendations, their arguments lacked specific reasons strong enough to overturn the rejection opinions. Therefore, AC agrees that the current version is not ready for acceptance. Nevertheless, the AC strongly encourages the authors to address all the reviewers' concerns and resubmit to a future venue.

Decision: Reject