SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution
Authors: Qi Tang
Yao Zhao
Meiqin Liu
Chao Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Super-Resolution
Diffusion Model
Primary area: machine_vision

Abstract: Diffusion-based Video Super-Resolution (VSR) is renowned for generating perceptually realistic videos, yet it grapples with maintaining detail consistency across frames due to stochastic fluctuations. The traditional approach of pixel-level alignment is ineffective for diffusion-processed frames because of iterative disruptions. To overcome this, we introduce SeeClear--a novel VSR framework leveraging conditional video generation, orchestrated by instance-centric and channel-wise semantic controls. This framework integrates a Semantic Distiller and a Pixel Condenser, which synergize to extract and upscale semantic details from low-resolution frames. The Instance-Centric Alignment Module (InCAM) utilizes video-clip-wise tokens to dynamically relate pixels within and across frames, enhancing coherency. Additionally, the Channel-wise Texture Aggregation Memory (CaTeGory) infuses extrinsic knowledge, capitalizing on long-standing semantic textures. Our method also innovates the blurring diffusion process with the ResShift mechanism, finely balancing between sharpness and diffusion effects. Comprehensive experiments confirm our framework's advantage over state-of-the-art diffusion-based VSR techniques.

Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution
Tang 1.2
Yao Zhao 1.2, Meiqin Lin Chao Yao
Institute of Information Science, Beijing Jiaotong University
2 Visual Intelligence + X International Cooperation Joint Laboratory of MOE, Beijing Jiaotong University
3 School of Computer and Communication Engineering. University of Science and Technology Beijing
{qitang, yzhao, mqliu)@bjjt edu. en, yaochaolustb. edu.
Abstract
Diffusion-based Video Aoper-Resolution (VSR) is renowned for generating percep- tually realistic videos, yet grapples with maintaining detail consistency across frames due to stochastic fluctuations. The traditional approach pixel-level align- ment is ineffective for diffusion diffusion-processee processed frames because of iterative disruptions. To overcome this, we introduce SeeClear-a novel VSR framework leveraging conditional video generation, orchestrated by instance-centric and channel-wise semantic controls. This framework integrates a Semantic Distiller and a Pixel Con- denser, which synergize to extract and upscale semantic details from low-resolution frames. The Instance-Centee tokens dynamically relate pixels within and across frames, enhancing coherency. Additionally, the Channel- -iie Texture Aggregation Memory (CaTeGory) infuses extrinsic knowledge, capitalizing on long-standie standing semantic textures. Our method also innovates the blurring diffusion process with the ResShift mechanism, finely balancing between sharpness and diffusion effects. Comprehensive experiments confirm our framework's advantage over state-of- ofte------th-tttteee diffusson-based VSR tech- niques. The code is available: https://github..omm http://aannoo//eeeeeeeeaeeaa
Alignment Module (InCAM) utilizes video-clip-wis
Introduction
Video super-resolution (VSR) is challenging low-level vision task that involves improving the resolution and visual quality of the given low-rcsolution (LR) observations and maintaining the temporal coherence of high-resolution (HR) components. Various deep-learning-beee based VSR ap- proaches 13 30 17 13 20 21 explore effective inter-frame alignment to reconstruct satisfactory new pole after another in the quantitative results, they struggle to generate photo-realistic textures.
sequences. Despite establishing
With the explosion of diffusion model (DM) in visual generation [10 31 26 super-resolution (SR) from the generative perspective also garners the broad attention 228 29
5. DM breaks generation process into sequential sub sub-processes and iteratively samples semantic-specific images from Gaussian noise, equipped with a paired forward diffusion process and reverse denoising process. The former progressively injects varied intensity noise into the image along Markov chain to simulate diverse image distributions. The latter leverages denoising network to generate an image based the given noise and conditions. Early efforts directly apply the generation paradigm to super-resolution, overlooking its characteristic while generating pleasing content, thus trapping in huge sampling overhead.
45
Conresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Different from generation from scratch, SR resembles partial generation. The structural information that dominates the early stages of diffusion is contained in the LR priors. while SR tends focus on Besides, the loss of high-frequency information in LR videos stems from the limited sensing range of imaging equipment. As a result, solely disrupting frames with additive noise is inadequate to depict the degradation of HR videas (9. Moreover, prevalent VSR methods employ delicate inter-frame alignment (e.g.. optical flow or deformable convolution) to fuse the sub-pixel information across adjacent frames. However, the disturbed pixels pose a severe challenge to these methods, rendering the accuracy to deteriorate in the pixel space.
generating high-frequency details 33 15
To alleviate the above issue, we introduce SeeClear, an innovative diffusion model em- powering distilled semantics to enhance the pixel condensation for video super-resolution. During the forward diffusion process, the low- pass filter is applied within the patch, gradually diminishing the high-frequency component, all while the residual is progressively shifted in the frequency domain to transform the HR frames to corresponding LR versions step by step. To re- duce the computational overhead, the intermedi- ate states are decomposed into various frequency sub-bands via 2D discrete wavelet transform and subsequently processed by the attention-base devise dual semantic-
4.4.
df Piut Contuar
Figure I: The sketch of SeeClear. It consists of a Seman- tie Distiller and a Pixel Condenser, which are responsible for distilling instance-centric semantics from LR frames and generating HR frames. The instance-centric and assembled channel-wise semantics act as thermometer control the condition for generation.
U-Net. Furthermore, controlled conditional generation schema to en- hance the temporal coherence of VSR. Specifi- cally, a segmentation framework for open vocab- ulary is employed to distill instance-centric se- mantics from LR frames. They serve prompts, enabling the Iettance-Centice Aligmment Module (InCAM) highlight and associate semantically related pixels within the local temporal scope. Besides, abundant semantic cues in channel dimensions are also explored to form an extensional memory dubbed Channel-wise Texture Aggregation Memory (CaTeGoory). It aids in global temporal coherence and boosts performance. Experimental results demonstrate that our method consistently outperforms existing state-of-the tfte-offtheearr methods.
In summary, the main contributions of this work are as follows:
We present SeeClear, a diffusion- framework for video super-resolution that distills semantic priors from low-resolution frames for spatial modulation and temporal association, controlling the condition of pixel generation.
We reformulate the diffusion process by integrating residual shifting with patch-level blurring, and introduce attention- based architecture explore valuable information among wavelet spectra during the sampling process, incorporating feature modulation of intra-frame semantics.
We devise a dual semantic distillation schema that extracts instance- centric semantics of each frame and further assembles them into texture memory based on the semantic category of channel dimension, ensuring both short-term and long-term temporal coherence.
2
Related Work
2.1 Video
Prominent video super-resolution techniques concentrate on leveraging sub-pixel information across frames to enhance performance. EDVR 37 employs cascading deformable convolution layers (DCN) for inter-frame alignment in a coarse-to-fing manner, tackling large amplitude video motion. BSR [I] comprehensively explores each module's 's role in VSR and delivers simple yet effective framework by reusing previous designs with slight modifications. Given the similarity between DCN and optical flow, BasicVSR++ [3 devises flow-guided deformable alignment, exploiting the offset diversity of DCN without instability during the training. VRT [66 combines mutual attention with self-attention, which is respectively in charge of fnter-frame alignment and information preservation.

---

extends this by incorporating optical flow with deformable attention, aligning and fusing features directly at non-integer locations clip-to-clip. PSRT 30 reassesses prevalent alignment meth- ods in transformer-based VSR and implements patch alignment to counteract inaccuracies in motion estimation and compensation. DFVSR [ represents video with the proposed directional frequency representation, amalgamating object motion into multiple directional frequencies, augmented with frequency-based implicit alignment, thus enhancing alignment.
RVRT 17
2.2 Diffusion-Based Super-Resolution
Building on the success of diffusion models in the realm of image generation [0 31 26 25 48),
diffusion-basee super-resolution (SR) is advancing. SR3 [28]. a pioneering approach, iteratively samples an HR image from Gaussian noise conditioned on the LR image. In contrast, StableSR [36] applies diffusion-basee SR a low-dimensional latent space using the pre-trained auto-encoder to reduce computation and generate improved results through the generative priors contained in weights of Latent Diffusion. ResDiff [29] combines a lightweight CNN with DM to restore low-frequency and predict high-freet freeueryyyyccy components, and ResShift [45] redefines the initial step blend of the row-uusolution image and random noise to boost efficiency. Applying a different approach, DiWa (23 migrates the diffusion process into the wavelet spectrum effectively hallucinate high-frequency information. SisscaleA--Ve A-iido [49], for video super-resolution, introduces temporal layers into the U-Net and VAF-Deccderr and deploys flow-guided recurrent latent propagation module to ensure temporal coherence and overall video stability when applying image-wise diffusion model.
2.3 Semantic- Restoration
Traditionally seen a preparatory step for subsequent tasks [50 12]. restoration is now reformulated with the assistance of semantics. SFT (38) utilizes semantic segmentation probability maps for spatial modulation of intermediate features in the SR network, yielding more realistic textures. SKF [40] supports low-light image enhancement model to learn diverse priors encapsulated in semantic segmentation model by semantic-aware embedding module paired with cemantic-guides losses. SeD [14] integrates semantics into the discriminator of GAN- -based SR for fine- fine-grained texture generation rather than solely learning coarse- -rarneegr distribution. CoSeR [32] bridges image appearance and language understanding to empower SR with global cognition buried in LR image, regarding priors of text-to-image (T2I) diffusion model and a high-resolution reference image as powerful conditions. SeeSR [39] analyzes several types of semantic prompts and opts tag-style semantics to harness the generative potential of the T21 model for real SR. Semantic Lens [34] forgoes pixel-level inter-frame alignment and distills diverse semantics for temporal association in the instance-centric semantic space, attaining better performance.
3 Methodology
Given low-resolution (LR) video sequence of N frames 1. NXCCHXH where is the frame index, H x W represents spatial dimensions, and C stands for the channel of frame. SeeClcar aims to exploit rich semantic priors to generate the high-resolution (HR) video I E N xx x with as the upscaling factor. In the iterative paradigm of the diffusion model, HR frames are corrupted according to handcrafted transition distribution each diffusion step (t 1,2.... .T). And a U-shaped network is employed to estimate the posterior distribution using LR frames as condition during reverse generation. As illustrated in Figure it consists of a Semantic Distiller and a Pixel Condenser, respectively responsible for semantic extraction and texture generation.
The LR video is initially split into non-overlapping clips composed of m frames for parallel processing. Semantic Distiller, pre-trained network for open-vocabulary segmentation, distills semantics related both instances and background clip by clip, denoted as instance-centric semantics. Pixel Condenser is an attention- based encoder-decode architecture, in which the encoder extracts multi-scale features under the control of LR frames, and the decoder generates HR frames from coarse fine. They are also bridged via skip connections to transmit high frequeeryynncy information at the same resolution. To maximize the network's generative capacity, instance-centric semantics are utilized as conditions for individual frame generation in the decoder. They also serve as the cues of inter- frame- alignment for temporal coherence within the video clip and further cluster into semantic-textur memory along channel dimension for consistency across clips.
3

---

3.1 Blurring ResShift
During the video capturing, frequencies exceeding the imaging range of the device are truncated, leading to the loss of high-frequency information in LR vidcos. Therefore, an intuition is to construct a Markov chain between HR frames and LR frames in the frequency domain. Inspired by blurring diffusion .. the forward diffusion process of SeeClear initializes with the approximate distribution of HR frames. It then iterates and terminates with the approximate distribution of LR frames using a Gaussian kernel convolution in frequency space facilitated by the Discrete Cosine Transformation (DCT). Considering the correlation of neighboring information, blurring is conducted within a local patch instead of the whole image. The above process formulated as:
q (u,) uo) = N (ut D, 17: E); E (1. (1,,,,,), ,T)
(1)
Uo V 1HR.
(2)
where uo and u, denote HR frames and intermediate states in the frequency space for brevity. VT denotes the projection matrix of DCT. D, e is diagonal blurring matrix with Aexpty
- + (² ) for coordinate (ær, y) within patch of size x p, and T is the variance of noise. E is the identity matrix.
In the realm of generation, the vanilla destruction process progressively transforms the image into pure Gaussian noise, leading to numerous sampling steps and tending to be suboptimal for VSR. An alternative way is to employ a transition kemee that shifts residuals between HR and LR frames, accompanied by patch-level blurring. The forward diffusion process is formulated as:
q (ue) Uo u)) N (u: Duu + The, R² t1,
(3)
e = .n D,u,,
where u, denotes LR frames transformed into the frequency space. e; indicates the residuals between LR and blurred HR frames at time step t. T} represents the shifting sequence and is hyper- parameter determining the intensity of noise. Upon this, SeeClear can yield HR frames by estimating the posterior distribution p(ug) us) in the reverse sampling progress, formulated
f (2u un) II pe (ut-1 uee uz) du =f
(uo ui) =
(5)
t=1
puur ux) Ro N (ut uy uu, E),
where pe (u--1 Ur w) represents the inverse transition kernel restoring u, u:-1- denotes
learnable parameters of attention-based U-Net.
To alleviate the computational overhead, preceding methods introduce autoencoder to transform pixel-level images in the perceptually equivalent space, concentrating on the semantic composition and bypassing the impedance of fighfreeuency details. However, the loss of high fighffrequency information during encoding is hard to recover in the decoding and will deteriorate the visual quality. Therefore, forgo the autoencoding method in SeeClear and incorporate discrete wavelet transform (DWT) in the diffusion process. Specifically, the HR and LR frames are recursively decomposed into four bub-bands:
H I
HR HR
DWT2D RI!
where TH R denotes the low-frequency approximation, THR JHR and Th correspond to horizontal,
vertical and diagonal high-frequency details. DWT22 (-) represents the 2D Discrete Wavelet Trans- form (DWT). After & decompositions, each of them possesses size of H w These coefficients 24 x 22.
are contacted along the channel dimension and serve in the diffusion process. The rationale behind
employing DWT as substitute is two-fold. Firstly, it enables the U-Net to perform on small spatial size without information loss. Secondly, it benefits from the U-Net scaling (4 hindered by additional parameters of the autoencoding network. To make full use of DWT, window-ban -baee self-attention followed by channel- wise self-attention stacked as the basic unit of U-Net, in charge of the correlation of intra-sub-bane bands and nuter-sub-badds. respectively.

---

P
Ft
Multi-Frame Self-Artention
P
Multi-Head Cross-Attcntion
Oc
Enc
Dec
Figure 2: The illustration of Destance-Centric Alignment Module (InCAM). It utilizes the segmenta- tion features to bridge the pixel-level information and instance-centric semantic tokens. And then, the semantic-aware features can be aligned the semantic space based on their semantic relevance.
3.2 Instance-Centric Alignment within Video Clips
Due to the destruction of the diffusion process, pixel-level inter-frame alignment, such as optical flow, is no longer applicable. With the premise of semantic embedding, we devise the Instance- Centric Alignment Module (InCAM) within video clips, as illustrated Figure 2 It establishes temporal association in the semantic space instead of intensity similarity among frames, avoiding the interference of noise and blurriness. Specifically, Semantic Distiller predicts a set of image features Fing and text embedding Fizt from LR frames and predefined vocabulary V. After that, the & image features with the highest similarity to the text embedding are retained, including token-level semantic priors and pixel-level segmentation features from LR frames. The above procedure is formulated as:
F Fang: Fius = S (T..R. V), O,, = tops. (Sim (Fing: Fint)),
(8)
(9)
where (-) represents Semantic Distiller. tops (.) and Sim (-) denote the operations of selecting the k largest items and calculating the similarity respectively. O; and P: are semantic tokens and segmentation features, in which the former represents high-level semantics and can locate related pixels in the segmentation features. The segmentation features contain both semantics and low- level structural information, which is suitable for bridging the semantics and features the Pixel Condenser. It is utilized to generate spatial modulation pairs prepared for semantic embedding, formulated as:
(x, 9 (P.),
(10)
F, o)
(11)
where and /3 represent scale and bias for modulation. fa and F; correspond to original and modulated features. G denotes convolutional layers followed by ReL.U activation. "o" represents the Hadamard product. After that, InCAM emheds semantics into modulated features based on multi-head cross-attention, yielding semantic-embeddde features P::
Q = F.WO, K; = K V. O,
(12)
A SoftMax (Q.K. V
(13)
where Q K, and V, denote matrices derived from modulated features and semantic tokens. Wº, W and WV represent the linear projections, and the dimension of projected matri- ces. SoftMax (.) denotes the SoftMax operation. To benefit from the adjacent supporting frames, is necessary to establish semantic associations among frames. The frame-wise semantic tokens further fed into the instance encoder-decoder for communicating semantics along the temporal
5

---

dimension, generating clip-wise semantic tokens. It gathers all the information of the same semantic object within clip and serves as the guide for inter-frame alignment. Specifically, InCAM combines semantic guidance and enhanced features to activate the related pixels across frames and utilizes multi-frame self-attention for parallel alignment and fusion. The above procedure formulated as:
Oc = =Dec EEcc (0.).
(14)
R; = MFSA (O..A.) R:).
(15)
where Og and Ô respectively denote clip-wise semantic and randomly initialized tokens. Enc (.) and Dec (-) represent instance encoder and decoder. MFSA (-) denotes the multi-frame self-attention 300 the extended version of self-attention in video. F; is aligned feature. The product of semantics and enhanced features is akin to the class activation mapping, which highlights the most similar pixels in the instance-centric semantic space among frames.
3.3 Channel-wise Aggregation across Video Clips
Due to the limited size of the temporal window, the per- formance of clip-wise mutual information enhancement in long video sequences is unsatisfactory, which could lead inconsistent content. To stabilize video content and enhance visual texture, the Channel-wise Texture Aggregation Memory (CaTeGory) constructed cluster abundant textures according to channel-ve mhineel-wise semantics, as graphically depicted in Figure 3 It is comprised of the channel-wise semantic and the corresponding texture. Specifically, channels of instance-centric semanties also contain distinguishing traits, which are assembled and divided into different groups to form the channel-wise semantic. Concurrently, hierarchical features from the decoder of the Pixel Condenser are clustered into the cor- responding semantic group portray the textures. The connection between them is established in a manner sim- ilar to the position embedding in the attention mechanism. The above process can be formulated as:
of
Chanedwbe Senuntk Aosiiing
Semantic Similarit
Refarm
Pielenn
Multi Elead Cross Atention
Figure 3: The illustration of Channel-wist Texture Aggregation Memory (CaTeGory). It assembles the textures based on the semantic class along the channel dimension.
TT, - I (C,,,,
(16)
where Fi,x is the features benefited from adjacent frames k-th layer. C and Tú respectively denote channel-wise semantics and textures of j-th group, which are zero-initialized as network parameters. They are iteratively updated towards the final version (i.e., Cj and T,) by injecting external knowledge from the whole dataset and previous clips. And M (-) represents the construction of CaTeGory. It concatenates the multi-scale features and incorporates them into channel-wise semantics and textures:
C TT,
(17)
Te SA CA (T) (FiN)
(18)
where 1; is the textures embedded channel-wise semantics. SA (-) and CA (-) indicate multi-head self-attention and cross-altention. The layer normalization and feed-forward network are omitted for brevity. It bridges the channel-wise semantics and textures via matrix multiplication and further fuses high-value information from the pyramid feature, delivering augmented semantic- hierarchical features not only provide rich structural information but also carry relatively abstract amid features in order to benefit different decoder layers more effectively. At each layer, the prior knowledge stored in CaTeGGry is firstly queried by the clip-wise semantics along channel dimension and aggregated for feature enhancement of the current clip, which is formulated as:
pairs. The
A = Max ()) C2),
(19)
F = CA (F, A,T,) + Ft.
(20)

---

where A, depicts the similarity between the clip-wise semantics and items of CaTeGory along channel dimensions, and F; is the refined features input of the next layer. As mentioned before, semantie-texture pairs are optimized as parts of the network during the training stage, absorbing ample priors from the whole dataset. In the sampling process, the update mechanism is reused to integrate the long-term information of video into memory to improve the super-resolution of subsequent clips.
4 Experiments
4.1 Experimental Setup
Datasets To assess the effectiveness of the proposed SeeClear, employ two commonly used datasets for training: REDS [24) and Vimeo-90K [41]. The REDS dataset, characterized by its realistic and dynamic scenes, consists of three subsets used for training and testing. In accordance with the conventions established in previous works [T] 3), we select four clips] from the training dataset to serve validation dataset, referred as REDS4. The Vid4 [19] dataset is used as the corresponding test dataset for Vimeo-90K. The LR sequences are degraded through bicubic downsampling (BI), with downsampling factor of 4x.
Implementation Details The pre-trained OpenSeeD 446 is opted as the semantic distiller with frozen weights, while all learnable parameters are contained in the pixel condenser. And the schedulers of blur and noise in the diffusion process follow the settings of IHDM (25 and ResShift [45]. During the training, the pixel condenser is first trained to generate an HR clip with 5 frames under the control of instance-centric semantics. And then, Channel-wise Texture Memory is independently trained to inject valuable textures from the whole dataset and be capable of fusing long-term information. Finally, the whole network is jointly fine-tuned. All training stages utilize the Adam optimizer with 31 = 0.5 and 32 0.999, where the learning rate decays with the cosine annealing scheme. The Charbonnier loss is applied on the whole frames between the ground truth and the reconstructed frame, formulated as L = v VIII. 12 - e², The SeeClear framework is implemented with PyTorch-2.0 and trained across 4 NVIDÍA 4090 GPUs, each accommodating video clips.
Evaluation Metries Comparative analysis is conducted among different VSR methods, with the evaluation being anchored on both pixel-based and perception-oriented metrics. Peak Signal-to- Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are utilized to evaluate the quantitative performance as pixel-based metrics. All of them are calculated based on the Y-channel, with the exception of the REDS4, for which the RGB-channel is used. On the perceptual side, Learned Perceptual Image Patch Similarity (LPIPS) 77 elected for assessment from the perspective of human visual preference, It leverages a VGG model to extract features from the generated HR video and the ground truth, subsequently measuring the extent of similarity between these features.
4.2 Comparisons with State-of-the-Art Methods
We compare SeeClear with several state-of-the-arr methods, including regression-based and diffusion- based ones. As shown in Table 3 SeeClear achieves superior perceptual quality compared to regression-based methods despite slightly underperforming in pixel-based metrics. We also provide an extended version, which leverages the generative capability of SeeClear to enhance the features of the engression-based model, akin to references such as [6] 2) An observable increase in fidelity is accompanied by a notable further improvement in the perceptual metrics of the reconstructed results. Similar performance trends can be noted Vid4 as those on REDS4. In particular, SeeClear achieves an LPIPS score of 0.1548, marking a relative improvement of 10.8% compared to the top competitor, SATeCo (6. When pitted against variant of SATeCo, which is not modulated by LR videos, SeeClear demonstrates a higher PSNR value with a comparable LPIPS score. It suggests that SeeClear benefits from the control of dual semantics, striking balance between superior fidelity and the generation of realistic textures.
As visualized in Figure 4 SeeClear showcases its ability to restore textures with high fidelity more effectively compared with other methods. Despite large blurriness, SeeClear still demonstrates robust restoration capabilities for video super-resolution, reinforcing the efficacy of utilizing instance- specific and channel-wine wise semantic priors for video generation control. To further substantiate
Cllp 000, 011, 015, 020
7

---

Table Performance comparisons in terms of pixel-based based (PSNR and SSIM) and perception-orieme oriented (LPIPS) evaluation metries on the REDS4 (24] and Vid4 [99 datasets. The extended version of
SeeClear marked with x. Red indicates the best, and blue indicates the runner-up performance
(best view in color) in each group of experiments.
REDS4 [24
Vid4 [9]
Methods
Frames
PSNR
SSIM 1
LPIPS
PSNR
SSIM T
LPIPS &
Bicubic
26. 14
0.7292
0.3519
23.78
0.6347
0.3947
TOFlow 41
7
29.98
0.7990
0.3104
25.89
0.7651
0.3386
EDVR-M 37
5
30.53
0.8699
0.2312
27.10
0.8186
0,2898
BSs
15
31.42
0.8909
0.2023
27.24
0.8251
0.2811
VRT 16
6
31.60
0.8888
0.2077
27.93
0.8425
0.2723
IconVSR 1
15
31.67
0.8948
0.1939
27.39
0.8279
0.2739
StableSR 36
1
24.79
0.6897
0,2412
22.18
0.5904
0,3670
ResShift 45
1
27.76
0.8013
0.2346
24.75
0.7040
0.3166
SATeCo 6
6
31.62
0.8932
0.1735
27.44
0.8420
0.2291
SeeClear (Ours)
5
28.92
0.8279
0.1843
25.63
0.7605
0.2573
SeeClear" (Ours)
5
31.32
0.8856
0.1548
27.80
0.8404
0.2054
Ricubic
EDVR
BasicVSR I
Clip
16
SeeClear" (Ours)
JANAMYY
JUJATYY
Bicubic
EDVR 17
BSsicVSR I
JANUARY
JANUARY
JANUARY
30
30
5
Clip Calendar, Vid4
VRT
SceClear" (Ours)
Figure Qualitative results on the REDS4 and Vid4 datasets. SeeClear generates clearer content and sharper textures.
the temporal coherence acquired by SeeClear, we also visualize two consecutive frames from the generated HR videos, constructed using different diffusion based VSR methodologies, as depicted in Figure 5 ResShift synthesizes varied visual contents across two frames, such as the fluctuating figures on the license plate. Contrarily. HR frames generated via SeeClear maintain a higher temporal consistency and deliver pleasing textures.
4.3 Ablation Study
To assess the contribution of each component within the proposed SeeClear, we begin with baseline model and gradually integrate these modules. Specifically, all semantic-related operations are bypassed, retaining solely spatial and channel self-attention and residual blocks, degenerating into diffusion-based image SR model without any condition. Subsequently, we incrementally introduce the crafted semantie-conditioaal module into the baseline and formulate several variants. Their results are listed in Table 2 and partially visualized in Figure 6

---

Table 2: Performance comparisons on REDS4 among variants with different semantic-condition
control by integrating InCAM and CaTeGory.
Baseline DWT
Semantic
MFSA InCAM CaTeGory
PSNR
SSIM
LPIPS
28.05
0.7993
0.2120
28.08
0.7998
0.2088
3
V
1
27.99
0.7961
0.2053
V
V
V
28.46
0.8098
0.1917
5
V
V
V
28.21
0.7986
0.2149
28.74
0.8267
0.1938
V
V
28.92
0.8279
0.1843
45866
45880
Frame Clip 010, REDS4
Frame 77, Clip REDS4
(8)
Figure 5: Qualitative comparison of regions between consecutive frames. (a) and (b) are patches produced by ResShift [45), derived from Frames 76 and 77 respectively. (c) and (d) display the corresponding regions generated through SeeClear.
First, the intra-frame semantic condition brings about 1.5% improvements in LPIPS. Albeit the multi-frame self-attention further improves the perceptual quality, also impairs the fidelity of the restored video. Under the control of InCAM, SeeClear can correlate semantically consistent pixels in adjacent frames by combining intra- frame and the PSNR from 27.99 dB to 28.46 dB, and bring- ing about 6.6% improvements in LPIPS. Further- more, upon integrating the semantic priors from CaTeGory, the fully-fledged SeeClear notably enhances both the rixel-based and perception- oriented metrics simultaneously. It indicates that the cooperative control of semantics is more beneficial for generating videos of higher fidelity and better perceptual quality. As illustrated in Figure 6 the baseline struggles to restore tiny and fine patterns without semantic condition, and gradually gains improvement accompanied by the strengthening of semantic control.
Model #1
Model #4
semantic priors, elevating
Clip 020, REDS4
Model #6
IIR
Figure 6: Visual comparisons of ablation for investigat- ing the contribution of key modules.
Conclusion
In this work, present a novel diffusion- -aased video super-resolution framework named SeeClear. formulates the diffusion process by incorporating residual shifting mechanism and patch-level blurring, constructing Markov chain initiated with high-resolution frames and terminated at low- resolution frames. It employs a semantic distiller and a pixel condenser for super-resolution during the inverse sampling process. The instance-centric semantics distilled by the semantic distiller prompts spatial modulation and temporal association in the devised Instance-Centric Alignment Module. They are further assembled into Channel-wiss Texture Aggregation Memory, providing abundant conditions for temporal coherence and realistic content.
Acknowledgment. This work is supported in part by the National Natural Science Foundation of China under Grant 62120906099 and Grant 62372036: and part by the National Key Research and Development Program of China under Grant 2022ZD011800) and Grant 202 12100.

---

References
[1] Kelvin C. K. Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. BasicVSR: The search for essential components in video super-resolution and beyond. In CVPR, 2021.
[2] Kelvin C. K. Chan, Xiangyu Xu, Xintao Wang, Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for image super-resolution and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3154 155-3386,,3 2022.
[3] Kelvin C. K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. BasicVSR++: Improving video super-resolution with enhanced propagation and alignment. In CVPR, 2022.
[4] Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half- quadratic regularization algorithms for computed imaging. In ICIP, 1994.
[5] Chaofeng Chen, Shangehen Zhou, Liang Liao, Haoning Wu, Wenxiu Sun, Qiong Yan, and Weisi Lin. Iterative token evaluation and refinement for real- world super-resolution. AAAI, 2024.
[6] Zhikai Chen, Fuchen Long, Zhaofan Qiu, Ting Yao, Wengang Zhou, Jiebo Luo, and Tao Mei. Learning spatial adaptation and temporal coherence in diffusion models for video sespr-resolution. In CVPR, 2024,
[7] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(5):5577-2881, 2020.
[8] Shuting Dong, Feng La, Zhe Wu, and Chun Yuan. DFVSR: Directional frequency video super-resolution via asymmetric and enhancement alignment network. In ICAAI, 2023.
[9] Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. In CVPR, 2021.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[11] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In ICLR, 2022.
[12] Siyu Jiao, Yunchao Wei, Yaowei Wang. Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zcro-shot segmentation. In NeurlPS, 2023.
[13] Shuo Jin, Meiqin Liu, Chao Yao, Chunyu Lin, and Yao Zhao Kernel dimension matters: To activate available kernels for real-time video super-resolution. In ACM MM, 2023.
[14] Bingchen Li, Xin Li, Hanxin Zhu, Yeying Jin, Ruoyu Feng. Zhizheng Zhang, and Zhibo Chen. ScD: Semantic-aware discriminator for image super-resolution. In CVPR, 2024.
[15] Feng Li, Yixuan Wu, Huihui Bai, Weisi Lin, Runmin Cong, and Yao Zhao. Learning detail-structure alternative optimization for blind super-resolution. IEEE Trunsactions on Multimedia, 25:2825-2838, 2022.
[16] Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. VRT: A video restoration transformer. IEEE Transactions on Image Processing, 33:2171-2188 2182, 2024.
[17] Jingyun Liang, Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy 1lg, Simon Green, Jiezhang Cao, Kai Zhang. Radu Timofte, and Lue Van Gool. Recurrent video restoration transformer with guided deformable attention. In NeurlPS, 2022.
[18] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, and Chao Dong. DiABBIR Towards blind image restoration with generative diffusion prior. In ECCV, 2024.
[19] Ce Liu and Deqing Sun. A bayesian approach to adaptive video super resolution. In CVPR, 2011.
[20] Meiqin Liu, Shuo Jin, Chao Yao, Chunyu Lin, and Yao Zhao. Temporal consistency learning of inter- frames for video super-resolution. IEEE Transactions Circuits and Systems for Video Technology. 33(4):1507-1155 15200, 2023.
[21] Meiqin Liu. Chenming Xu, Chao Yao, Chunyu Lin, and Yao Zhao. JNMR: Joint non-linear motion regression for video frame interpolation. IEEE Transactions on Image Processing, 01.5288-5295, 2023.
[22] Anish Mittll, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209 202-222,22122222222 2012.
10

---

[23] Brian Moser. Stanislav Frolov, Federico Raue, Sebastian Palacio, and Andreas Dengel. Waving goodbye to low-res: A diffusion-wavelee approach for image super-resolution. In IJCNN, 2024.
[24] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In CVPRW, 2019.
[25] Severi Rissanen, Markus Heinonen, and Amno Solin. Generative modelling with inverse heat dissipation. In ICLR, 2022.
[26] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[27] Claudio Rota, Mareo Buzzelli, and Joost van de Weijer. Enhancing perceptual quality in video super- resolution through temporally consistent detail synthesis using diffusion models. In ECCV, 2024.
[28] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713-4725 2022.
[29] Shuyao Shang. Zhengyang Shan, Guangxing Liu, LunQian Wang, XingHua Wang. Zekai Zhang, and Jinglin Zhang. ResDiff: Combining cnn and diffusion model for image super-resolution. AAAI, 2024.
[30] Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang. Yujuu Yang, and Chao Dong. Rethinking alignment in video super-resolution transformers. In NewrlPS, 2022.
[31] Yang Song, Conor Durkan, lain Murray, and Stefamo Ermon, Maximum likelihood training of score-based diffusion models. In NeurIPS, 2021.
[32] Haoze Sun, Wenbo Li, Jianzhuang Liu, Haoyu Chen, Renjing Pei, Xueyi Zou, Youliang Yan, and Yujiu Yang. CoSeR: Bridging image and language for cognitive super-resolution. In CVPR, 2024.
[33] Lingchen Sun. Rongyuan Wu, Zhenggiang Zhang, Hongwei Yong, and Lei Zhang. Improving the stability of diffusion models for content consistent super-resolution. arXiv preprint arXiv:2401.00 00877, 2023.
[34] Qi Tang, Yao Zhao, Meiqin Liu. Jian Jin, and Chao Yao. Semantic Lens: Instance-centric semantic alignment for video super-resolution. In AAAI, 2024.
[35] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
[36] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, pages 1-21. 2024.
[37] Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, and Chen Change Loy. EDVR: Video restoration with enhanced deformable convolutional networks. In CVPRW, 2019.
[38] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture image super- resolution by deep spatial feature transform. In CVPR, 2018
[39] Rongyuan Wu, Tao Yang. Lingchen Sun, Zhengqiang Zhang. Shuai Li, and Lei Zhang. SeeSR: Towards semantics-wware real- world image super-resolution. In CVPR, 2024.
[40] Yuhui Wu, Chen Pan, Guoqing Wang, Yang Yang, Jiwei Wei, Chongyi Li, and Heng Tao Shen. Learning semantic-aware knowledge guidance for low-light image enhancement. In CVPR, 2023.
[41] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 27(8):1106-111 2019.
[42] Tao Yang. Rongyuan Wu, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. In ECCV, 2024.
[43] Xi Yang, Chenhang He, Jianqi Ma, and Lei Zhang. Motion-guided latent diffusion for temporally consistent real-world vidco super-resolution. In ECCV, 2024.
[44] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong Scaling up to excellence: Practicing model scaling for photo- meatiste image restoration in the wild. In CVPR, 2024.
11

---

[45] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. ResShift: Efficient diffusion model for image super-resolution by residual shifting, In NewrIPS, 2023.
[46] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang, A simple framework for open-vocabulary segmentation and detection. In CVPR, 2023.
[47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[48] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang. Wangmeng Zuo, and Qi Tian. ControlViden: Training-free free controllable text-to-video generation. In ICLR, 2024.
[49] Shangehen Zhou, Peiqing Yang, Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscale-A-Viden Temporal-consistent diffusion model for real-world video super-resolution In CVPR, 2024.
[50] Hongguang Zhu, Yunchao Wei, Xiaodan Liang. Chunjic Zhang, and Yao Zhao. CTP: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation. In ICCV, 2023.
12

---

A Mathematical Details
Patch-level Blurring Diffusion. Blurring Diffusion 11 designs the destruction process based on
heat dissipation (25) that retains low-frequency components of images over high frequencies. describes the thermodynamic process in which the temperature w(x, y, 8) changes at position (x, y) in 2D plane with respect to the time t via a partial differential equation:
u (x, = Au (r,u,t), E {1, ,T)
(21)
Ot where A 2 denotes the Laplace operator. Based on Neumann boundary conditions (Qu/Oz 0u/00 /yy- = 0) with zero-derivatives at the boundaries of the image, the solution is given by a diagonal matrix in the frequency domain of the discrete cosine transform:
U1 = e uo I E (1,
(22)
where up is the initial state and is a diagonal matrix with negative squared frequencies on the diagonal. It is equivalent to convolution with Gaussian kernel with variance T = 2/ in the realm of image processing. Blurring Diffusion further mixes Gaussian noise with variance of into the blurring process to incorporate stochasticity into deterministic dissipation.
Patch-level blurring diffusion conducts the blurring process within a patch with the size of p x to
make all pixel intensity the same instead of the whole image, which means Azxpty 2 f + L2) More generally, it can be extended with any invertible transformation, corresponding the marginal distribution in the pixel space:
a ( R) = N (7; Rdiag (ox.) R T,"" R diag (8.) R R (1, (1......) (23)
where R and R-1 1 represent invertible transformation and corresponding inverse transformation. diag (-) denotes the operation that projects a vector to diagonal matrix. Or and B, are specific schedules for mean and noise.
Blurring ResShift. ResShift 45 introduces a monotonically increasing sequence {n- to gradually shift residual between low-resolution image and high-resolution one, whose marginal distribution is defined as:
= 4 (1. (2 (4; meo, = E), I (1,
(24)
where en is the residuals between low-resolution and high-resolution frames. controls the speed of residual shifting and satisfies / - and v/r + 1. After incorporation of patch-level blurring diffusion and ResShift, un can be sampled via
U E uno mh + KVTTEEE + un (1 - 11) e un + mp u + KY/TIEL
Nuo
(25)
uy Diu + = u e un.
(26)
where E ~ N (w 0, E). Thus, the relation between Ue and u: can be obtained:
U1-1 111-181
ûo (1 - 1--111)eA((-)
(27)
U, Qt A (u - 12-1 + n + Kvaase:
(28)
Th-1
where a; = -t-1 dg is approximate HR frame. By recursively applying the sampling procedure
and reparameterizatioo trick, can rewrite the marginal distribution of Blurring ResShift follows:
q quer Uo u:) N (uz Druo + mhe, x² E), te (1. ,T},
(29)
According to Bayes's 's theorem, there is
(ut-1 24,, uo. ui) x q (2t u4-1, ui) 9 (2: uo uz),
(30)
where
q (u,) uc-1, ui) =N (mI (u 1- 1/-1 1-
1- mt A (1h_1 12) + ne4,, x²2 ax
(31)
(1:-1 no, uz) = N (u:11
(u-1 | (1 1)-1) AlL-1) + 1/-- 1 k22 x2 E
(32)
13

---

20th
Pewe brakt
12h-
1HDM
ResShin
woallllle Trpg Frogundio
Comparative Power Spectral Density (PSD) analysis
LR
Patch-Lovel Blurring
SocClear
Figure 4: Visual comparison of intermediate state at time step via different diffusion processes.
And then, it only needs focus on the quadratic term in the exponent of q (ut-1 Ue Wds u1):
1- ^ (1 77--111) ui) 21 u
1-
e (u4-1 1--1 12)) - 17c 17:24)
u1 1-7--1
1-7--1
2x2cx.
(u:-1 1)t-1) ((11-1))
T1 11-121)
(14-1 - (1 1/1 17--1) cA(c-1) - 111 (0--144)
2x 11-1
(1-1))2
U: +
[ T 1- ea 11
[1 [--)--]
Ut u
- 1/t c
2
1²as a1
16211-11
97-1
k2au
(1 17p 1) c A(t 1) 1)20 17-121
+
u const
k2111-1
(u,11 11) (u,-1 11²
+ const,
20²
(33)
where
App + 04 (1 1B. 1) A(1-1) u0 + (121)=1 App Amp- 17h -1()--) 4
1)
(34)
1211-11 + +0t
0² 1² k2a01h111
(35)
X1)-11 1 + Ot.
2 -
(36)
78-1
where 'const' denotes the item that is independent of u
B Rational Explanation
We analyze the final states of different diffusion processes via the power spectral density, which reflects the distribution of frequency content in an image, illustrated in Figure 4 It can be observed frequency distribution
that IHDM performs blurring globally and has significant difference compared to the LR image, while the patch-level blurring is closer to the frequency distribution of LR. On this basis, SeeClear further introduces residual and noise. Compared to ResShift without blurring, the diffusion process adopted by SeeClear makes the image more consistent with the LR in the frequency components and introduces more randomness the high-frequency components, compelling the model to focus on the generation of high- high-ffrequncy components.
14

---

C Network Structure
Modale
Apprcpation
Cunder
Perrge Qarry
Sdlctios
15
Pluat Contuner
(-e-dea
Baat 15
Pioel Candemer
Dittller
24/05
Figure& 8: The illustration of SeeClear. It comprises the diffusion process incorporating patch-level blurring and residual shift mechanism and a reverse process. During the reverse process, Semantic Distiller for semantic embedding extraction and U-shaped Pixel Condenser are employed for iterative denoising. The devised InCAM and CaTeGory are inserted into the U-Net to utilize the diverse semantics for
alignment in the diffusion-based VSR framework.
SeeClear consists of a forward diffusion process and a reverse process for VSR. In the diffusion process, patch-level blurring and residual shift mechanism are integrated to degrade HR frames based on the handcrafted time schedule. During the reverse process, transformer- for open vocabulary segmentation and a U-Net are employed for iterative denoising. The former is responsible for extracting semantic embeddings related to instances from LR videos, similar to the process of distillation in physics, and is therefore named the semantic distiller. The latter is utilized to filter out interfering noise and retain valuable information from low-quality frames, similar to the condensation process. All of them are tailored for image processing, and SceClear takes diverse semantic embeddings conditions enable the network to be aware of the generated content and determine the aligned pixels from adjacent frames for the temporal consistency of the whole video.
network
As depicted in Figure 8] the attention based Pixel Condenser primarily consists of three parts, i.e. encoder, decoder, and middle block. As the input of the encoder, low-resolution frames are concatenated with the intermediate states and processed through a convolution layer. The encoder incorporates a window-based self-attention and channel-wise self-attention, alternating between two residual blocks. These attention operations mine valuable information within and across the wavelet sub-bands, and the residual blocks infuse the features with the intensity of degradation. Following this, wavelet-based downsample layer is deployed for feature downsampling
Specifically, the features are decomposed into various sub-bands via 2D discrete wavelet transform, reducing the spatial dimensions while keeping the original data intact. Low- Low-frequency features are further fed into the subsequent layer of the encoder, while others are transmitted to the corresponding wavelet-based upsample layer in the decoder via a skip connection. Additionally, the wavelet-based downsample layers are utilized parallel along the encoder, refilling the downsampled features with information derived from the low-resolution frames.
The devised Destance-Centrie Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) are inserted into both the middle and decoder. Firstly, the InCAM spatially modulates the features based on instance-centric semantics and aligns adjacent frames within the clip in semantic space. Spatial self- employs these aligned features, substituting the original features as input. Subsequently, features enhanced through channel-wise self-attention are embedded as queries to seek assistance from the CaTeGory. Furthermore, the wavelet-based upsample layer accepts features from the decoder and figghffre information transmitted from the encoder. reinfuses the lost information from the encoder while scaling the feature size. The network concludes with convolution layer refining the features, which are added to the interpolated frames to generate the final output.
15

---

Table Performance comparisons terms of Full- full-Reference IQA (DISTS and No-Reference
IQA (NIQE [22] and CLIP-IQA (B5)) evaluation metrics on the REDS4 24 and Vid4 [19] datasets.
The extended version of SeeClear marked with x. Red indicates the best, and blue indicates the
runner-up performance (best view in color) in each group of experiments.
REDS4 [24
Vid4 [1]]
Methods
Frames
DISTS +
NIQE +
CLIP-IQA
DISTS
NIQE
CLIP-IQA 1
Bicubic
0.1876
7.257
0.6045
0.2201
7.536
0.6817
TOFlow
41
7
0.1468
6,260
0.6176
0.1776
7.229
0.7356
EDVR-M 37
0.0943
4.544
0.6382
0.1468
5,528
0.7380
Basic VSR
15
0.0808
4.197
0.6353
0.1442
5.340
0.7410
VRT [6
0.0823
4.252
0.6379
0.1372
5.242
0.7434
ISR
15
0.0762
4.117
0.6162
0.1406
5.392
0.7411
StableSR [36
0,0755
4.116
0.6579
0.1385
5.237
0.7644
ResShift 45
0.1432
6.391
0.6711
0.1716
6.868
0.7157
SATeCo 6
0.0607
4.104
0.6622
0.1015
5.212
0.7451
SeeClear (Ours)
0.0762
4.381
0.6870
0.0947
5.305
0.7106
SeeClcar" (Ours)
5
0.0641
3.757
0.6848
0.0919
4.896
0.7303
NOW 7563
20N 7863
20 7863
7EA3
Frames
Bicubic
EDVR AY
BasicVSR 11
VRT ITC
SceClear
Figure 9: Visual examples of video super-resolution results by different approaches on the REDS4 and Vid4 datasets. The region in the red box is presented in the zoom-in view for comparison.
D Additional Experiments
We provide additional evaluation metrics and visual comparisons contrasting the existing VSR methods with proposed SceClear. As demonstrated in Figure our proposed method successfully generates pleasing images without priors pre-trained on large-scale image datasets, showcasing sharp edges and clear details, evident in vertical bar patterns of windows and characters on license plates. Conversely, prevalent methods seemingly struggle, causing texture distortion or detail loss in analogous scenes.
also execute several experiments focused on the degradation scheme of the diffusion process, verifying the performance of the models the first frame, as indicated in Table Compared to the straightforward diffusion process of simply adding Gaussian noise into HR frames, the combination of blurring diffusion proves beneficial in generating the high- fighffrequency details discarded in LR
16

---

Table 4: Performance comparisons on the first frame of REDS4 among variants with different deterioration during the diffusion process and U-Net.
Noise
Model
PSNR t
SSIM 1
LPIPS +
1
ResShift
0
WaveDiff
26.78
0.7960
0.2096
2
ResShift
2
WaveDiff
26.45
0.7927
0.2008
3
ResShift
3
WaveDiff
27.74
0.8047
0.2067
4
ResShift
0
SwinUNet
27.76
0,8013
0.2346
5
ResShift
2
SeeClear 28.04
0.8134
0.1971
frames. Specifically, the variation in the intensity blur (Line 1-3) affects the fidelity and perceptual quality. Among them, there no blurring when o 0, and the greater the value of o the greater the blurring intensity. can be observed there is a 0.96 dB improvement in PSNR and the value of LPIPS ranges from 0.2096 to 0.2067 with the increasement of ø.
On another note, implementing attention mechanisms in the wavelet spectrum proves more successful in uncovering valuable insights than in the pixel domain. Additionally, SeeClear introduces the alternation of spatial self-attention and channel self-attention, refining the modeling of intra and inter-frequeng sub-band correlations and remarkably enhancing the quality of the generated high- resolution frames.
E Complexity Analysis
3 compares the efficiency between our proposed method and diffusion-based meth- ods. It presents the number of parame- ters of different models and their inference time for super-resolving 512 x 512 frames from 128 128 inputs. Combining these comparative results, draw the follow- ing conclusions: i) Compared to semantic- assisted single-image super-resolution (e.g., CoSeR [32] and SeeSR (39),, our proposed method possesses fewer parameters and higher inference efficiency. ii) In contrast to existing diffusion-based methodologies for VSR [27 431, SeeClear much smaller and runs faster, benefiting from the reasonable module de- signs and diffusion process combing patch- level blurring and residual shift mechanism.
Table 5: Complexity comparisons between different
diffusion-baa baseed super-resolution methods.
Time
Infer
Methods
Params
Steps
Time (s)
LDM [26]
169.0 M
200
5.21
DiffBIR 18
1716.7M
50
5.85
ResShift 45
173.9M
15
1.12
SISR StableSR
336
1409.1M
200
18.70
CoSeR [32
2655.52M
200
SeeSR 139
2283.7M
50
7.24
PASD 42
1900.4M
20
6.07
StableVSR 27
712M
50
28.57
VSR MGLD-VSR
45
1.5B
50
1.113
SeeClear (Ours)
229.23M
15
1.142
Generation Process
video clip consisting five frames is parallelly sampled during the inference process. These LR frames first fed into the semantic distiller extract semantic tokens and then corrupted by random noise as the input of the pixel condenser. The pixel condenser iteratively generates the corresponding HR counterparts from noisy LR frames under the condition of LR frames and semantic priors. The pseudo-code of the inference depicted in the Algorithm
G Limitations and Societal Impacts
Limited by the size and diversity of the dataset, SeeClear, being solely trained on video super- resolution datasets, does not fully leverage the generative capabilities of diffusion models as efficiently as those that benefit from pre-training capable of generating sharp textures and maintaining consistent details, it falls short in restoring tiny objects intricate structures with complete satisfaction.
large-scale image datasets such as ImageNet. While SeeClear
17

---

// k refers to 4th layer of decoder D Algorithm 1: Generation Process of SleeCear
Input: LR frames ILR E RRNxCxHxW, time steps T; and predefined schedule {a, 7) Output: SR frames n
1 IRNxXxsHXWW
TLR (
un ( FE,E tGEEN((0,
3
Parallel Generation within Local Video Clip
for m = 1 to N do
// M refers to the number of frames in a video clip
for t= == M
to 0 do
/= then
On Pn = S(T.R, V)
u,, un
else
" = DWT 2n 11+11)
10
11
skipH = [] // Array of high-frequency spectrums for skip connection
12
for i to 4 do
// i refers to ith layer of encoder
13
if i=/ then
f= [I u',]]
15
else
16
f 1-1 = )
// WD denotes Wavelet- based DownSample
fn t
17
18
if #4 4 then
L Hi WD(fi) Jh L'n skipH skipH H else
21
f1 fm
23
On Dec(Enc((O,) Ô)
24
for j 10 do
// j refers to jth layer of middle blocks
26
N/. =(Pm))
f/ -f B fī-1
27
28
fú = CA(FL, Om 0m)
f4 (MFSSA((O. f2)
30
Seals []
// Array of multi-scale features for CaTeGory
for 1 1 to 4 do
32
if k/4 then
33
f WU([A skippllkk))l) // WU denotes Wavelet-based UpSample
b 8hh = G(Pr)
34
35
ft (f%- Onh
2h + ph) + Fk-1
36
8 CA(it,Oc,,
f Da(MFSA(O. Fn))
37
38
S CA(jK Cms Tm)
feats = feats + fk
39
T = IDWT20(F)) (Fm) + n
FLR
40
7 Cs 1 I[+1] + THE
41
42
Update CaTeGory for Global Video Consistency
43
if then
44
Cm+1, Tm+1 M(Cm, Tm, feals)
1 1
45
18

---

In addition, compared to synthetic datasets, videos captured in real-world scenarios display more complex and unknown degradation processes. Although real-world VSR is garnering significant attention, it remains an unexplored treasure and a steep summit to conquer for diffusion-based VSR, including SeeClear.
Significantly, substituting an Autoencoder with traditional wavelet transform lessens the computa- tional burden while ensuring the preservation of the original input information. Nevertheless, in the process of inverse wavelet transform, the spatio-temporal information within videos is not delved into further, leading to subpar outcomes. Meanwhile, some methods develop trainable wavelet- like transformations based on the lifting scheme, allowing for end-to-end training of the whole network. Such a schema presents a promising direction for future research by potentially boosting model performance.
As for societal impacts, similar to other restoration methods, SeeClear may bring privacy concerns after restoring blurry videos and lead to misjudgments used for medical diagnosis and intelligent security systems, etc.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section and of the main paper and Section supplementary material.
A
C and D
of the
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section of the supplemental material.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
See Section of the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section
4 of the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b)
If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
See Section of the main paper. We plan to release the code after the paper is accepted.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section of the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Section of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma eπrоr bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section
4 of the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We confirm that the research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considera- tion due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section G of the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original paper that produced the code package and dataset used.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. ctme./cooaaaaoaaeeee dottaattssss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We plan to gradually open-source after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "P98EqfWz2c", "reviewer": "Reviewer_NXC1", "summary": "The paper presents a framework for video super-resolution (VSR) that improves temporal coherence and high-resolution detail generation. The proposed method, SeeClear, integrates a Semantic Distiller and a Pixel Condenser to extract and upscale semantic details from low-resolution frames. The framework employs an Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) to enhance inter-frame coherence and incorporate long-standing semantic textures. The methodology also introduces a blurring diffusion process with the ResShift mechanism to balance sharpness and diffusion effects. Experimental results show that SeeClear outperforms state-of-the-art diffusion-based VSR techniques in terms of perceptual quality and temporal consistency.", "questions": "1. The performance of the proposed method is not significant. In Table 1, the improvement is very marginal or is worse than other methods. Moreover, in Figure 4, the generated texture is comparable to other methods.\n\n2. It would be better to compare more methods. (a) Transformer-based (e.g., VSR Transformer) or RNN-based method (e.g., BasicVSR++); (b) Diffusion-based image restoration methods (e.g., DDRM, DDNM, DeqIR, etc). (c) Compare methods trained with more frames (e.g., VRT-16).\n\n3. The authors should compare the efficiency, including model size, training/inference time and FLOPs. The efficiency comparisons can demonstrate the effectiveness of the proposed method.", "limitations": "Please refer to the details above.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The SeeClear framework introduces a combination of semantic distillation and pixel condensation, which significantly enhances video super-resolution.\n2. The Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) improve the temporal coherence of the generated high-resolution videos.\n3. The integration of blurring diffusion with the ResShift mechanism effectively balances sharpness and diffusion, leading to high-quality detail generation.", "weaknesses": "1. While the method demonstrates robust restoration capabilities, it may still struggle with accurately restoring tiny objects or intricate structures, especially under severe degradation conditions.\n2. The method has been tested primarily on specific benchmark datasets. Its performance in real-world applications, where video degradation processes are more varied and unpredictable, remains to be thoroughly evaluated.\n3. The experiments are not sufficient and should be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The performance of the proposed method is not significant. In Table 1, the improvement is very marginal or is worse than other methods. Moreover, in Figure 4, the generated texture is comparable to other methods.\n\nFor the sake of fair comparison, SeeClear is trained only on five frames and achieves PSNR/SSIM comparable to those models trained on longer sequences, along with the best LPIPS. SeeClear also acquires the superior PSNR on the Vid4 dataset, surpassing the runner-up [1] by **0.36 dB**. It is worth highlighting that the novelty of SeeClear lies in its pioneering exploration of semantic tokens in the activation and association of semantic-related pixels across frames during the inter-frame alignment of diffusion-based VSR, rather than merely the improvement of performance metrics. The instance-centric semantic tokens directly extracted from frames not only stimulate the generation potential of the diffusion model but also avoid the cross-modal alignment between text and images. Besides, the inter-frame related conditional pixels determined by semantic similarity equips the diffusion-based VSR model with cognitive ability along the temporal dimension, reducing the interference of irrelevant pixels and motion estimation errors and enhancing the quality of the reconstructed videos. Moreover, the restoration of tiny objects or intricate structures and real-world VSR demands specific training datasets [2, 3, 4], which are not coincident with the configurations and benchmarks of prevailing VSR. These issues have been thoroughly discussed in **Section D** of the supplementary materials and are considered as our future undertaking.\n\n[1] Zhikai Chen, Fuchen Long, Zhaofan Qiu, Ting Yao, Wengang Zhou, Jiebo Luo, and Tao Mei. Learning spatial adaptation and temporal coherence in diffusion models for video super-resolution. In CVPR, 2024. \n\n[2] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR, 2024. \n\n[3] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023.\n\n[4] Jiezhang Cao, Yue Shi, Kai Zhang, Yulun Zhang, Radu Timofte, and Luc Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In CVPR, 2024.\n\n> Q2. It would be better to compare more methods. (a) Transformer-based (e.g., VSR Transformer) or RNN-based method (e.g., BasicVSR++); (b) Diffusion-based image restoration methods (e.g., DDRM, DDNM, DeqIR, etc). (c) Compare methods trained with more frames (e.g., VRT-16).\n\nWe add more methods for comparison, including RNN-based (BasicVSR++), Transformer-based (RVRT, VRT) and Diffusion-based (DDNM) methods. All these methods except DDNM are trained with more frames. It can be observed that more complex architectures (e.g., second-order grid propagation) and longer sequence used for training can significantly enhance the quality of reconstructed video but inevitably result in substantial computational overload.\n\n|Methods|Frames|PSNR $\\uparrow$|SSIM  $\\uparrow$|LPIPS  $\\downarrow$|\n|:---:|:---:|:---:|:---:|:---:|\n|BasicVSR++|30|32.38|0.9070|0.1462|\n|RVRT|30|32.75|0.9113|0.1410|\n|VRT|16|32.19|0.9005|0.1544|\n|DDNM|1|27.05|0.7660|0.2608|\n\n> Q3. The authors should compare the efficiency, including model size, training/inference time and FLOPs. The efficiency comparisons can demonstrate the effectiveness of the proposed method.\n\nPlease see the answer to Q2 of the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7pjzbCOjNV", "reviewer": "Reviewer_vLVk", "summary": "This paper presents a diffusion-based video super-resolution method, and proposes Instance-Centric Alignment Module and Channel-wise Texture Aggregation Memory. The former leverages a pre-trained open-vocabulary segmentation model (i.e., OpenSeeD), which is utilized to perform alignment within video clips by modulating the spatial and temporal features. The latter leverages channel-wise attention and memory mechnism to better super-resolve the video frames. The results on publich benchmarks indicate that the proposed method achieves state-of-the-art perceptual performance.", "questions": "1. In the abstract, what is the \"conditional video generation\" (L6)? I do not see any pre-trained conditional video generation module in the described method.\n2. In L206-207, what is the role of \"randomly initialized tokens\"? And what is specific role of the encoder-decoder module?\n3. In L187-188, are the \"semantic tokens\" actually text embeddings ? What is the difference?\n4. In L223, how to divide channels into different groups and what is rationale behind it?\n5. It is hard to understand Eq. (16), (17) and (18). From (17) and (18), it seems T_j is used to calculate itself, which is confusing. Please elaborate.\n6. In InCAM, the way of introducing semantic features seems similar to SFT. Please compare with it and illustrate the significance of the proposed module.\n7. Please provide comparison with the following related methods, and illustrate the novelty of the proposed modules. \n(a) The necessity and rationale of blurring ResShift, and the ablation study.\n(b) Upscale-A-Video [36]. And please compare performance with it quantitatively.\n8. In L35-36, I think it is hard to reach the given conclusion from [8]. Please elaborate.", "limitations": "The authors have adequately addressed the limitations in Section D.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves state-of-the-art perceptual results on REDS4 and Vid4.", "weaknesses": "Although the proposed method achieves promising results on the public benchmarks, there are some concerns that greatly affect the rating of this paper.\n\n1. The presentation of the method needs to be improved. The readability of the paper is unsatisfactory. The technical details and the rationale behind it is not clearly described and explained.\n(a) The main figure (Figure 1) is ambiguous. It is hard to understand the workflow of the framework based on this figure. It is also hard to see the connection among different modules.\n(b) In the abstract, what is the \"conditional video generation\" (L6)? I do not see any pre-trained conditional video generation module in the described method. Maybe it should be rephrased.\n(c) In L206-207, what is the role of \"randomly initialized tokens\"? And what is specific role of the encoder-decoder module?\n(d) In L187-188, are the \"semantic tokens\" actually text embeddings ? What is the difference?\n(e) In L223, how to divide channels into different groups and what is rationale behind it?\n(f) It is hard to understand Eq. (16), (17) and (18). From (17) and (18), it seems T_j is used to calculate itself, which is confusing.\n(g) The choice of mathematical notations is sub-optimal and confusing.\n(h) In L149, I think the \"belta_t\" should be \"yita_t\".\n\n2. The novelty of this paper is limited.\n(a) Some of the modules are based on existing methods. For example, the way of introducing semantic features is similar to SFT (but no comparison in the paper); the multi-frame self attention is from [21].\n(b) The proposed blurring ResShift is a modification version based on ResShift, but the rationale behind it is not fully explained. Also, there is no direct ablation.\n\n3. The comparison with other related methods are not thorough. \n(a) The authors should explicitly compare with ResShift [33], since residual shifting technique is also exploited (but no citation in L48). Also, there is no comparison with it in Sec. 2.2.\n(b) The authors should compare with Upscale-A-Video [36], another diffusion-based video super-resolution method. Also, it is recommended to compare the performance of [36].\n(c) The authors should compare with SFT[28], another method also leveraging semantic segmentation information.\n\n4. The proposed method is not fully ablated. There is no direct ablation for exploitation of DWT and blurring resshift.\n\n5. Some of the statements could be inappropriate. \n(a) In L35-36, I think it is hard to reach the given conclusion from [8]. Please elaborate.\n(b) The naming of \"semantic distiller\" could be inappropriate. The pre-trained semantic segmentation model is directly leveraged and frozen. I don't see any distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and detailed comments. We will rectify some confusing statements and formulas in the subsequent edition. Nevertheless, we deem it necessary to highlight our novelty and restate the proposed method. Different from the **text** in the realm of T2I and **segmentation mask** in existing generative super-resolution, we pioneer the exploration of the utilization of temporal information based on the semantic similarity for diffusion-based VSR. The instance-centric semantic embeddings and channel-wise semantics are employed to determine the conditional pixels with high quality and semantic similarity from adjacent frames and long-term temporal information, enhancing the generative quality and temporal consistency.\n\n> Q1. What is the \"conditional video generation\"? \n\nAs depicted in Figure 1 of the rebuttal document, SeeClear consists of a forward diffusion process and a reverse process for VSR. In the diffusion process, patch-level blurring and residual shift mechanism are integrated to degrade HR frames based on the handcrafted time schedule. During the reverse process, a transformer-based network for open vocabulary segmentation and a U-Net are employed for iterative denoising. The former is responsible for extracting semantic embeddings related to instances from LR videos, similar to the process of distillation in physics, and is therefore named the semantic distiller. The latter is utilized to filter out interfering noise and retain valuable information from low-quality frames, similar to the condensation process. All of them are tailored for image processing, and SeeClear takes diverse semantic embeddings as conditions to enable the network to be aware of the generated content and determine the aligned pixels from adjacent frames for the temporal consistency of the whole video. More details of U-Net for denoising are elaborated in **Section B** of the supplementary material.\n\n> Q3 & Q4. Are the \"semantic tokens\" actually text embeddings?  How to divide channels into different groups?\n\nThe conditions in SeeClear refer to instance-centric semantic embeddings, semantics within the channel dimension and discriminative textures, rather than text or segmentation mask. The semantic distiller comprises three components, i.e., backbone, encoder and decoder. The encoder takes multi-scale features from backbone as input and models the relationship among pixels via self-attention. Randomly initialized tokens serve as the input of the decoder together with the dense features yielded by the encoder, which integrate instance-related semantic information through cross-attention. Eventually, those semantic tokens are utilized to predict classes, bounding boxes, and masks by simple modules like MLP. \n\nBesides, we dig abundant semantics within the channel dimension via CaTeGory. For example, different channels response to edges/corners in shallow layer, and deeper layers further extract complex patterns by combining the information of different channels in the previous layer. CaTeGory consists of semantic and texture embeddings, which are zero-initialized as parameters of network. Throughout the training process, semantic embeddings iteratively aggregate the channel semantic from instance-centric semantics via cross-attention, and texture embeddings formulate representative textures from the multi-scale features of the U-Net. To establish the one-to-one correspondence between channel-wise semantics and textures, CaTeGory correlates semantic and texture embeddings through element-wise multiplication.\n\n> Q2 & Q6. What is the role of \"randomly initialized tokens\"? And what is specific role of the encoder-decoder module? In InCAM, the way of introducing semantic features seems similar to SFT.\n\nTo effectively utilize these semantic conditions, InCAM is inserted before the spatial self-attention of U-Net, aiming at performing inter-frame alignment in the semantic space implicitly. Distinct from SFT, which merely generates coefficients for feature modulation based on class priors derived from the segmentation mask, InCAM utilizes two types of semantic priors, i.e., semantic embeddings and dense features. Among them, dense features and those of VSR are jointly employed for the generation of modulation coefficients, enabling the modulated features to lie in the domain between the segmentation and the super-resolution, narrowing the gap between semantic priors and pixel features. Moreover, the semantic embeddings are not only used as conditions for the generation of each frame, but also can locate pixels related to specific instances within the frame. Therefore, InCAM incorporates the instance encoder-decoder (L206-207) to yield clip-wise semantics based on the instance-centric semantics of adjacent frames, which associate the same instances and align the semantically related pixels between adjacent frames. The random initialized tokens input in the decoder will serve as clip-wise semantics after the iterative interaction along temporal axis. From this perspective, InCAM also possesses the ability of inter-frame alignment that SFT lacks.\n\n> Q5. It is hard to understand Eq. (16), (17) and (18). From (17) and (18). Please elaborate.\n\nPlease see the official comment.\n\n> Q7 & Q8. Please provide comparison with the following related methods. I think it is hard to reach the given conclusion from [8]. Please elaborate.\n\nDue to the unavailability of Upscale-A-Video’s code and the relevant datasets (e.g., REDS30), a fair comparison cannot be conducted. But we are willing to carry out the comparison in our subsequent versions once the code and related datasets are released. While SFT is an image-level SR method and lacks quantitative results, thus we add other representative methods in the response to Q2 for Reviewer NXC1. \n\nPlease refer to the answer to Q3 in the global response for the elaboration of the given conclusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZceSyv6zD", "reviewer": "Reviewer_srYj", "summary": "The paper introduces a novel video super-resolution framework leveraging semantic distillation to enhance pixel condensation in diffusion-based models. SeeClear addresses stochastic fluctuations by using a Semantic Distiller and a Pixel Condenser to extract and upscale semantic details from LR frames. The framework includes an Instance-Centric Alignment Module and a Channel-wise Texture Aggregation Memory to improve temporal consistency and visual quality. Experimental results demonstrate SeeClear's superiority over state-of-the-art diffusion-based VSR techniques.", "questions": "Diffusion-based models usually show poor performance on PSNR (e.g., StableSR and Reshift), but SeeClear demonstrates a significant improvement. Could you analyze which parts of SeeClear contribute to this improvement?\nPlease refer to the weaknesses part above.", "limitations": "The authors have largely addressed their limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The combination of semantic distillation and pixel condensation is novel and effectively addresses the challenges of maintaining detail consistency across frames in diffusion-based VSR.\n- The Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) significantly improve short-term and long-term temporal coherence.\n- The paper provides extensive experiments to demonstrate SeeClear's advantages over sotas across multiple benchmarks.", "weaknesses": "- Lack of computation analysis. Diffusion-based methods are often criticized for unbearable inference time, so it would be better to list params, runtime, and FLOPs/MACs for a fair comparison.\n- Lack of an ablation study on the wavelet transform which is introduced in Section 3.1.\n- Table 2 is incomplete, making it difficult to assess the effect of the CaTeGory.\n- The Other baselines such as VRT and IconVSR are also evaluated on Vimeo-90K-T and UDM10 datasets. Could you complete it for a fair comparison?\n- Figure 7 needs more explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Diffusion-based models usually show poor performance on PSNR (e.g., StableSR and Reshift), but SeeClear demonstrates a significant improvement. Could you analyze which parts of SeeClear contribute to this improvement? Please refer to the weaknesses part above.\n\nDiffusion-based image super-resolution acquires remarkable perceptual quality thanks to the generative capabilities of the models, yet the consistency with low-resolution images is somewhat overlooked. However, the highly correlated content within adjacent frames of a video supplies more references and constraints for the generation of individual frames. Coupled with InCAM, which activates and associates semantic-related pixels among adjacent frames, it's possible to enhance the perceptual quality of the reconstructed video and strengthen the constraints on fidelity, as shown in Table 2 of the paper. Similar experimental results can also be observed in the comparative method SATeCo. Besides, the pixel condenser employs wavelet transform to change the resolution of features and continually reinject significant information into downsampled features from LR frames, guaranteeing the content fidelity. And the split high-frequency components of LR frames are transmitted to the decoder via skip connections, guiding the generation of details and textures. Additionally, some works [1, 2] also provide theoretical support for this experimental result. In conclusion, we hold the belief that the constraints of multiple frames of the video and the appropriate module designs jointly constrain the solution space of the model.\n\n[1] Theo Adrai, Guy Ohayon, Michael Elad, and Tomer Michaeli. Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image Restoration. In NeurIPS, 2023. \n\n[2] Dror Freirich, Tomer Michaeli, and Ron Meir. A Theory of the Distortion-Perception Tradeoff in Wasserstein Space. In NeurIPS, 2021.\n\n> Q2. Lack of computation analysis. \n\nPlease see the answer to Q2 of the global response.\n\n> Q3. Lack of an ablation study on the wavelet transform. \n\nPlease see the answer to Q1 of the global response.\n\n> Q4. The Other baselines such as VRT and IconVSR are also evaluated on Vimeo-90K-T and UDM10 datasets. Could you complete it for a fair comparison? \n\nDue to the lack of diffusion-based VSR evaluated onVimeo-90K-T and UDM10 datasets, we can only just provide the quantitative result generated via our proposed method.\n\n|Datasets|\tPSNR  $\\uparrow$|\tSSIM  $\\uparrow$|\tLPIPS  $\\downarrow$|\n|:---:|:---:|:---:|:---:|\n|Vimeo-90K-T (BI)\t|37.64|0.9503|0.0982|\t\t\n|UDM10 (BD)|\t39.72|\t0.9675|\t0.0609|\n\n\n> Q5. Figure 7 needs more explanation.\n\nPlease refer to the answer to the Q3 of the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rBDNY59obz", "reviewer": "Reviewer_ERJ1", "summary": "The authors propose SeeClear for Video Super-Resolution (VSR). SeeClear is a diffusion-based method that improves restoration performance by introducing semantic priors. The authors design an Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) to utilize semantic information effectively. Comparisons on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance.", "questions": "1. Why do the comparison methods in Table 1 use different numbers of frames? If the same frame is used, what is the performance like?\n2. In the ablation study (model 2, Table 2), how to use semantic conditions without MFSA, InCAM, and CaTeGory?\n3. In InCAM, what is the value of k in top k, and how is it determined? Is there experimental support for this choice?\n4. Others see weaknesses.", "limitations": "The authors discuss the method's limitations and societal impacts.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces semantic priors to achieve spatial modulation and temporal correlation, improving diffusion-based VSR performance. This idea is both reasonable and effective.\n2. The authors design the Instance-Centric Alignment Module (InCAM) to align using semantic information, avoiding pixel inconsistencies and being well-suited for diffusion models.\n3. Additionally, the authors propose the Channel-wise Texture Aggregation Memory (CaTeGory) to transfer semantic information between different frames.\n4. Comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed method.\n5. The paper is well-organized, with clear and aesthetically pleasing layouts, figures, and tables.", "weaknesses": "1. The method uses pre-trained models to extract semantic information, introducing significant additional computation, which limits the method's applicability. Meanwhile, the paper lacks comparisons of complexity and parameter counts.\n2. The method lacks experimental support for some critical hyperparameters, such as the choice of k in InCAM and the number of frames used in SR.\n3. The paper proposes using wavelet transform to improve UNet but lacks experimental justification for why simple downsampling and upsampling wouldn't be more efficient.\n4. Figure 1, while aesthetically pleasing, is challenging to understand. It would be better to clearly explain the network structure (e.g., Figure 8) and the inference process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Why do the comparison methods in Table 1 use different numbers of frames? If the same frame is used, what is the performance like?\n\nThe selection of numbers of frames for training depends on the architecture, such as sliding-window-based (e.g., EDVR-M) and recurrent-based (e.g., IconVSR) methods. The longer temporal information the model can access during training, the better the performance could be generally attained [1, 2]. Nevertheless, the increase in the number of frames will also lead to huge computational overhead. Therefore, we propose to parallelly align inter-frame within the short clip and maintain long-term temporal consistency via additional texture memory. \n\n[1] Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang, Yujiu Yang, and Chao Dong. Rethinking Alignment in Video Super-Resolution Transformers. In NeurIPS, 2022. \n\n[2] Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. Investigating Tradeoffs in Real-World Video Super-Resolution. In CVPR, 2022.\n\n> Q2. In the ablation study (model 2, Table 2), how to use semantic conditions without MFSA, InCAM, and CaTeGory?\n\nThe semantic conditions are utilized for the generation of each frame via cross-attention, enabling the SeeClear to be cognizant of the content to be generated. Specifically, the query and key/value are respectively projected from features of SR and instance-centric semantic embedding of segmentation. It embeds the semantic priors into the pixel-level features, similar to the position embedding.\n\n> Q3. In InCAM, what is the value of k in top k, and how is it determined? Is there experimental support for this choice?\n\nOn the one hand, the resolution of the LR frames constrains the value range of k. On the other hand, there is saturation in the choice of k, as shown in Table 3 of the rebuttal document. As long as the instances and backgrounds in the frames can be retrieved completely, a larger k won't lead to performance improvement while increasing the computational cost of the network.  \n\n> Q4. The method uses pre-trained models to extract semantic information, introducing significant additional computation, which limits the method's applicability. Meanwhile, the paper lacks comparisons of complexity and parameter counts.\n\nAlbeit an additional segmentation network is introduced, the auto-encoder is replaced by DWT simultaneously, making it possible to increase the network scale and improve the performance. Additionally, the option of the segmentation network is highly flexible, as long as it is transformer-based architecture. Especially with the continuous development of segmentation tasks in recent years, it is highly beneficial for performance gain of SeeClear. The semantic-assisted generative super-resolution comes forth in recent advanced works [1, 2]. Some of them not only require additional training for the semantic extraction network but also add ControlNet as a condition control mechanism, bringing more parameters. To further demonstrate the effectiveness and efficiency of the SeeClear, we also compare it with other representative diffusion-base SR methods in terms of the number of parameters and inference time, please see Q2 of the global response for more information.\n\n[1] Haoze Sun, Wenbo Li, Jianzhuang Liu, Haoyu Chen, Renjing Pei, Xueyi Zou, Youliang Yan, and Yujiu Yang. CoSeR: Bridging Image and Language for Cognitive Super-Resolution. In CVPR, 2024. \n\n[2] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li and Lei Zhang. SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution. In CVPR, 2024.\n\n> Q5. The paper proposes using wavelet transform to improve UNet but lacks experimental justification for why simple downsampling and upsampling wouldn't be more efficient.\n\nPlease see the answer to Q1 of the global response.\n\n> Q6. Figure 1, while aesthetically pleasing, is challenging to understand. It would be better to clearly explain the network structure (e.g., Figure 8) and the inference process.\n\nThe Figure 1 illustrates the two main networks (i.e., semantic distiller for semantic extraction and pixel condenser for denoising) as a conceptual sketch, rather than a main figure. To further demonstrate the network architecture of the network and the its connection with the devised components, we supplement detailed illustration and explains, as you seen in the Figure 8 and subsequent text in the **Section B** of supplemental materials. \n\nA video clip consisting of five frames is parallelly sampled during the inference process. These LR frames are first fed into the semantic distiller to extract semantic tokens and then corrupted by random noise as the input of the pixel condenser. The pixel condenser iteratively generates the corresponding HR counterparts from noisy LR frames under the condition of LR frames and semantic priors. The specific information flow of the pixel condenser is elaborated in Section B of supplemental materials. Due to the space limitation, we would like to supplement the pseudo-code of the inference in the final edition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and reviewers,

We sincerely thank all reviewers for your constructive comments. We are glad that the reviewers appreciate the **novelty** (srYj, ERJ1), **writing** (ERJ1), **impressive experimental results** (vLVk, srYj, ERJ1, NXC1) and limitations adequately discussed (vLVk, srYj, ERJ1) in the paper. Since reviewer vLVk, reviewer srYj, and reviewer NXC1 all concern the ablation results for specific components and require supplementing the efficiency comparison with other related methods. Thus, we address these issues in the following parts.

> Q1. Ablation study about wavelet transform and Blurring ResShift.

We add an ablation study to validate the efficacy of the introduced wavelet transform on the REDS4 dataset. After substituting the wavelet transform with simple downsampling and upsampling, denoted as model #6 in Table 1 of the rebuttal document, a noticeable decline in the values of PSNR/SSIM can be observed compared to SeeClear. It can be drawn that wavelet transform not only changes the resolution of features like downsampling/upsampling but also assists SeeClear in achieving consistency with LR frames in the low-frequency component (indicated by a drop in PSNR value) and efficiently generating high-frequency spectrums via skip connections between the encoder and the decoder (shown by the deterioration of the perceptual metric), thereby further promoting the performance.

We also combine different degrees of blur intensity and noise schedule of ResShift to comprehensively verify the necessity of Blurring ResShift in Table 2 of the rebuttal document. It can be concluded that different options of blur intensity impact the trade-off between fidelity and perceptuality (Line 1-3), and the introduction of wavelet transform and self-attention across frequency spectrums can significantly improve both fidelity and perceptuality at the same time (Line 2 v.s. Line 5).

> Q2. Efficiency comparisons to other competing methods.

We provide a comprehensive comparison of the efficiency between our proposed method and diffusion-based methods in Table 4 of the rebuttal document. It presents the number of parameters of different models and their inference time for super-resolving 512 × 512 frames from 128 × 128 inputs. Combining these comparative results, we draw the following conclusions: i) Compared to semantic-assisted single-image super-resolution (e.g., CoSeR and SeeSR), our proposed method possesses fewer parameters and higher inference efficiency. ii) In contrast to existing diffusion-based methodologies for VSR, SeeClear is much smaller and runs faster, benefiting from the reasonable module designs and diffusion process combing patch-level blurring and residual shift mechanism. 

> Q3. The rational explanation of Blurring ResShift.

Blurring ResShift, a patch-level blurring version of ResShift, is elaborated in the related works, methods, and supplementary materials. Comparisons to ResShift and ablation experiments were exhibited in the main text and supplementary materials. We also supplement additional ablation experiments for the issues that concern the reviewers. Additionally, we analyze the final states of different diffusion processes via the power spectral density, which reflects the distribution of frequency content in an image, as illustrated in Figure 2 of the rebuttal document. It can be observed that IHDM performs blurring globally and has a significant difference in frequency distribution compared to the LR image, while the patch-level blurring is closer to the frequency distribution of the LR. On this basis, SeeClear further introduces residual and noise. Compared to ResShift without blurring, the diffusion process adopted by SeeClear makes the image more consistent with the LR in the low-frequency components and introduces more randomness in the high-frequency components, compelling the model to focus on the generation of high-frequency components.

Meta-review: This paper received three positive and a negative score. After checking the rebuttal, the AC agreed with three positive reviewers, considering the paper's technical contribution and performance improvement.

If the paper is accepted, please proofread the submission more to make it easier to understand. For example, the method section needs more clarification. And add more evaluation metrics such as DISTS, FID, NIQE, MANIQA, MUSIQ and CLIPIQA.
Decision: Accept (poster)