SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution

Authors: qi  tang
yao  zhao
meiqin  liu
chao  yao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 12:51:22

Abstract: Diffusion-based Video Super-Resolution (VSR) is renowned for generating perceptually realistic videos, yet it grapples with maintaining detail consistency across frames due to stochastic fluctuations. The traditional approach of pixel-level alignment is ineffective for diffusion-processed frames because of iterative disruptions. To overcome this, we introduce SeeClear--a novel VSR framework leveraging conditional video generation, orchestrated by instance-centric and channel-wise semantic controls. This framework integrates a Semantic Distiller and a Pixel Condenser, which synergize to extract and upscale semantic details from low-resolution frames. The Instance-Centric Alignment Module (InCAM) utilizes video-clip-wise tokens to dynamically relate pixels within and across frames, enhancing coherency. Additionally, the Channel-wise Texture Aggregation Memory (CaTeGory) infuses extrinsic knowledge, capitalizing on long-standing semantic textures. Our method also innovates the blurring diffusion process with the ResShift mechanism, finely balancing between sharpness and diffusion effects. Comprehensive experiments confirm our framework's advantage over state-of-the-art diffusion-based VSR techniques.

Main paper: 
 introduction: 
 Video super-resolution (VSR) is a challenging low-level vision task that involves improving the resolution and visual quality of the given low-resolution (LR) observations and maintaining the temporal coherence of high-resolution (HR) components. Various deep-learning-based VSR approaches [1, 2, 30, 17, 13, 20, 21] explore effective inter-frame alignment to reconstruct satisfactory sequences. Despite establishing one new pole after another in the quantitative results, they struggle to generate photo-realistic textures.With the explosion of diffusion model (DM) in visual generation [10, 31, 26] , super-resolution (SR) from the generative perspective also garners the broad attention [28, 29, 45, 5] . DM breaks the generation process into sequential sub-processes and iteratively samples semantic-specific images from Gaussian noise, equipped with a paired forward diffusion process and reverse denoising process. The former progressively injects varied intensity noise into the image along a Markov chain to simulate diverse image distributions. The latter leverages a denoising network to generate an image based on the given noise and conditions. Early efforts directly apply the generation paradigm to super-resolution, overlooking its characteristic while generating pleasing content, thus trapping in huge sampling overhead.Different from generation from scratch, SR resembles partial generation. The structural information that dominates the early stages of diffusion is contained in the LR priors, while SR tends to focus on generating high-frequency details [33, 15] . Besides, the loss of high-frequency information in LR videos stems from the limited sensing range of imaging equipment. As a result, solely disrupting frames with additive noise is inadequate to depict the degradation of HR videos [9] . Moreover, prevalent VSR methods employ delicate inter-frame alignment (e.g., optical flow or deformable convolution) to fuse the sub-pixel information across adjacent frames. However, the disturbed pixels pose a severe challenge to these methods, rendering the accuracy to deteriorate in the pixel space. To alleviate the above issue, we introduce SeeClear, an innovative diffusion model empowering distilled semantics to enhance the pixel condensation for video super-resolution. During the forward diffusion process, the lowpass filter is applied within the patch, gradually diminishing the high-frequency component, all while the residual is progressively shifted in the frequency domain to transform the HR frames to corresponding LR versions step by step. To reduce the computational overhead, the intermediate states are decomposed into various frequency sub-bands via 2D discrete wavelet transform and subsequently processed by the attention-based U-Net. Furthermore, we devise a dual semanticcontrolled conditional generation schema to enhance the temporal coherence of VSR. Specifically, a segmentation framework for open vocabulary is employed to distill instance-centric semantics from LR frames. They serve as prompts, enabling the Instance-Centric Alignment Module (InCAM) to highlight and associate semantically related pixels within the local temporal scope. Besides, abundant semantic cues in channel dimensions are also explored to form an extensional memory dubbed Channel-wise Texture Aggregation Memory (CaTeGory). It aids in global temporal coherence and boosts performance. Experimental results demonstrate that our method consistently outperforms existing state-of-the-art methods.In summary, the main contributions of this work are as follows:• We present SeeClear, a diffusion-based framework for video super-resolution that distills semantic priors from low-resolution frames for spatial modulation and temporal association, controlling the condition of pixel generation.• We reformulate the diffusion process by integrating residual shifting with patch-level blurring, and introduce an attention-based architecture to explore valuable information among wavelet spectra during the sampling process, incorporating feature modulation of intra-frame semantics.• We devise a dual semantic distillation schema that extracts instance-centric semantics of each frame and further assembles them into texture memory based on the semantic category of channel dimension, ensuring both short-term and long-term temporal coherence.2 Related Work 
video super-resolution: 
 Prominent video super-resolution techniques concentrate on leveraging sub-pixel information across frames to enhance performance. EDVR [37] employs cascading deformable convolution layers (DCN) for inter-frame alignment in a coarse-to-fine manner, tackling large amplitude video motion.BasicVSR [1] comprehensively explores each module's role in VSR and delivers a simple yet effective framework by reusing previous designs with slight modifications. Given the similarity between DCN and optical flow, BasicVSR++ [2] devises flow-guided deformable alignment, exploiting the offset diversity of DCN without instability during the training. VRT [16] combines mutual attention with self-attention, which is respectively in charge of inter-frame alignment and information preservation.RVRT [17] extends this by incorporating optical flow with deformable attention, aligning and fusing features directly at non-integer locations clip-to-clip. PSRT [30] reassesses prevalent alignment methods in transformer-based VSR and implements patch alignment to counteract inaccuracies in motion estimation and compensation. DFVSR [8] represents video with the proposed directional frequency representation, amalgamating object motion into multiple directional frequencies, augmented with a frequency-based implicit alignment, thus enhancing alignment.Building on the success of diffusion models in the realm of image generation [10, 31, 26, 25, 48] , diffusion-based super-resolution (SR) is advancing. SR3 [28] , a pioneering approach, iteratively samples an HR image from Gaussian noise conditioned on the LR image. In contrast, StableSR [36] applies diffusion-based SR in a low-dimensional latent space using the pre-trained auto-encoder to reduce computation and generate improved results through the generative priors contained in weights of Latent Diffusion. ResDiff [29] combines a lightweight CNN with DM to restore low-frequency and predict high-frequency components, and ResShift [45] redefines the initial step as a blend of the low-resolution image and random noise to boost efficiency. Applying a different approach, DiWa [23] migrates the diffusion process into the wavelet spectrum to effectively hallucinate high-frequency information. Upscale-A-Video [49] , for video super-resolution, introduces temporal layers into the U-Net and VAE-Decoder and deploys a flow-guided recurrent latent propagation module to ensure temporal coherence and overall video stability when applying image-wise diffusion model.Traditionally seen as a preparatory step for subsequent tasks [50, 12] , restoration is now reformulated with the assistance of semantics. SFT [38] utilizes semantic segmentation probability maps for spatial modulation of intermediate features in the SR network, yielding more realistic textures. SKF [40] supports low-light image enhancement model to learn diverse priors encapsulated in a semantic segmentation model by semantic-aware embedding module paired with semantic-guided losses. SeD [14] integrates semantics into the discriminator of GAN-based SR for fine-grained texture generation rather than solely learning coarse-grained distribution. CoSeR [32] bridges image appearance and language understanding to empower SR with global cognition buried in LR image, regarding priors of text-to-image (T2I) diffusion model and a high-resolution reference image as powerful conditions. SeeSR [39] analyzes several types of semantic prompts and opts tag-style semantics to harness the generative potential of the T2I model for real SR. Semantic Lens [34] forgoes pixel-level inter-frame alignment and distills diverse semantics for temporal association in the instance-centric semantic space, attaining better performance.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section G of the supplemental material. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 Given a low-resolution (LR) video sequence of N framesI LR i ∈ R N ×C×H×W, where i is the frame index, H × W represents spatial dimensions, and C stands for the channel of frame, SeeClear aims to exploit rich semantic priors to generate the high-resolution (HR) video I HR i ∈ R N ×C×sH×sW , with s as the upscaling factor. In the iterative paradigm of the diffusion model, HR frames are corrupted according to handcrafted transition distribution at each diffusion step (t = 1, 2, • • • , T ). And a U-shaped network is employed to estimate the posterior distribution using LR frames as condition during reverse generation. As illustrated in Figure 1 , it consists of a Semantic Distiller and a Pixel Condenser, respectively responsible for semantic extraction and texture generation.The LR video is initially split into non-overlapping clips composed of m frames for parallel processing. Semantic Distiller, a pre-trained network for open-vocabulary segmentation, distills semantics related to both instances and background clip by clip, denoted as instance-centric semantics. Pixel Condenser is an attention-based encoder-decoder architecture, in which the encoder extracts multi-scale features under the control of LR frames, and the decoder generates HR frames from coarse to fine. They are also bridged via skip connections to transmit high-frequency information at the same resolution. To maximize the network's generative capacity, instance-centric semantics are utilized as conditions for individual frame generation in the decoder. They also serve as the cues of inter-frame alignment for temporal coherence within the video clip and further cluster into a semantic-texture memory along channel dimension for consistency across clips.During the video capturing, frequencies exceeding the imaging range of the device are truncated, leading to the loss of high-frequency information in LR videos. Therefore, an intuition is to construct a Markov chain between HR frames and LR frames in the frequency domain. Inspired by blurring diffusion [11] , the forward diffusion process of SeeClear initializes with the approximate distribution of HR frames. It then iterates and terminates with the approximate distribution of LR frames using a Gaussian kernel convolution in frequency space facilitated by the Discrete Cosine Transformation (DCT). Considering the correlation of neighboring information, blurring is conducted within a local patch instead of the whole image. The above process is formulated as:EQUATIONEQUATIONwhere u 0 and u t denote HR frames and intermediate states in the frequency space for brevity.V T denotes the projection matrix of DCT. D t = e Λt is diagonal blurring matrix with Λ x×p+y = -π 2 ( x 2 p 2 + y 2 p 2 ) for coordinate (x, y) within patch of size p × p, and η t is the variance of noise. E is the identity matrix.In the realm of generation, the vanilla destruction process progressively transforms the image into pure Gaussian noise, leading to numerous sampling steps and tending to be suboptimal for VSR. An alternative way is to employ a transition kernel that shifts residuals between HR and LR frames, accompanied by patch-level blurring. The forward diffusion process is formulated as:EQUATIONEQUATIONwhere u l denotes LR frames transformed into the frequency space. e t indicates the residuals between LR and blurred HR frames at time step t. η t represents the shifting sequence and κ is a hyperparameter determining the intensity of noise. Upon this, SeeClear can yield HR frames by estimating the posterior distribution p(u 0 |u l ) in the reverse sampling progress, formulated as:EQUATIONEQUATIONwhere p θ (u t-1 | u t , u l ) represents the inverse transition kernel restoring u t to u t-1 . θ denotes learnable parameters of attention-based U-Net.To alleviate the computational overhead, preceding methods introduce an autoencoder to transform pixel-level images in the perceptually equivalent space, concentrating on the semantic composition and bypassing the impedance of high-frequency details. However, the loss of high-frequency information during encoding is hard to recover in the decoding and will deteriorate the visual quality. Therefore, we forgo the autoencoding method in SeeClear and incorporate discrete wavelet transform (DWT) in the diffusion process. Specifically, the HR and LR frames are recursively decomposed into four sub-bands:EQUATIONwhere I HR ll denotes the low-frequency approximation, I HR lh , I HR hl and I HR hh correspond to horizontal, vertical and diagonal high-frequency details. DWT 2D (•) represents the 2D Discrete Wavelet Transform (DWT). After k decompositions, each of them possesses a size of H 2 k × W 2 k . These coefficients are contacted along the channel dimension and serve in the diffusion process. The rationale behind employing DWT as a substitute is two-fold. Firstly, it enables the U-Net to perform on a small spatial size without information loss. Secondly, it benefits from the U-Net scaling [44] hindered by additional parameters of the autoencoding network. To make full use of DWT, window-based self-attention followed by channel-wise self-attention is stacked as the basic unit of U-Net, in charge of the correlation of intra-sub-bands and inter-sub-bands, respectively. .Multi-Frame Self-Attention.𝑓 ! 𝐹 ! 𝐹 , ! 𝑃 ! 𝑚 -1 𝐹 1 ! Figure 2:The illustration of Instance-Centric Alignment Module (InCAM). It utilizes the segmentation features to bridge the pixel-level information and instance-centric semantic tokens. And then, the semantic-aware features can be aligned in the semantic space based on their semantic relevance.Due to the destruction of the diffusion process, pixel-level inter-frame alignment, such as optical flow, is no longer applicable. With the premise of semantic embedding, we devise the Instance-Centric Alignment Module (InCAM) within video clips, as illustrated in Figure 2 . It establishes temporal association in the semantic space instead of intensity similarity among frames, avoiding the interference of noise and blurriness. Specifically, Semantic Distiller predicts a set of image features F img and text embedding F txt from LR frames and predefined vocabulary V. After that, the k image features with the highest similarity to the text embedding are retained, including token-level semantic priors and pixel-level segmentation features from LR frames. The above procedure is formulated as:EQUATIONEQUATIONwhere S (•) represents Semantic Distiller. top k (•) and Sim (•) denote the operations of selecting the k largest items and calculating the similarity respectively. O i and P i are semantic tokens and segmentation features, in which the former represents high-level semantics and can locate related pixels in the segmentation features. The segmentation features contain both semantics and lowlevel structural information, which is suitable for bridging the semantics and features of the Pixel Condenser. It is utilized to generate spatial modulation pairs prepared for semantic embedding, formulated as:EQUATIONEQUATIONwhere γ and β represent scale and bias for modulation. f i and F i correspond to original and modulated features. G denotes two convolutional layers followed by a ReLU activation. "⊙" represents the Hadamard product. After that, InCAM embeds semantics into modulated features based on multi-head cross-attention, yielding semantic-embedded features Fi :EQUATIONEQUATIONwhere Q i , K i and V i denote matrices derived from modulated features and semantic tokens. W Q , W K and W V represent the linear projections, and d is the dimension of projected matrices. SoftMax (•) denotes the SoftMax operation. To benefit from the adjacent supporting frames, it is necessary to establish semantic associations among frames. The frame-wise semantic tokens are further fed into the instance encoder-decoder for communicating semantics along the temporal dimension, generating clip-wise semantic tokens. It gathers all the information of the same semantic object within a clip and serves as the guide for inter-frame alignment. Specifically, InCAM combines semantic guidance and enhanced features to activate the related pixels across frames and utilizes multi-frame self-attention for parallel alignment and fusion. The above procedure is formulated as:EQUATIONEQUATIONwhere O c and Ô respectively denote clip-wise semantic and randomly initialized tokens. Enc (•) and Dec (•) represent instance encoder and decoder. MFSA (•) denotes the multi-frame self-attention [30] , the extended version of self-attention in video. Fi is aligned feature. The product of semantics and enhanced features is akin to the class activation mapping, which highlights the most similar pixels in the instance-centric semantic space among frames. Due to the limited size of the temporal window, the performance of clip-wise mutual information enhancement in long video sequences is unsatisfactory, which could lead to inconsistent content. To stabilize video content and enhance visual texture, the Channel-wise Texture Aggregation Memory (CaTeGory) is constructed to cluster abundant textures according to channel-wise semantics, as graphically depicted in Figure 3 . It is comprised of the channel-wise semantic and the corresponding texture. Specifically, channels of instance-centric semantics also contain distinguishing traits, which are assembled and divided into different groups to form the channel-wise semantic. Concurrently, hierarchical features from the decoder of the Pixel Condenser are clustered into the corresponding semantic group to portray the textures. The connection between them is established in a manner similar to the position embedding in the attention mechanism.The above process can be formulated as:EQUATIONwhere Fi,k is the features benefited from adjacent frames of k-th layer. Cj and Tj respectively denote channel-wise semantics and textures of j-th group, which are zero-initialized as network parameters.They are iteratively updated towards the final version (i.e., C j and T j ) by injecting external knowledge from the whole dataset and previous clips. And M (•) represents the construction of CaTeGory. It concatenates the multi-scale features and incorporates them into channel-wise semantics and textures:EQUATIONEQUATION)where Tj is the textures embedded channel-wise semantics. SA (•) and CA (•) indicate multi-head self-attention and cross-attention. The layer normalization and feed-forward network are omitted for brevity. It bridges the channel-wise semantics and textures via matrix multiplication and further fuses high-value information from the pyramid feature, delivering augmented semantic-texture pairs. The hierarchical features not only provide rich structural information but also carry relatively abstract amid features in order to benefit different decoder layers more effectively. At each layer, the prior knowledge stored in CaTeGory is firstly queried by the clip-wise semantics along channel dimension and aggregated for feature enhancement of the current clip, which is formulated as:EQUATIONEQUATIONwhere A j depicts the similarity between the clip-wise semantics and items of CaTeGory along channel dimensions, and Fi is the refined features as input of the next layer. As mentioned before, semantic-texture pairs are optimized as parts of the network during the training stage, absorbing ample priors from the whole dataset. In the sampling process, the update mechanism is reused to integrate the long-term information of video into memory to improve the super-resolution of subsequent clips.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: See Section A of the supplementary material. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 Datasets To assess the effectiveness of the proposed SeeClear, we employ two commonly used datasets for training: REDS [24] and Vimeo-90K [41] . The REDS dataset, characterized by its realistic and dynamic scenes, consists of three subsets used for training and testing. In accordance with the conventions established in previous works [1, 2] , we select four clips2 from the training dataset to serve as a validation dataset, referred to as REDS4. The Vid4 [19] dataset is used as the corresponding test dataset for Vimeo-90K. The LR sequences are degraded through bicubic downsampling (BI), with a downsampling factor of 4×.Implementation Details The pre-trained OpenSeeD [46] is opted as the semantic distiller with frozen weights, while all learnable parameters are contained in the pixel condenser. And the schedulers of blur and noise in the diffusion process follow the settings of IHDM [25] and ResShift [45] . During the training, the pixel condenser is first trained to generate an HR clip with 5 frames under the control of instance-centric semantics. And then, Channel-wise Texture Memory is independently trained to inject valuable textures from the whole dataset and be capable of fusing long-term information. Finally, the whole network is jointly fine-tuned. All training stages utilize the Adam optimizer with β 1 = 0.5 and β 2 = 0.999, where the learning rate decays with the cosine annealing scheme. The Charbonnier loss [4] is applied on the whole frames between the ground truth and the reconstructed frame, formulated asL = ||I HR i -I SR i || 2 + ϵ 2 .The SeeClear framework is implemented with PyTorch-2.0 and trained across 4 NVIDIA 4090 GPUs, each accommodating 4 video clips.Evaluation Metrics Comparative analysis is conducted among different VSR methods, with the evaluation being anchored on both pixel-based and perception-oriented metrics. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are utilized to evaluate the quantitative performance as pixel-based metrics. All of them are calculated based on the Y-channel, with the exception of the REDS4, for which the RGB-channel is used. On the perceptual side, Learned Perceptual Image Patch Similarity (LPIPS) [47] is elected for assessment from the perspective of human visual preference. It leverages a VGG model to extract features from the generated HR video and the ground truth, subsequently measuring the extent of similarity between these features.We compare SeeClear with several state-of-the-art methods, including regression-based and diffusionbased ones. As shown in Table 3 , SeeClear achieves superior perceptual quality compared to regression-based methods despite slightly underperforming in pixel-based metrics. We also provide an extended version, which leverages the generative capability of SeeClear to enhance the features of the regression-based model, akin to references such as [6, 3] . An observable increase in fidelity is accompanied by a notable further improvement in the perceptual metrics of the reconstructed results. Similar performance trends can be noted on Vid4 as those on REDS4. In particular, SeeClear achieves an LPIPS score of 0.1548, marking a relative improvement of 10.8% compared to the top competitor, SATeCo [6] . When pitted against a variant of SATeCo, which is not modulated by LR videos, SeeClear demonstrates a higher PSNR value with a comparable LPIPS score. It suggests that SeeClear benefits from the control of dual semantics, striking a balance between superior fidelity and the generation of realistic textures.As visualized in Figure 4 , SeeClear showcases its ability to restore textures with high fidelity more effectively compared with other methods. Despite large blurriness, SeeClear still demonstrates robust restoration capabilities for video super-resolution, reinforcing the efficacy of utilizing instancespecific and channel-wise semantic priors for video generation control. To further substantiate Table 1 : Performance comparisons in terms of pixel-based (PSNR and SSIM) and perception-oriented (LPIPS) evaluation metrics on the REDS4 [24] and Vid4 [19] EDVR [37] SeeClear ⋆ (Ours)BasicVSR [1] HR Clip Calendar, Vid4Bicubic VRT [16] EDVR [37] SeeClear ⋆ (Ours)BasicVSR [1] HR the temporal coherence acquired by SeeClear, we also visualize two consecutive frames from the generated HR videos, constructed using different diffusion-based VSR methodologies, as depicted in Figure 5 . ResShift synthesizes varied visual contents across two frames, such as the fluctuating figures on the license plate. Contrarily, HR frames generated via SeeClear maintain a higher temporal consistency and deliver pleasing textures.To assess the contribution of each component within the proposed SeeClear, we begin with a baseline model and gradually integrate these modules. Specifically, all semantic-related operations are bypassed, retaining solely spatial and channel self-attention and residual blocks, degenerating into a diffusion-based image SR model without any condition. Subsequently, we incrementally introduce the crafted semantic-conditional module into the baseline and formulate several variants. Their results are listed in Table 2 and partially visualized in Figure 6 . First, the intra-frame semantic condition brings about 1.5% improvements in LPIPS. Albeit the multi-frame self-attention further improves the perceptual quality, it also impairs the fidelity of the restored video. Under the control of InCAM, SeeClear can correlate semantically consistent pixels in adjacent frames by combining intraframe and inter-frame semantic priors, elevating the PSNR from 27.99 dB to 28.46 dB, and bringing about 6.6% improvements in LPIPS. Furthermore, upon integrating the semantic priors from CaTeGory, the fully-fledged SeeClear notably enhances both the pixel-based and perceptionoriented metrics simultaneously. It indicates that the cooperative control of semantics is more beneficial for generating videos of higher fidelity and better perceptual quality. As illustrated in Figure 6 , the baseline struggles to restore tiny and fine patterns without semantic condition, and it gradually gains improvement accompanied by the strengthening of semantic control.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See Section 4 of the main paper. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this work, we present a novel diffusion-based video super-resolution framework named SeeClear.It formulates the diffusion process by incorporating residual shifting mechanism and patch-level blurring, constructing a Markov chain initiated with high-resolution frames and terminated at lowresolution frames. It employs a semantic distiller and a pixel condenser for super-resolution during the inverse sampling process. The instance-centric semantics distilled by the semantic distiller prompts spatial modulation and temporal association in the devised Instance-Centric Alignment Module. They are further assembled into Channel-wise Texture Aggregation Memory, providing abundant conditions for temporal coherence and realistic content.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: See Section 4 of the main paper. We plan to release the code after the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 4 of the main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: See Section D of the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See Section 4 of the main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Section G of the supplementary material.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package and dataset we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Patch-level Blurring Diffusion. Blurring Diffusion [11] designs the destruction process based on heat dissipation [25] that retains low-frequency components of images over high frequencies. It describes the thermodynamic process in which the temperature u(x, y, t) changes at position (x, y) in a 2D plane with respect to the time t via a partial differential equation:EQUATIONwhere ∆ = ∇ 2 denotes the Laplace operator. Based on Neumann boundary conditions (∂u/∂x = ∂u/∂y = 0) with zero-derivatives at the boundaries of the image, the solution is given by a diagonal matrix in the frequency domain of the discrete cosine transform:EQUATIONwhere u 0 is the initial state and Λ is a diagonal matrix with negative squared frequencies on the diagonal. It is equivalent to a convolution with a Gaussian kernel with variance σ 2 blur = 2t in the realm of image processing. Blurring Diffusion further mixes Gaussian noise with variance σ 2 t into the blurring process to incorporate stochasticity into deterministic dissipation.Patch-level blurring diffusion conducts the blurring process within a patch with the size of p × p to make all pixel intensity the same instead of the whole image, which means Λ x×p+y = -π 2 ( x 2 p 2 + y 2 p 2 ). More generally, it can be extended with any invertible transformation, corresponding to the marginal distribution in the pixel space:EQUATIONwhere R and R -1 represent invertible transformation and corresponding inverse transformation. diag (•) denotes the operation that projects a vector to a diagonal matrix. α t and β t are specific schedules for mean and noise.Blurring ResShift. ResShift [45] introduces a monotonically increasing sequence {η t } T t=1 to gradually shift residual between low-resolution image and high-resolution one, whose marginal distribution is defined as:EQUATIONwhere e 0 is the residuals between low-resolution and high-resolution frames. η t controls the speed of residual shifting and satisfies η 1 → 0 and η T → 1. After incorporation of patch-level blurring diffusion and ResShift, u t can be sampled viaEQUATIONEQUATIONwhere ϵ t ∼ N (u | 0, E). Thus, the relation between u t and u t-1 can be obtained:EQUATIONEQUATIONwhere α t = η t -η t-1 . û0 is approximate HR frame. By recursively applying the sampling procedure and reparameterization trick, we can rewrite the marginal distribution of Blurring ResShift as follows:EQUATIONAccording to Bayes's theorem, there isEQUATIONwhere And then, it only needs to focus on the quadratic term in the exponent of q (u t-1 | u t , u 0 , u l ):EQUATIONEQUATION-EQUATIONwhereEQUATIONEQUATIONEQUATIONwhere 'const' denotes the item that is independent of u t-1 .We analyze the final states of different diffusion processes via the power spectral density, which reflects the distribution of frequency content in an image, as illustrated in Figure 4 . It can be observed that IHDM performs blurring globally and has a significant difference in frequency distribution compared to the LR image, while the patch-level blurring is closer to the frequency distribution of the LR. On this basis, SeeClear further introduces residual and noise. Compared to ResShift without blurring, the diffusion process adopted by SeeClear makes the image more consistent with the LR in the low-frequency components and introduces more randomness in the high-frequency components, compelling the model to focus on the generation of high-frequency components. SeeClear consists of a forward diffusion process and a reverse process for VSR. In the diffusion process, patch-level blurring and residual shift mechanism are integrated to degrade HR frames based on the handcrafted time schedule. During the reverse process, a transformer-based network for open vocabulary segmentation and a U-Net are employed for iterative denoising. The former is responsible for extracting semantic embeddings related to instances from LR videos, similar to the process of distillation in physics, and is therefore named the semantic distiller. The latter is utilized to filter out interfering noise and retain valuable information from low-quality frames, similar to the condensation process. All of them are tailored for image processing, and SeeClear takes diverse semantic embeddings as conditions to enable the network to be aware of the generated content and determine the aligned pixels from adjacent frames for the temporal consistency of the whole video.As depicted in Figure 8 , the attention-based Pixel Condenser primarily consists of three parts, i.e., encoder, decoder, and middle block. As the input of the encoder, low-resolution frames are concatenated with the intermediate states and processed through a convolution layer. The encoder incorporates a window-based self-attention and channel-wise self-attention, alternating between two residual blocks. These attention operations mine valuable information within and across the wavelet sub-bands, and the residual blocks infuse the features with the intensity of degradation. Following this, a wavelet-based downsample layer is deployed for feature downsampling.Specifically, the features are decomposed into various sub-bands via a 2D discrete wavelet transform, reducing the spatial dimensions while keeping the original data intact. Low-frequency features are further fed into the subsequent layer of the encoder, while others are transmitted to the corresponding wavelet-based upsample layer in the decoder via a skip connection. Additionally, the wavelet-based downsample layers are utilized parallel along the encoder, refilling the downsampled features with information derived from the low-resolution frames.The devised Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) are inserted into both the middle and decoder. Firstly, the InCAM spatially modulates the features based on instance-centric semantics and aligns adjacent frames within the clip in semantic space. Spatial self-attention employs these aligned features, substituting the original features as input. Subsequently, features enhanced through channel-wise self-attention are embedded as queries to seek assistance from the CaTeGory. Furthermore, the wavelet-based upsample layer accepts features from the decoder and high-frequency information transmitted from the encoder. It reinfuses the lost information from the encoder while scaling the feature size. The network concludes with a convolution layer refining the features, which are added to the interpolated frames to generate the final output.Table 3 : Performance comparisons in terms of Full-Reference IQA (DISTS [7] ) and No-Reference IQA (NIQE [22] and CLIP-IQA [35] ) evaluation metrics on the REDS4 [24] and Vid4 [19] We provide additional evaluation metrics and visual comparisons contrasting the existing VSR methods with our proposed SeeClear. As demonstrated in Figure 9 , our proposed method successfully generates pleasing images without priors pre-trained on large-scale image datasets, showcasing sharp edges and clear details, evident in vertical bar patterns of windows and characters on license plates. Conversely, prevalent methods seemingly struggle, causing texture distortion or detail loss in analogous scenes.We also execute several experiments focused on the degradation scheme of the diffusion process, verifying the performance of the models on the first frame, as indicated in Table 4 . Compared to the straightforward diffusion process of simply adding Gaussian noise into HR frames, the combination of blurring diffusion proves beneficial in generating the high-frequency details discarded in LR frames. Specifically, the variation in the intensity of blur (Line 1-3) affects the fidelity and perceptual quality. Among them, there is no blurring when σ 2 B = 0, and the greater the value of σ 2 B , the greater the blurring intensity. It can be observed there is a 0.96 dB improvement in PSNR and the value of LPIPS ranges from 0.2096 to 0.2067 with the increasement of σ 2 B . On another note, implementing attention mechanisms in the wavelet spectrum proves more successful in uncovering valuable insights than in the pixel domain. Additionally, SeeClear introduces the alternation of spatial self-attention and channel self-attention, refining the modeling of intra and inter-frequency sub-band correlations and remarkably enhancing the quality of the generated highresolution frames. Steps Time (s) SISR LDM [26] 169.0 M 200 5.21 DiffBIR [18] 1716.7M 50 5.85 ResShift [45] 173.9M 15 1.12 StableSR [36] 1409.1M 200 18.70 CoSeR [32] 2655.52M 200 -SeeSR [39] 2283.7M 50 7.24 PASD [42] 1900.4M 20 5 compares the efficiency between our proposed method and diffusion-based methods. It presents the number of parameters of different models and their inference time for super-resolving 512 × 512 frames from 128 × 128 inputs. Combining these comparative results, we draw the following conclusions: i) Compared to semanticassisted single-image super-resolution (e.g., CoSeR [32] and SeeSR [39] ), our proposed method possesses fewer parameters and higher inference efficiency. ii) In contrast to existing diffusion-based methodologies for VSR [27, 43] , SeeClear is much smaller and runs faster, benefiting from the reasonable module designs and diffusion process combing patchlevel blurring and residual shift mechanism.A video clip consisting of five frames is parallelly sampled during the inference process. These LR frames are first fed into the semantic distiller to extract semantic tokens and then corrupted by random noise as the input of the pixel condenser. The pixel condenser iteratively generates the corresponding HR counterparts from noisy LR frames under the condition of LR frames and semantic priors. The pseudo-code of the inference is depicted in the Algorithm 1.Limited by the size and diversity of the dataset, SeeClear, being solely trained on video superresolution datasets, does not fully leverage the generative capabilities of diffusion models as efficiently as those that benefit from pre-training on large-scale image datasets such as ImageNet. While SeeClear is capable of generating sharp textures and maintaining consistent details, it falls short in restoring tiny objects or intricate structures with complete satisfaction. I SR n ∈ R N ×C×sH×sW 1 ĪLR n = VRT(I LR n ) 2 u T n = DWT 2D ( ĪLR n ) + ϵ, ϵ ∼ N (0, I) 3 ----------Parallel Generation within Local Video Clip ---------- 4 for m = 1 to N M do // Mf i-1 n = [I t n , u t n ] 15 else 16 f i-1 n = [WD(I LR n ), f i-1 n ] // WD denotes Wavelet-based DownSample 17 f i n = E i (f i-1 n ) 18 if i ̸ = 4 then 19 L i n , H i n = WD( f i n ) 20 f i n = L i n 21 skip H = skip H + H i n 22 else 23 f i n = f i n 24 O m = Dec(Enc(O n ), Ô)25 for j = 1 to 3 do // j refers to j th layer of middle blocks B γ j n , β j n = G(P n ) 27 f j n = ( f j-1 n γ j n + β j n ) + f j-1 n 28 f j n = CA(f j n , O m , O m ) 29 f j n = B j (MFSA(O c • f j n )) 30 f eats = [ ] //f k n = WU([ f k-1 n , skip H [-k]]) // WU denotes Wavelet-based UpSample 34 γ k n , β k n = G(P n ) 35 f k n = (f k-1 n γ k n + β k n ) + f k-1 n 36 f k n = CA( f k n , O c , O c ) 37 f k n = D k (MFSA(O c • f k n )) 38 f k n = CA( f k n , C m , T m ) 39 f eats = f eats + f k n 40 I t n = IDWT 2D ( fn ) + ĪLR n 41 Īt n = α t µ(I t n , Īt+1 n ) + η t ϵ 42 --------Update CaTeGory for Global Video Consistency -------- 43 if t=0 then 44 C m+1 , T m+1 = M(C m , T m , f eats) 45 I SR n = I t nIn addition, compared to synthetic datasets, videos captured in real-world scenarios display more complex and unknown degradation processes. Although real-world VSR is garnering significant attention, it remains an unexplored treasure and a steep summit to conquer for diffusion-based VSR, including SeeClear.Significantly, substituting an Autoencoder with a traditional wavelet transform lessens the computational burden while ensuring the preservation of the original input information. Nevertheless, in the process of inverse wavelet transform, the spatio-temporal information within videos is not delved into further, leading to subpar outcomes. Meanwhile, some methods develop trainable wavelet-like transformations based on the lifting scheme, allowing for end-to-end training of the whole network. Such a schema presents a promising direction for future research by potentially boosting model performance.As for societal impacts, similar to other restoration methods, SeeClear may bring privacy concerns after restoring blurry videos and lead to misjudgments if used for medical diagnosis and intelligent security systems, etc.NeurIPS Paper Checklist the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We confirm that the research conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Clip 000, 011, 015, 020 


Decision: This paper received three positive and a negative score. After checking the rebuttal, the AC agreed with three positive reviewers, considering the paper's technical contribution and performance improvement.

If the paper is accepted, please proofread the submission more to make it easier to understand. For example, the method section needs more clarification. And add more evaluation metrics such as DISTS, FID, NIQE, MANIQA, MUSIQ and CLIPIQA.

Reviews: review: paper_summary: paper_summary: The paper presents a framework for video super-resolution (VSR) that improves temporal coherence and high-resolution detail generation. The proposed method, SeeClear, integrates a Semantic Distiller and a Pixel Condenser to extract and upscale semantic details from low-resolution frames. The framework employs an Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) to enhance inter-frame coherence and incorporate long-standing semantic textures. The methodology also introduces a blurring diffusion process with the ResShift mechanism to balance sharpness and diffusion effects. Experimental results show that SeeClear outperforms state-of-the-art diffusion-based VSR techniques in terms of perceptual quality and temporal consistency.
strength_weakness: strength_weakness: 1. While the method demonstrates robust restoration capabilities, it may still struggle with accurately restoring tiny objects or intricate structures, especially under severe degradation conditions.
2. The method has been tested primarily on specific benchmark datasets. Its performance in real-world applications, where video degradation processes are more varied and unpredictable, remains to be thoroughly evaluated.
3. The experiments are not sufficient and should be improved.
questions: questions: 1. The performance of the proposed method is not significant. In Table 1, the improvement is very marginal or is worse than other methods. Moreover, in Figure 4, the generated texture is comparable to other methods.

2. It would be better to compare more methods. (a) Transformer-based (e.g., VSR Transformer) or RNN-based method (e.g., BasicVSR++); (b) Diffusion-based image restoration methods (e.g., DDRM, DDNM, DeqIR, etc). (c) Compare methods trained with more frames (e.g., VRT-16).

3. The authors should compare the efficiency, including model size, training/inference time and FLOPs. The efficiency comparisons can demonstrate the effectiveness of the proposed method.
limitations: limitations: Please refer to the details above.
paper_summary: paper_summary: This paper presents a diffusion-based video super-resolution method, and proposes Instance-Centric Alignment Module and Channel-wise Texture Aggregation Memory. The former leverages a pre-trained open-vocabulary segmentation model (i.e., OpenSeeD), which is utilized to perform alignment within video clips by modulating the spatial and temporal features. The latter leverages channel-wise attention and memory mechnism to better super-resolve the video frames. The results on publich benchmarks indicate that the proposed method achieves state-of-the-art perceptual performance.
strength_weakness: strength_weakness: Although the proposed method achieves promising results on the public benchmarks, there are some concerns that greatly affect the rating of this paper.

1. The presentation of the method needs to be improved. The readability of the paper is unsatisfactory. The technical details and the rationale behind it is not clearly described and explained.
(a) The main figure (Figure 1) is ambiguous. It is hard to understand the workflow of the framework based on this figure. It is also hard to see the connection among different modules.
(b) In the abstract, what is the "conditional video generation" (L6)? I do not see any pre-trained conditional video generation module in the described method. Maybe it should be rephrased.
(c) In L206-207, what is the role of "randomly initialized tokens"? And what is specific role of the encoder-decoder module?
(d) In L187-188, are the "semantic tokens" actually text embeddings ? What is the difference?
(e) In L223, how to divide channels into different groups and what is rationale behind it?
(f) It is hard to understand Eq. (16), (17) and (18). From (17) and (18), it seems T_j is used to calculate itself, which is confusing.
(g) The choice of mathematical notations is sub-optimal and confusing.
(h) In L149, I think the "belta_t" should be "yita_t".

2. The novelty of this paper is limited.
(a) Some of the modules are based on existing methods. For example, the way of introducing semantic features is similar to SFT (but no comparison in the paper); the multi-frame self attention is from [21].
(b) The proposed blurring ResShift is a modification version based on ResShift, but the rationale behind it is not fully explained. Also, there is no direct ablation.

3. The comparison with other related methods are not thorough. 
(a) The authors should explicitly compare with ResShift [33], since residual shifting technique is also exploited (but no citation in L48). Also, there is no comparison with it in Sec. 2.2.
(b) The authors should compare with Upscale-A-Video [36], another diffusion-based video super-resolution method. Also, it is recommended to compare the performance of [36].
(c) The authors should compare with SFT[28], another method also leveraging semantic segmentation information.

4. The proposed method is not fully ablated. There is no direct ablation for exploitation of DWT and blurring resshift.

5. Some of the statements could be inappropriate. 
(a) In L35-36, I think it is hard to reach the given conclusion from [8]. Please elaborate.
(b) The naming of "semantic distiller" could be inappropriate. The pre-trained semantic segmentation model is directly leveraged and frozen. I don't see any distillation.
questions: questions: 1. In the abstract, what is the "conditional video generation" (L6)? I do not see any pre-trained conditional video generation module in the described method.
2. In L206-207, what is the role of "randomly initialized tokens"? And what is specific role of the encoder-decoder module?
3. In L187-188, are the "semantic tokens" actually text embeddings ? What is the difference?
4. In L223, how to divide channels into different groups and what is rationale behind it?
5. It is hard to understand Eq. (16), (17) and (18). From (17) and (18), it seems T_j is used to calculate itself, which is confusing. Please elaborate.
6. In InCAM, the way of introducing semantic features seems similar to SFT. Please compare with it and illustrate the significance of the proposed module.
7. Please provide comparison with the following related methods, and illustrate the novelty of the proposed modules. 
(a) The necessity and rationale of blurring ResShift, and the ablation study.
(b) Upscale-A-Video [36]. And please compare performance with it quantitatively.
8. In L35-36, I think it is hard to reach the given conclusion from [8]. Please elaborate.
limitations: limitations: The authors have adequately addressed the limitations in Section D.
paper_summary: paper_summary: The paper introduces a novel video super-resolution framework leveraging semantic distillation to enhance pixel condensation in diffusion-based models. SeeClear addresses stochastic fluctuations by using a Semantic Distiller and a Pixel Condenser to extract and upscale semantic details from LR frames. The framework includes an Instance-Centric Alignment Module and a Channel-wise Texture Aggregation Memory to improve temporal consistency and visual quality. Experimental results demonstrate SeeClear's superiority over state-of-the-art diffusion-based VSR techniques.
strength_weakness: strength_weakness: - Lack of computation analysis. Diffusion-based methods are often criticized for unbearable inference time, so it would be better to list params, runtime, and FLOPs/MACs for a fair comparison.
- Lack of an ablation study on the wavelet transform which is introduced in Section 3.1.
- Table 2 is incomplete, making it difficult to assess the effect of the CaTeGory.
- The Other baselines such as VRT and IconVSR are also evaluated on Vimeo-90K-T and UDM10 datasets. Could you complete it for a fair comparison?
- Figure 7 needs more explanation.
questions: questions: Diffusion-based models usually show poor performance on PSNR (e.g., StableSR and Reshift), but SeeClear demonstrates a significant improvement. Could you analyze which parts of SeeClear contribute to this improvement?
Please refer to the weaknesses part above.
limitations: limitations: The authors have largely addressed their limitations.
paper_summary: paper_summary: The authors propose SeeClear for Video Super-Resolution (VSR). SeeClear is a diffusion-based method that improves restoration performance by introducing semantic priors. The authors design an Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) to utilize semantic information effectively. Comparisons on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance.
strength_weakness: strength_weakness: 1. The method uses pre-trained models to extract semantic information, introducing significant additional computation, which limits the method's applicability. Meanwhile, the paper lacks comparisons of complexity and parameter counts.
2. The method lacks experimental support for some critical hyperparameters, such as the choice of k in InCAM and the number of frames used in SR.
3. The paper proposes using wavelet transform to improve UNet but lacks experimental justification for why simple downsampling and upsampling wouldn't be more efficient.
4. Figure 1, while aesthetically pleasing, is challenging to understand. It would be better to clearly explain the network structure (e.g., Figure 8) and the inference process.
questions: questions: 1. Why do the comparison methods in Table 1 use different numbers of frames? If the same frame is used, what is the performance like?
2. In the ablation study (model 2, Table 2), how to use semantic conditions without MFSA, InCAM, and CaTeGory?
3. In InCAM, what is the value of k in top k, and how is it determined? Is there experimental support for this choice?
4. Others see weaknesses.
limitations: limitations: The authors discuss the method's limitations and societal impacts.

Comments: title: Response to the authors
After Rebuttal
Response to the rebuttal
Reformulation of Eq. (16)-(18).
comment: Thank you for your rebuttal. I raised my rating to borderline accept.
Thank you for your detailed response. 
I have actually misunderstood some technical details, and the response helps a lot. 
The technical soundness and the novelty can be rated with a higher score.
I will raise my rating to "weak accept".
> Q7: Still no further explanation on the significance of the proposed blur ResShift.

We conducted comprehensive ablation experiments of Blurring ResShift in the complementary materials and presented it in the attached rebuttal document again. As shown in Table 2 of the rebuttal document, the variation in the intensity of blur (Line 1-3) affects the fidelity and perceptual quality. Among them, there is no blurring when $\sigma_B^2=0$, and the greater the value of $\sigma_B^2$, the greater the blurring intensity. It can be observed there is a 0.96 dB improvement in PSNR and the value of LPIPS ranges from 0.2096 to 0.2067 with the increasement of $\sigma_B^2$. Although the PSNR if not good enough when $\sigma_B^2=2$, the incorporation of SeeClear enables it to achieve a double-win in terms of both PSNR and LPIPS, such as the alternation of spatial self-attention and channel-wise self-attention, wavelet-based downsampling, etc. We believe that the sufficient experiments demonstrate the effectiveness and necessity of Blurring ResShift.

> Q8: The answer to Q3 seems to have no relation with Q8. 

To answer the question you previously asked in Q8, we offer explanations from two  perspectives. Firstly, there are less high-frequency information in LR frames compared to HR frames [1], and further advancing DDPMs requires finer high-frequency details prediction [2]. Besides, diffusion-based SR methods tend to generate rather different outputs for the same LR image due to inherent stochasticity. To address these issues, an ideal diffusion process can follow a similar distribution in the low-frequency components and introduce more stochasticity in the high-frequency spectra. In other words, the network is forced to focus more on the feature extraction and modeling in the high-frequency bands, thus generating more pleasant details and textures with consistent structures [3, 4]. Therefore, we visualize the Power Spectral Density of terminated states generated by different diffusion processes in the rebuttal document, as illustrated in Figure 2. Based on the lack of high-frequency information and importance of fidelity, sole blurring or additive noise is not enough for SR, as they respectively eliminate too much low-frequency information and introduces interfered high-frequency information. Secondly, the ablation experiments on the intensity of blurring also validate the significance of combination of blurring and noise, which has been elaborated above. In a nutshell, there are sufficient theories and experimental evidence to support the conclusion.

[1] Wang X, Chai L, Chen J. Frequency-Domain Refinement with Multiscale Diffusion for Super Resolution[J]. arXiv preprint arXiv:2405.10014, 2024.

[2] Moser B, Frolov S, Raue F, et al. Waving goodbye to low-res: A diffusion-wavelet approach for image super-resolution[J]. arXiv preprint arXiv:2304.01994, 2023.

[3] Sun L, Wu R, Zhang Z, et al. Improving the stability of diffusion models for content consistent super-resolution[J]. arXiv preprint arXiv:2401.00877, 2023.

[4] Xiao X, Li S, Huang L, et al. Multi-Scale Generative Modeling in Wavelet Domain[J].
Thanks for your valuable comments. We provide more materials and details for understanding semantic tokens and the way to divide channels. Besides, there’re sufficient theories and experimental results to demonstrate the necessity of combination of blurring and additive noise in the diffusion process for SR, i.e., Blurring ResShift. We have also provided detailed ablation experiments and explanations to validate the significance of the proposed Blurring ResShift during the rebuttal period.

> Q3: The answer is still unclear. I still do not unstand the meaning of "semantic tokens" (O) and what are the relation with the visual features (F). 

The same terms are employed in previous works [3, 5]. Encoder-decoder based transformer architecture for segmentation has been prevalently utilized in recent years, and the concept of semantic token is blatantly simple to understand and widely accepted. We firmly don't think it absolutely constitute a reason for rejection.

As clearly stated in the rebuttal, transformer-based segmentation network [1, 2, 3, 4] requires **two types of input, i.e., images and learnable tokens**, which is well-understood in the realm of visual tasks and has also been applied in numerous image segmentation tasks.  The encoder of the transformer extracts multi-scale features from images via self-attention (Refer to Fig. 2 and its caption in [1]), yielding visual features (F). Meanwhile, the decoder of the transformer establishes relationships between learnable tokens and multi-scale features (Refer to “box embeddings”, “encoded image” and “Resnet features” of Fig. 8 in [1]), generating **semantic tokens (O) that can retrieve pixels related to instances and background from visual features** (Refer to “object queries” of Fig. 2 in [1]). They are combined to produce the final segmentation mask through MLP or element-wise multiplication. 

[1] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]//**European Conference on Computer Vision (ECCV)**. Cham: Springer International Publishing, 2020: 213-229.

[2] Cheng B, Misra I, Schwing A G, et al. Masked-attention mask transformer for universal image segmentation[C]//Proceedings of the **IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)**. 2022: 1290-1299.

[3] Zou X, Dou Z Y, Yang J, et al. Generalized decoding for pixel, image, and language[C]//Proceedings of the **IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)**. 2023: 15116-15127.

[4] Li X, Ding H, Yuan H, et al. Transformer-based visual segmentation: A survey[J]. **IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)**, 2024.

[5] Zhang H, Li F, Zou X, et al. A simple framework for open-vocabulary segmentation and detection[C]//Proceedings of the **IEEE/CVF International Conference on Computer Vision (CVPR)**. 2023: 1020-1031.

> Q4: I do not see any direct answer to the question of how to divide channels.

The direct answer has been elaborated in the second paragraph beginning with “we dig abundant”，but we’d like to explain it in a simple way. The cluster of channel relies on two zero-initialized parameters of network, i.e., channel-wise semantics ($C_j$) and textures ($T_j$). The semantic parameters initially model the relationship with channels of instance-centric tokens ($O_i$) via cross-attention along channel dimension instead of vanilla spatial cross-attention, which is formulated as:

$$
Q=C_j W^Q,K=O_i W^K,V=O_i W^V
$$
$$
A=\operatorname{softmax}((Q^T K)/\sqrt{d})
$$
$$
\widetilde{C}_j =AV+C_j
$$

After stacked cross-attention and residual connection, the semantics contained in channels of instance-centric semantic tokens are resembled into semantic parameters, thus grouping different channel-wise semantics. Then, the texture parameter is combined with semantic parameter via matrix multiplication. It appoints each channel semantic to a corresponding texture. The texture parameter assembles valuable textures from multi-scale features via cross-attention, similarly to the above process. The motivation and technical details are elaborated, we think that you should read our paper and rebuttal more attentively.

> Q5: Where is the "official comments"?

You may not be able to access the reformulation of Eq. (16)-(18) provided during the rebuttal period due to system. We’d like to exhibit the same contents here.

We reformulate Eq. (16)-(18) as follows:

$$
\left(C_j, T_j\right)=\mathcal{M}\left(\bar{C}_j, \bar{T}_j,\left\\{\bar{F} _{i, k}\right\\} _{k=1}^4\right)
$$

$$
\widehat{T}_j=\bar{C}_j \times \bar{T}_j
$$

$$
T_j=\operatorname{SA}\left(\operatorname{CA}\left(\widehat{T}_j,\left\\{\bar{F} _{i, k}\right\\} _{k=1}^4\right)\right)
$$

where $\bar{C}_j$ and $\bar{T}_j$ respectively denote zero-initialized channel-wise semantics and textures of j-th group. $\widehat{T}_j$ is the texture embedding that possesses one-to-one correspondence with semantic embedding via matrix multiplication.
Thanks for the rebuttal. The authors provide extensive experiments to demonstrate the effectiveness of the proposed method. 

I also read the comments of other reviewers. The main concerns are about the lack of experiments, and I think the authors' experiments addressed these issues to some extent.

Overall, the authors address my concerns, and I am willing to raise my score to 6.
Q3: The answer is still unclear. I still do not unstand the meaning of "semantic tokens" (O) and what are the relation with the visual features (F).
Q4: I do not see any direct answer to the question of how to divide channels.
Q8: The answer to Q3 seems to have no relation with Q8. 

I would like to maintain my current rating.
We reformulate Eq. (16)-(18) as follows:

$$
\left(C_j, T_j\right)=\mathcal{M}\left(\bar{C}_j, \bar{T}_j,\left\\{\bar{F} _{i, k}\right\\} _{k=1}^4\right)
$$

$$
\widehat{T}_j=\bar{C}_j \times \bar{T}_j
$$

$$
T_j=\operatorname{SA}\left(\operatorname{CA}\left(\widehat{T}_j,\left\\{\bar{F} _{i, k}\right\\} _{k=1}^4\right)\right)
$$

where $\bar{C}_j$ and $\bar{T}_j$ respectively denote zero-initialized channel-wise semantics and textures of j-th group. $\widehat{T}_j$ is the texture embedding that possesses one-to-one correspondence with semantic embedding via matrix multiplication.