Title: SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution
Venue: NeurIPS 2024
Accepted: yes
Keywords: Video Super-Resolution

Diffusion Model
Abstract: Diffusion-based Video Super-Resolution (VSR) is renowned for generating perceptually realistic videos, yet it grapples with maintaining detail consistency across frames due to stochastic fluctuations. The traditional approach of pixel-level alignment is ineffective for diffusion-processed frames because of iterative disruptions. To overcome this, we introduce SeeClear--a novel VSR framework leveraging conditional video generation, orchestrated by instance-centric and channel-wise semantic controls. This framework integrates a Semantic Distiller and a Pixel Condenser, which synergize to extract and upscale semantic details from low-resolution frames. The Instance-Centric Alignment Module (InCAM) utilizes video-clip-wise tokens to dynamically relate pixels within and across frames, enhancing coherency. Additionally, the Channel-wise Texture Aggregation Memory (CaTeGory) infuses extrinsic knowledge, capitalizing on long-standing semantic textures. Our method also innovates the blurring diffusion process with the ResShift mechanism, finely balancing between sharpness and diffusion effects. Comprehensive experiments confirm our framework's advantage over state-of-the-art diffusion-based VSR techniques.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose SeeClear for Video Super-Resolution (VSR). SeeClear is a diffusion-based method that improves restoration performance by introducing semantic priors. The authors design an Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) to utilize semantic information effectively. Comparisons on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance.\n\nSTRENGTHS:\n1. The paper introduces semantic priors to achieve spatial modulation and temporal correlation, improving diffusion-based VSR performance. This idea is both reasonable and effective.\n2. The authors design the Instance-Centric Alignment Module (InCAM) to align using semantic information, avoiding pixel inconsistencies and being well-suited for diffusion models.\n3. Additionally, the authors propose the Channel-wise Texture Aggregation Memory (CaTeGory) to transfer semantic information between different frames.\n4. Comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed method.\n5. The paper is well-organized, with clear and aesthetically pleasing layouts, figures, and tables.\n\nWEAKNESSES:\n1. The method uses pre-trained models to extract semantic information, introducing significant additional computation, which limits the method's applicability. Meanwhile, the paper lacks comparisons of complexity and parameter counts.\n2. The method lacks experimental support for some critical hyperparameters, such as the choice of k in InCAM and the number of frames used in SR.\n3. The paper proposes using wavelet transform to improve UNet but lacks experimental justification for why simple downsampling and upsampling wouldn't be more efficient.\n4. Figure 1, while aesthetically pleasing, is challenging to understand. It would be better to clearly explain the network structure (e.g., Figure 8) and the inference process.\n\nQUESTIONS:\n1. Why do the comparison methods in Table 1 use different numbers of frames? If the same frame is used, what is the performance like?\n2. In the ablation study (model 2, Table 2), how to use semantic conditions without MFSA, InCAM, and CaTeGory?\n3. In InCAM, what is the value of k in top k, and how is it determined? Is there experimental support for this choice?\n4. Others see weaknesses.\n\nLIMITATIONS:\nThe authors discuss the method's limitations and societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel video super-resolution framework leveraging semantic distillation to enhance pixel condensation in diffusion-based models. SeeClear addresses stochastic fluctuations by using a Semantic Distiller and a Pixel Condenser to extract and upscale semantic details from LR frames. The framework includes an Instance-Centric Alignment Module and a Channel-wise Texture Aggregation Memory to improve temporal consistency and visual quality. Experimental results demonstrate SeeClear's superiority over state-of-the-art diffusion-based VSR techniques.\n\nSTRENGTHS:\n- The combination of semantic distillation and pixel condensation is novel and effectively addresses the challenges of maintaining detail consistency across frames in diffusion-based VSR.\n- The Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) significantly improve short-term and long-term temporal coherence.\n- The paper provides extensive experiments to demonstrate SeeClear's advantages over sotas across multiple benchmarks.\n\nWEAKNESSES:\n- Lack of computation analysis. Diffusion-based methods are often criticized for unbearable inference time, so it would be better to list params, runtime, and FLOPs/MACs for a fair comparison.\n- Lack of an ablation study on the wavelet transform which is introduced in Section 3.1.\n- Table 2 is incomplete, making it difficult to assess the effect of the CaTeGory.\n- The Other baselines such as VRT and IconVSR are also evaluated on Vimeo-90K-T and UDM10 datasets. Could you complete it for a fair comparison?\n- Figure 7 needs more explanation.\n\nQUESTIONS:\nDiffusion-based models usually show poor performance on PSNR (e.g., StableSR and Reshift), but SeeClear demonstrates a significant improvement. Could you analyze which parts of SeeClear contribute to this improvement?\nPlease refer to the weaknesses part above.\n\nLIMITATIONS:\nThe authors have largely addressed their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a diffusion-based video super-resolution method, and proposes Instance-Centric Alignment Module and Channel-wise Texture Aggregation Memory. The former leverages a pre-trained open-vocabulary segmentation model (i.e., OpenSeeD), which is utilized to perform alignment within video clips by modulating the spatial and temporal features. The latter leverages channel-wise attention and memory mechnism to better super-resolve the video frames. The results on publich benchmarks indicate that the proposed method achieves state-of-the-art perceptual performance.\n\nSTRENGTHS:\n1. The proposed method achieves state-of-the-art perceptual results on REDS4 and Vid4.\n\nWEAKNESSES:\nAlthough the proposed method achieves promising results on the public benchmarks, there are some concerns that greatly affect the rating of this paper.\n\n1. The presentation of the method needs to be improved. The readability of the paper is unsatisfactory. The technical details and the rationale behind it is not clearly described and explained.\n(a) The main figure (Figure 1) is ambiguous. It is hard to understand the workflow of the framework based on this figure. It is also hard to see the connection among different modules.\n(b) In the abstract, what is the \"conditional video generation\" (L6)? I do not see any pre-trained conditional video generation module in the described method. Maybe it should be rephrased.\n(c) In L206-207, what is the role of \"randomly initialized tokens\"? And what is specific role of the encoder-decoder module?\n(d) In L187-188, are the \"semantic tokens\" actually text embeddings ? What is the difference?\n(e) In L223, how to divide channels into different groups and what is rationale behind it?\n(f) It is hard to understand Eq. (16), (17) and (18). From (17) and (18), it seems T_j is used to calculate itself, which is confusing.\n(g) The choice of mathematical notations is sub-optimal and confusing.\n(h) In L149, I think the \"belta_t\" should be \"yita_t\".\n\n2. The novelty of this paper is limited.\n(a) Some of the modules are based on existing methods. For example, the way of introducing semantic features is similar to SFT (but no comparison in the paper); the multi-frame self attention is from [21].\n(b) The proposed blurring ResShift is a modification version based on ResShift, but the rationale behind it is not fully explained. Also, there is no direct ablation.\n\n3. The comparison with other related methods are not thorough. \n(a) The authors should explicitly compare with ResShift [33], since residual shifting technique is also exploited (but no citation in L48). Also, there is no comparison with it in Sec. 2.2.\n(b) The authors should compare with Upscale-A-Video [36], another diffusion-based video super-resolution method. Also, it is recommended to compare the performance of [36].\n(c) The authors should compare with SFT[28], another method also leveraging semantic segmentation information.\n\n4. The proposed method is not fully ablated. There is no direct ablation for exploitation of DWT and blurring resshift.\n\n5. Some of the statements could be inappropriate. \n(a) In L35-36, I think it is hard to reach the given conclusion from [8]. Please elaborate.\n(b) The naming of \"semantic distiller\" could be inappropriate. The pre-trained semantic segmentation model is directly leveraged and frozen. I don't see any distillation.\n\nQUESTIONS:\n1. In the abstract, what is the \"conditional video generation\" (L6)? I do not see any pre-trained conditional video generation module in the described method.\n2. In L206-207, what is the role of \"randomly initialized tokens\"? And what is specific role of the encoder-decoder module?\n3. In L187-188, are the \"semantic tokens\" actually text embeddings ? What is the difference?\n4. In L223, how to divide channels into different groups and what is rationale behind it?\n5. It is hard to understand Eq. (16), (17) and (18). From (17) and (18), it seems T_j is used to calculate itself, which is confusing. Please elaborate.\n6. In InCAM, the way of introducing semantic features seems similar to SFT. Please compare with it and illustrate the significance of the proposed module.\n7. Please provide comparison with the following related methods, and illustrate the novelty of the proposed modules. \n(a) The necessity and rationale of blurring ResShift, and the ablation study.\n(b) Upscale-A-Video [36]. And please compare performance with it quantitatively.\n8. In L35-36, I think it is hard to reach the given conclusion from [8]. Please elaborate.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations in Section D.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework for video super-resolution (VSR) that improves temporal coherence and high-resolution detail generation. The proposed method, SeeClear, integrates a Semantic Distiller and a Pixel Condenser to extract and upscale semantic details from low-resolution frames. The framework employs an Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) to enhance inter-frame coherence and incorporate long-standing semantic textures. The methodology also introduces a blurring diffusion process with the ResShift mechanism to balance sharpness and diffusion effects. Experimental results show that SeeClear outperforms state-of-the-art diffusion-based VSR techniques in terms of perceptual quality and temporal consistency.\n\nSTRENGTHS:\n1. The SeeClear framework introduces a combination of semantic distillation and pixel condensation, which significantly enhances video super-resolution.\n2. The Instance-Centric Alignment Module (InCAM) and Channel-wise Texture Aggregation Memory (CaTeGory) improve the temporal coherence of the generated high-resolution videos.\n3. The integration of blurring diffusion with the ResShift mechanism effectively balances sharpness and diffusion, leading to high-quality detail generation.\n\nWEAKNESSES:\n1. While the method demonstrates robust restoration capabilities, it may still struggle with accurately restoring tiny objects or intricate structures, especially under severe degradation conditions.\n2. The method has been tested primarily on specific benchmark datasets. Its performance in real-world applications, where video degradation processes are more varied and unpredictable, remains to be thoroughly evaluated.\n3. The experiments are not sufficient and should be improved.\n\nQUESTIONS:\n1. The performance of the proposed method is not significant. In Table 1, the improvement is very marginal or is worse than other methods. Moreover, in Figure 4, the generated texture is comparable to other methods.\n\n2. It would be better to compare more methods. (a) Transformer-based (e.g., VSR Transformer) or RNN-based method (e.g., BasicVSR++); (b) Diffusion-based image restoration methods (e.g., DDRM, DDNM, DeqIR, etc). (c) Compare methods trained with more frames (e.g., VRT-16).\n\n3. The authors should compare the efficiency, including model size, training/inference time and FLOPs. The efficiency comparisons can demonstrate the effectiveness of the proposed method.\n\nLIMITATIONS:\nPlease refer to the details above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}