Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tries to integrate the advantages of parallel methods and recurrent methods, and proposes a recurrent video restoration transformer (RVRT). RVRT first divides the video into fixed-length video clips and refines the subsequent clip feature based on the previously inferred clip feature and the old features of the current clip from shallower layers. Within each clip, the authors use a self-attention mechanism to fuse the feature. Across the clips, the authors propose guided deformable attention (GDA) to achieve video-to-video alignment. The proposed RVRT achieves state-of-the-art performance in video super-resolution, video deblurring, and video denoising. 
strengths_and_weaknesses: - Strengths:

1. Parallel methods and recurrent methods are dominant strategies in the VSR area, and most researchers only adopt one strategy in their work. Therefore, the attempt of adopting these two strategies in a single framework is valuable.

2. From Tab. 5, Tab. 7, Tab. 8, and Tab. 9, RVRT achieves state-of-the-art performance in video super-resolution (BI-REDS/Vid4, BD-Vid4), video deblurring, and video denoising. 

- Weaknesses:

1. The effectiveness of the core idea (which takes advantage of parallel and recurrent methods) is not well verified. 

    1.1. From Tab. 1, when N=3, the performance is worse than N=2. When N=3, the frames within each clip can aggregate more information (the information of the other 2 frames) than N=2. Besides, it also aggregates more information from the neighboring clips (by aggregating the features of 3 frames). Therefore, it is unreasonable that the performance of RVRT saturates when N=3. More detailed analyses (visualization or experiments) are supposed to be provided.

    1.2. From Tab. 6, when BasicVSR++ is equipped with RSTB, the performance gains 0.26dB and achieves 32.61dB. Compared to RVTB, its performance is only 0.14dB lower with 1.2M fewer parameters. Therefore, it is hard to distinguish the effectiveness of the core idea. What is the performance of BasicVSR++ + RSTB, whose parameters are the same as RVRT? 

2. Some results are confusing. Wait for the authors’ explanations/analyses.

    2.1. From Tab. 5, it is clear that the performance of RVRT is better than VRT on Vid4 and REDS. However, on Vimeo-90K-T and UM10 datasets, RVRT achieves inferior performance. 

    2.2. From Tab. 7, when sigma is 10 and 20, the performance of RVRT is inferior to VRT. However, when sigma is larger than 30, the performance of RVRT is comparable to or better than VRT. 

questions: See questions in Weaknesses
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposed a recurrent video restoration transformer. It divides the video into multiple clips and the correlations among clips and inside clips are explored jointly. Within each clip, different frame features (2 frames in this work) are updated with implicit feature aggregation. Across clips, guided deformable attention is proposed to perform clip-to-clip alignment. Experimental results demonstrate that the proposed method achieves SOTA results on video SR, delurring, and denoising. 
strengths_and_weaknesses: Strengths:
1.	The proposed guided deformable attention for clip-to-clip alignment is the main technical contribution of this work. The ablation study also demonstrates the effectiveness of the proposed module. 
2.	The proposed method can efficiently utilize long-term correlations 
3.	The proposed method works well on multiple video restoration tasks. 

Weaknesses:
More explanations about the proposed module are expected. For example, from table 3, the MLP module is essential for high performance (0.17 dB gain). Why? If we introduce the MLP module to DCN, will it outperform the proposed GDA?  In other words, the gain of GDA over DCN comes from which part of GDA? In addition, since GDA utilizes flow for guidance, a more fair comparison with GDA should be the flow-guided DCA, which was utilized in  BasicVSR++. 

In Eq. (5), the predicted M offsets are averaged. Why? In DCN, we utilize multiple offsets for the same pixel and it is demonstrated to be more effective than the single-offset-based optical flow (each pixel only has one offset). What is the value of m in experiments?  Are the results sensitive to different settings of m? 

As shown in Table 7, the denoising performance of the proposed method is worse than the compared methods when the sigma is smaller than 20. Why? 



questions: The authors are suggested to give responses for my concerns about the proposed GDA module. Can it be directly plugged into benchmark video restoration networks (i.e., replacing their original alignment modules) to further improve their performance? If yes, the impact of this work will be great. 
limitations: Yes, the authors have addressed their limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a general recurrent video restoration model for various video restoration tasks, including video SR, video deblurring and video denoising. It divides the video sequence into multiple video clips and deal with them by two strategies: globally, it propagates clip features in a recurrent way to reuse model parameters and save memory; locally, it jointly updates different frame features from one clip in parallel. Besides, since it processes the video as clips, it proposes the guided deformable attention for video clip-to-clip alignment. Extensive experiments on various benchmark datasets show the effective and the generalizability of the model.
strengths_and_weaknesses: The proposed RVRT architecture is novel, effective and technically sound. Unlike previous methods that are either recurrent or parallel, RVRT takes the advantages of both directions and alleviates their corresponding problems. As validated in ablation studies and comparisons with existing methods, RVRT makes a good trade-off. It is much smaller, quicker and more memory efficient than parallel methods, and, still, it achieves state-of-the-art performance on benchmark datasets. RVRT also provides a useful way to tackle the information loss and noise application problems that are inherent to recurrent models, as proved in the supplementary. Furthermore, it proposes a guided deformable attention module that is directly applicable for clip-to-clip alignment, in which relevant features from different frames are aggregated dynamically and efficiently. This module is well illustrated and is proved to be effective in experiments. 

Overall, this paper makes a good contribution to video restoration and conducts extensive experiments to support its arguments. It provides an alternative video sequence modelling option, rather than focusing on the design of specific modules. It should also be an interesting paper for the wider community. I think this paper should be accepted.


Pros:

1, It proposes a novel architecture that extracts features locally in parallel and accumulates information globally in a recurrent way. It has many benefits due to reduced sequence length, larger hidden state and local parallel processing.

2, It proposes a one-stage guided deformable attention module for dynamic and global feature aggregation in clip-to-clip alignment. It fits the overall architecture well.

3, The proposed method is evaluated on most standard video restoration benchmarks (8 different datasets), and the performance is solid and convincing. In particular, on deblurring and denoising, it remains the surprising performance of transformer models and outperforms most of its competitor by up to 2.27~2.37dB. 

4, The paper writing is fairly good.

Cons:

1, What are the training time and memory cost for different tasks?

2, Typo in the caption of Fig. 2.

3, The comparison of temporal modelling ability is interesting and is suggested to be put in the main paper. 
questions: See weakness above.
limitations: The authors have discussed the limitations and potential societal impact.
review_title: null

Meta-review: metareview: The paper introduces a recurrent video restoration transformer with guided attention, which combines recurrent and parallel methods in some extent. All reviewers found that the proposed method is sound and that experiments are adequate to demonstrate the effectiveness of the proposed method.
confidence: Certain
Decision: Accept