Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the alignment problem in video super-resolution. It conducts a series of experiments to show that attention plays a key role for the problem. Using larger window size for unaligned videos can also achieve good performance. Besides, to reduce computation cost, it proposes a patch alignment strategy that aligns different patches. It achieves competitive performance on multiple video super-resolution benchmark datasets.

strengths_and_weaknesses: Strengths:

1, interesting finding on the relationship between attention window size and motion magnitude.

2, a patch alignment strategy that crops and aligns patches rather than pixels





Weakness:

1, Previous attention-based method [21] proposes to use attention for implicit alignment, along with which feature alignment is used to deal with misalignment outside of the attention window. The key idea seems similar: attention can deal with misalignments.

2, When the motion is small, the feature after alignment is largely still within the attention window. Why no alignment is better for small motions? Is there any more evidence (except for the final PSNR comparison)? Besides, due to the shifted window strategy, one pixel should have direct interactions with its neighbouring 12x12 pixels, why no alignment is worse when pixel movement magnitude is larger than 8 in Fig. 4 (a)?

3, If alignment is unnecessary, global attention (largest window size) should achieve best performance.  Why does the PSNR of [3] (using global attention) is far from good? How about the performance of using global attention for the proposed framework?

4, The patch alignment strategy is very similar to [24], which also crops patches from image features. [24] is even more flexible than this paper, as it samples several patches at a time. Attention is used in a similar way.

5, The patch alignment strategy is also very similar to [1] that samples tokens (patches) for attention according to optical flow.

6, Contradictory experiments results. When using nearest sampling in warping, the optical flow cannot be updated during training as there is no gradient. According to your conclusion, using an updated smooth flow is better than using the original pre-trained flow.

7, If we need to find corresponding patches for each window, do we need to do it again when the windows are shifted?

8, Some conclusions are not new. For example, the superiority of feature warping to image warping is already well-studied in previous works.

9, No runtime and FLOPS comparison.
questions: See weaknesses.
limitations: The authors have discussed the limitations and potential societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper rethinking the alignment in video super-resolution (VSR) Transformer and has two observations. (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. Based on these two observations, the authors remove the alignment in VSR, and propose a patch alignment to reduce the computational burden. Many experiments demonstrate the state-of-the-art performance on REDS.
strengths_and_weaknesses: Strengths:
1. The authors conduct extensive experiments to explore the role of the alignment in VSR and have two observations.
2. The proposed VSR Transformer method achieves the state-of-the-art performance on multiple datasets.

Weaknesses:
1. For the first observation, VSR-T and VRT are able to  utilize multi-frame information.
2. The second observation "existing alignment methods are sometimes harmful to VSR Transformers" is two strong.
questions: 1. The authors state that "existing alignment methods are sometimes harmful to VSR Transformers". The importance of the alignment has been proved in most existing VSR methods. The performance of the alignment highly depends on the specific architecture of VSR Transformers. The observation may be only suitable for the proposed Transformer in this paper.

2. The proposed VSR Transformer implicitly conduct the alignment and track the motion between frames. The authors provide visualization of local attribution map to highlight the pixels in each frame. It would better to explain from mechanism why the proposed VSR Transformer implicitly can align the feature in each frame.

3. Recently, some studies demonstrate that large window size has better performance. In Table 4, the performance gain on REDS4 may mainly come from the large window size of 16x16. To verify the effectiveness of the patch alignment. It would better to report results with the window size of 8 in the supplementary. 

4. Patch alignment is sensitive to the similarity metric. When the adjacent frames has different illumination and scale, the performance of patch alignment may drop. How to relieve this issue in the experiment.

5. The authors state that the proposed method can del with large motions. However, the performance is worse than VRT on Vimeo-90K-T which has different degrees of motions. Vimeo training set has 7 frames

6. How many days for training PSRT-recurrent on the RED dataset? Besides, it would be better to report the inference time.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work explores the alignment modules in Transformer-based video super-resolution (VSR) methods. The research begins with experiments finding that VSR Transformers can directly utilize the unaligned videos and existing alignment methods can harm VSR Transformer. To solve this problem, this paper proposes a new and efficient alignment method named patch alignment, which aligns image patches instead of pixels. The experiments show the proposed method achieves state-of-the-art performance. 
strengths_and_weaknesses: Strengths:
1) The paper is easy to follow. The motivation is clear and straight, and the main problem is well defined.
2) The problem of alignment for VSR Transformers is interesting, this paper shows VSR Transformers can utilize the unaligned videos.
3) The performance is state-of-the-art, the proposed method achieves the first or second best performance on each benchmark.
4) The method is efficient, there are not many extra parameters and computations introduced.

Weakness:
1) The main weakness seems to be the limited novelty and contribution. The main contribution of this work is the patch alignment method, but the method of aligning the patch instead of pixel seems too simple. The problems and solutions presented are relatively narrow, and I'm afraid this doesn't contribute much. The problem and solution proposed are relatively narrow, and this may not contribute much.
2) The experiment is not very clear, I see that the VSR Transformers with patch alignment method achieves good performance, but I cannot find how much improvement the patch alignment brings compared with the baseline method.

questions: My main concern of this paper is the novelty and contribution. The problem of alignment does not seem broad and relevant, and I'm not sure if this research will contribute enough to the field. Another concern is that I did not find a clear experiment of how much improvement the patch alignment method brings compared to the baseline method, so I’m not sure if it's efficient enough.
limitations: Yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper rethinks alignment in video Super-Resolution Transformers, demonstrates that existing alignment methods are unnecessary or even harmful to the VSR Transformer, and will cause damage to the sub-pixel information, and proposes a simple but effective alignment method.
strengths_and_weaknesses: Strengths
1.	There is a certain novelty to this work, reconsidering the alignment in video super-resolution Transformers, while other methods retain the complex alignment module
2.	The paper makes a reasonable analysis of the experimental results, proposes that the existing alignment methods are unnecessary or even harmful to the VSR Transformer, and will cause damage to the sub-pixel information.
3.	Proposed a effective alignment method --- patch alignment, and experimental results show that this method achieves state-of-the-art VSR performance.

Weaknesses
1. Details about patch alignment are unclear. Section 5.1 could have further additions, in particular how to find the corresponding patches in the supporting frames for each patch, and how to move the patch to the corresponding position.


questions: 1.	It’s not clear about the explanation of Figure 4,how to get the result of Figure 4,especially Figure 4(c).
2.	There should place one line space before the figure caption and one line space after the figure, the layout of the figure is too crowded.
limitations: This paper use bicubic interpolation to produce LR video frames. It could be further discussed whether the experimental results will be affected by different degradation processes.
review_title: null

Meta-review: metareview: This paper re-thinks the role of alignment in video super-resolution based on transformer models. The video alignment is costly which may need manual efforts. This paper proposed several inspiring and counter-intuitive remarks, such as that alignment is unnecessary and may be harmful to the transformer model. The authors presented a new model with only patch alignment instead of the pixel ones and larger window size, which achieves non-trivial improvements over the SOTA methods. Most of the reviewers agreed with the contribution and significance of this paper to the community.
confidence: Certain
Decision: Accept