Rethinking Alignment in Video Super-Resolution Transformers
Authors: Shuwei Shi
Jinjin Gu
Liangbin Xie
Xintao Wang
Yujiu Yang
Chao Dong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video Super-Resolution
Transformer
Self-attention
Alignment

Abstract: The alignment of adjacent frames is considered an essential operation in video super-resolution (VSR). Advanced VSR models, including the latest VSR Transformers, are generally equipped with well-designed alignment modules. However, the progress of the self-attention mechanism may violate this common sense. In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations. Our experiments show that: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. These observations indicate that we can further improve the performance of VSR Transformers simply by removing the alignment module and adopting a larger attention window. Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions. Therefore, we propose a new and efficient alignment method called patch alignment, which aligns image patches instead of pixels. VSR Transformers equipped with patch alignment could demonstrate state-of-the-art performance on multiple benchmarks. Our work provides valuable insights on how multi-frame information is used in VSR and how to select alignment methods for different networks/datasets. Codes and models will be released at https://github.com/XPixelGroup/RethinkVSRAlignment.

Rethinking Alignment in Video Super-Resolution Transformers
Shuwei Shi Jinjin Gu Liangbin Xie2,5,6 Xintao Wang", Yujiu Yang', Chao Dong 2.311
Shenzhen International Graduate School, Tsinghua University
2 Shenzhen Institutes of Advanced Technology. Chinese Academy of Sciences
Shanghai AI Laboratory
The University of Sydney ARC Lab, Tencent PCG
5 University of Chinese Academy of Sciences
jinjin.gussydre gu@sydney. edu au, ssw200mails. tetteeuunnnaaaa edu. en, {chao. dong, 1b. xieeesiattt...... ac en, sintaowangetencentt com yang. yujiuusz. tsinghua. edu. cn
Abstract
The alignment of adjacent frames is considered an essential operation in video super-resolution (VSR). Advanced VSR models, including the latest VSR Trans- formers, are generally equipped with well-designed alignment modules. However, the progress of the self-attention mechanism may violate this common sense. In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations. Our experiments show that: (i) VSR Transformers can directly utilize multi-framm information from unaligned videos, and (ii) existing alignment methods are sometimes harmful VSR Transformers. These observa- tions indicate that we can further improve the performance of VSR Transformers simply by removing the alignment module and adopting a larger attention window. Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions. Therefore, we propose new and efficient alignment method called patch alignment, which aligns image patches instead of pixels. VSR Transformers equipped with patch alignment could demonstrate ofth-offthe-art performance on multiple benchmarks. Our work provides valuable insights on how multi-frame information is used in VSR and how to select align- ment methods for different networks/datassts Codes and models will be released at
com/ XP www.XPexxrooopp http//iennknniiinnneeeeee
Introduction
Video Super-Resolution (VSR) could provide more accurate SR results than Single-Image Super- Resolution (SISR) by exploiting the complementary sub-pixel information from multiple frames. The well-established paradigms of VSR networks usually include an alignment module to compensate for the motion of objects between frames. The alignment module is critical for CNN-based VSR networks, because the locality inductive bias of CNNs only allows them to utilize spatial-close distributed information effectively. Many VSR networks have achieved better performance by introducing more advanced alignment methods 43
39
Recently, the general paradigm of vision network design has gradually shifted from CNNs to Transformers [10 23 77.. Unlike the locality property of CNNs, the self-attention operation in
Transformers is very efficient for processing elements with spatially long-term distribution. VSR is
no exception: Transformers have also been introduced into VSR for better results B 22 24
However,
"Jinjin Gu and Shuwei Shi contribute equally to this work. CCoresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

these VSR Transformers still retain complex alignment modules. The ability of Transformers to efficiently process non-local information has not yet been exploited for inter-frame information processing.
In this paper, we rethink the role of the alignment module in VSR Transformers. We make two arguments: (i) VSR Transformers can directly utilize multi-frame information from umaligned videos, and (if) continuing to use the existing alignment methods will sometimes degrade the performance of VSR Transformers. For the first argument, we report that for Transformers using the shifted window mechanism (30 23], misalignment within a certain range does not affect the SR performance. while alignment only has a positive effect on the pixels beyond that range. The attribution map visualization results [T]] also show that Transformers without alignment modules behave similarly to CNNs with alignment modules. For the second argument, we quantitatively compare the effects of various alignment methods. Using alignment will only have negative effects for pixels without large misalignment (e.g., Vimeo-90K 477 ). We blame this phenomenon on the noise of optical flow and the destruction of sub-pixel information by alignment resampling operations. Although implicit alignment based on deformable convolution can minimize these negative effects, the additional cost of parameters and computation makes this approach no longer advantageous.
According to our findings, we can build an alignment-free VSR Transformer, which requires large attention window (>16) to cover misalignment between frames. However, enlarging the window size will lead to higher computational costs, making this approach no longer feasible. As a better alternative, we propose a simple yet effective alignment method called Patch Aligmment, which aligns image patches instead of pixels. It will find the corresponding patches in the supporting frames and compute self-attention among them. This method uses simple crop-then- to compensate for situations where the misalignment exceeds the Transformer's 's attention window. Experiments show that the proposed method achieves state-of-the-art VSR performance with a simple design and fewer parameters.
strategy
2 Related Works
VSR networks. The uniqueness of the VSR task lies in the utilization of inter-frame sub- sub-pixel information [29]. Due to the motion of objects between frames, the computation and compensation for this motion often the focus of VSR research. The pipeline of most VSR networks mainly includes an alignment module, feature extraction module, a fusion module, and a reconstruction module. These networks are trained using a large number of tow-resolution (LR) and high-resolution
(HR) video sequences. The paradigm of early methods 225 [5] 28 16 47 14
26 can be summarized as: first estimate and compensate object motion (using optical flow), then pass through different feature extraction (36 46 19, fusion [38 2 4. and reconstruction modules 37 respectively. These methods can be further divided into methods for compensating motion in the image space (image alignment) and that in the feature space (feature alignment) 4 42). As Chan et al. [4] claims, feature alignment can achieve better results because inaccuracy in flow estimation may be mitigated after feature extraction.
However, the above methods generally suffer from inaccurate optical flow estimation and defective alignment operation, resulting in noticeable artifacts in the aligned images (39]. With the invention of Deformable convolutions 9 CNN networks also have the ability to model geometric transformations. Deformable convolutions gradually replace explicit warping operations in VSR networks with their learnable and flexible properties [43] [1 39]. EDVR (4) integrates deformable convolution in the VSR network for the first time and aligns features at multiple scales. Chan et al. [5] incorporate optical flow as a guidance for offsets learning. The latest BastcVSR++ 6 based on the bidirectional loop structure also uses the alignment based on deformable convolution.
Differently, our paper aims to study Transformers' ability to model multi-frame information implicitly without using alignment. Before Transformer, attempts have been made to achieve the same effect in CNN, i.e., the VSR methods without alignment. Both 2D [32 48 and 3D convolutions 440
13 20 [6] are used to extract correlations among frames. Similar to Transformers, some methods use correspondence calculation to build inter-framt frame connections without alignment. For example, MuCAN (21) uses temporal multi-correspondenc aggregation module and cross-scale non-local- correspondence aggregation module to replace explicit alignment. Yi et al. [50) introduce an improved non-local operation avoid the complex alignment procedure. Generally speaking, models without explicit alignment are less effective or rely on special module design. However, the Transformers discussed in this paper could efficiently handle unaligned frames without additional special design.
2

---

Image Alggmment
Alggment
Deformable Corvolution
Vill Care
Supoorting
Festure Extraction
Frame
Reconsitruction
MFSAB
MFSAB
Vill Paaee
Rafarence Framo
Supporting Frams
N MFSAB
Video Resolutton Transformer
Multi-Frama Salf Attention Block (MFSAE)
Figure 1: A brief illustration of the VSR transformer network used. The illustrated structure is based on the sliding window mechanism. We mark the position of the existing alignments methods.
1x7
Vimeo-900 90K
3.0
2.5
1e6
REDS
1.0
0.14-
Pi trained Spshic
Mten Total Varatton
2.0
0.13
100
0.6
Frebabeing
Paal Courte
2.5
0.6
Probsahiy
0.12
Cummrative Probabiity
2.0
Cumilatbe Probanility
0.50
0,2
0.00
0.0
0.0
0.0
25
Pod Movemont Magnitud
Pudi Movement Hagnitude
3e5
Figure 2: The distribution of the movement for the Vimeo-90K 47] and REDS 33 test The distribution of pixel movement exhibits long-tailed feature. of total variation of the fine-tuned flow.
Figure 3: The curve
The REDS dataset contains a larger movement.
Transformer networks. Transformers [41] are new type of networks characterized by self- attention operations. A Transformer is considered to have a parameter- parameter-indeeenddee independent global receptive field within the scope of self-attention. Dosovitskiy al. [10] first introduces Transformer into image recognition by projecting large image patches into a sequence of tokens. Inspired by the success of vision Transformers, many attempts have been made employ Transformers for low-level vision tasks [49 52 45 53, 8 As a representative model, Liang et al.proposes SwinIR 23 that integrates powerful shifted window strategy and achieves promising results. Transformer- based models have also been proposed for VSR. Cao el al. (3] employ the spatial-temporal self-attention on patches to perform implicit motion compensation. Liang al. [22) propose parallel frame prediction Transformer to model oangrange temporal dependency. Both the above Transformers consist of alignment modules. VSRT 3 uses feature alignment method and VRT (22) employs complex deformable convolution to perform forward and backward alignment. In this paper, we show that these complex alignment operations are unnecessary or even harmful for VSR Transformers.
3 Preliminary Settings
In this section, we describe the settings used in our experiments in detail, including the Transformer architecture, alignment methods, datasets, metrics and implementation details. If the readers already familiar with the common VSR settings and VSR networks, it is okay to skip this section and go to Section 4 for experimental results and analysis. Due space constraints, record the detailed setup of the different experiments in Appendix ?? to facilitate the readers to reproduce our results.
The VSR Transformer architecture. We first describe the basic VSR Transformer backbone used in this study, which follows the sliding window design. Transformer based on shifted window mechanism [23] is proven to be flexible and effective for image processing tasks. We only make minimal changes when applying to VSR maintain its good performance without the loss of generality. This model is illustrated
Figure I The VSR Transformer takes a reference frame
I' and 2n adjacent supporting frames { ..... 1'-1 11+1 ..... [ } input, and outputs the super-resolved reference frame nn the beginning, feature extraction module is used to extract features for the subsequent Transformer. We use single 2D convolution layer for feature extraction. The extracted features are denoted as 2n + feature maps {X*-" ".......""" X
X',..,,'++n)
Then N Multi-Frame Self-Attention Blocks (MFSAB) are used the backbone network. The MESAB is modified from RSTB in SwinIR (3), which contains two LayerNorm [I] layers, a head self-attention layer and multi-layer perceptron layer. We mainly modify the self-
3

---

(a) VSR Transformer (windew size B)
(b) VSR Transformer (window size 12)
(c) VSR Transformer (feature alignment)
No aiggemers better VSR Transformer
wandowr sizee of
i slgrrent is better
wiigmmmnt
Differeces in Square
b. 1.
=8-
Aligmment
better
Algnmmnntss is beher
J VBR Transfomer
-10
Algement better
condioe sdne
Transforer
15
15 20
25
20 Pixal Movement Magnitude
10
15
Pisol
Magnitude
Pissl Movement Magnitude
Figure 4: This figure illustrates the performance differences between VSR Transformers with and without alignment module for different pixel movement conditions. Parts greater than zero indicate better performance without alignment.
attention layer to make it suitable for the VSR task. Given these 2n- 1 feature maps of the frames with size H x W C' (H, W and C are the height, width and feature dimension), the shifted window mechanism first reshapes the feature maps of each frame to ArT M² x C' features by partitioning the input into non-overlapping M x M local windows, where !WW is the total number of windows. We calculate self-attention on the features in the windows corresponding to the positions in different frames. Therefore, (2n- + 1) 1)M features are involved in each standard self-attention operation, and we concatenate features from different frames to produce a local window feature
HW
M
X E IR ((2++1)MM** xC, In each self-attention layer, the query Q, key K and value V are computed as Q = XW K XW V = XWV, where Wº, W WVV E RRCDD are weight matrices, and D is the channel number of projected vectors. Then, we use Q to query K to generate the attention map x(2++))a1 x(26+))) where B is the learnable relative positional encoding. This attention map A is then used for the weighted sum of (2n + 1)M vectors in V. The multi-head settings are aligned with SwinIR (3] and ViT [TO. In the final reconstruction module, only the features of the reference frame are used to generate the SR result. We use the pixel-shuffle layer (37] to upsample the feature maps. Besides sliding window based VSR Transformer, the MFSAB can also be applied to recurrent frameworks (6). We describe the details in Appendix ??.
A sof teatttxx/////a + +B) [(2++1)M
A Alggnment methods. Alignment methods follow a long line of research. To investigate the role of alignment, we need to implement and compare different alignment methods. These alignment methods can be classified into four types, and their respective representative methods are included in experiments. We next describe their characteristies and then describe our implementation.
Image alignment is the earliest and most intuitive alignment method, which is also the easiest to use. Image alignment relies on the explicitly calculated optical flow between frames. According to the estimated frame motion, different frames are aligned by warping operation. Following the existing successful experience [47 4. we use the SpyNet
[35] to estimate the optical flow, and fine-tune the SpyNet simultancously during training. The resampling method used is the bilinear (BI) method.
Feature alignment also estimates the optical flow but performs the warping operation on the deep features instead of on the images. The flow estimation module still uses SpyNet, which is optimized during training. In addition to the shallow 2D convolution in Figure] D we add five additional residual blocks [12 to extract deep features.
Deformable convolution (DC) based methods use learnable dynamic deformable convolution for alignment. Almost all state-of-the-art VSR networks use deformable convolution to perform the alignment. We employ the flow-guided deformable convolution (FGDC) alignment used in BasicV [6] and VRT [2] as the representative method.
No alignment. The input is processed directly using the VSR Transformer.
Datasets and metries. In the VSR literature, the REDS 33 and the Vimeo-90K 47 datasets are the de-facto benchmarks. REDS has 270 available video sequences, each containing 100 frames. We follow the common splitting methods and split the data into training (266 sequences) and testing S
(4 sequences) sets. Vimeo-90K 90K contains 64,612 and 7,824 video sequences for training and testing, respectively. Although REDS and Vimeo-90K are widely used benchmarks, these two datasets have different motion conditions. The motion in the limeo-90K dataset is generally small, and movement magnitudes of 99% pixels are less than 10 (for each clip, we measure the motion of the 4th and the 7th frames). Differently, there are large motions in the REDS dataset. There are at least 20% pixels that have movement magnitudes larger than 10 (for each clip, we measure the motion of the 3rd and the 5th frames). The distributions of the movement conditions for these two datasets are shown in

---

We use Table 1: Quantitative comparison of different VSR methods. The results marked with achieve similar performance as no alignment. This is due to the vanishing of optical flow in this experiment. Details are shown Figure and discussed in Section 4.2
Exp- Index
Vimeo80K-TT
REDS4
Method
Alignment
Remark
PSNR
SSIM
PSNR
SSIM
VSR CNN
Image alignment
Finetume flow
36.13
0.9342
29.81
0.8541
VSR-CNN
No alignment
36.24
0.9159
28.95
0,8280
VSR Transformer
Image alignment
Fix flow
36.87
0.9429
30.25
0.8637
VSR Transformer
Image alignment
Finetune flow
37.44
0.9472
30.43
0.8677
VSR Transformer
Feature alignment
Finetume flow
37.36
0.9468
30.74
0.8740
VSR Transformer
No alignment
Window size
37.43
0.9470
30.56
0.8696
VSR Transformer
No alignmmntt
Window size 16
37.46
0.9474
30.81
0.8745
Vilidation Performance on VimenGKK
15050 Iteration
25000 Beration
30000 Neration
50000 Iteration
11000]
23000
Jooce
16030
VSR Trane Praneformer
37.0
PVSR
25.-
V3R-CNN
55.0
VISV rebuiggment
VSS Trurs sul selle signnert fow fore signnssr
107
10
Hermooos
Pool Movement Magnitude
Figure 5: The figure on the left shows the validation curves tested on Vimeo-90K test set. On the right are the histogram distribution of the estimated optical flow at different training iterations. The moment when the optical flow distribution changes corresponds to the moment when the performance of the VSR Transformer with alignment increases.
Figure 2 In experiments, we mainly study x4 VSR task. We use bicubic interpolation to produce LR video frames. Peak signal-to-note ratio (PSNR) and structural similarity index (SSIM) are used for evaluation.
Implementation. We implement all these methods using the BasicSR framework 44
the Charbonnier loss [18] as the training objective. The Adam optimization [77 method is used for training with 31 0.9 and B3 0.999. The initial learning rate is set to 4 x 10 4, and a Consine Annealing scheme [3]] used to decay the learning rate. The total iteration number set to 300,000.
The mini-batch size is 8, and the LR patch size is 64 x 64. The experiments are implemented based on PyTorch 34 and conducted on NVIDIA A100 GPUs.
Rethinking Alignment
In this section, we conduct experiments based on the above preliminary settings. We use four questions to guide our rethinking. We present our experimental results and analysis in each subsection.
4.1 Does alignment always benefit VSR Transformers?
In previous VSR research, we often judge the pros and cons of the method based on the average performance on the test set. But for the VSR Transformers, the performance of alignment inside the local window different from that outside the window because of the limited range of self-attention. A finer perspective on how different methods perform different data may help us understand the effect of aligmment. We investigate the performance difference between the VSR Transformer with and without alignment for different pixel movement conditions. Figure shows the results tested on the REDS dataset and Table reports the quantitative results for these experiments.
One can draw the following observations. First, as can be observed from Figure 4 (a), for pixels with small movements, VSR Transformer can achieve good results without alignment. Using image alignment in these cases brings negative effects. This range of pixel movement is related to the window size used by the VSR Transformer. As there is no locality inductive bias the processing of pixels within local window, the Transformer can handle misalignment in this range. Second, as the movement increases, the information required by VSR exceeds the scope of the local windows. In these cases, image alignment can improve performance. Third, according to Figure 2 the movement
5

---

VSR Transforor
VBR CNN image Aighment
VSR CNN No Algemment
The Altibutionn larget Windows
Figure The attribution results of different models. We use the local attribution map to highlight the responsible pixels in each frame for the SR of the selected red window.
Table 2: Comparison of VSR Transformers with different alignment methods on the REDS4 dataset.
Aligament Method
Position
Resampling
Params. (M)
REDS4
No Ali
Img. Ali.
Feat Ali.
FGDC
Img.
Featt
BI
NN
PSNR / SSIM
12.9
30.92 / 0.8759
7
30.84/0.877
14.8
31,06 / 0,8792
16.1
31,11 0.8880
of about 70% pixels is less than eight on the REDS test set. This makes the model without alignment still perform better than the image alignment on this test set.
We then increase the size of the shifted window, and conduct the same experiment. The result is shown in Figure (b). We can that VSR Transformer can handle larger range of unaligned pixels with larger window size. This shows that the processing capability of VSR Transformers for unaligned frames is related the size of the shifted window, and also implies that this capability mostly relies on the self-attention mechanism. To investigate whether better alignment methods can eliminate the negative effects of alignment on small motion pixels, we use feature alignment to conduct the same experiment, and the result is shown in Figure (c). As one can see, feature alignment narrows the gap between alignment and no alignment on VSR Transformer, but it still has negative effects on the pixels of small motion.
In summary, the VSR Transformer can handle misalignment within certain range, and using aligmment at this range will bring negative effects. This range is closely related to the window size of the VSR Transformer. However, alignment necessary for motions beyond the VSR Transformer's processing range.
4.2
What kinds of flow are better for VSR?
Although using optical flow for alignment can have negative impact, different flows can also lead to differences in performance. This inspires us to think about what kinds of flow are better for VSR? According to Table [ one can see that optimizing the flow estimator while training the VSR network will bring better results, because the flow estimator at this time learns the optimized flow for VSR 477. The property of this task-oriented flow can imply how the VSR Transformers use multi-frame information. Our first observation is that VSR Transformers tend to use smooth flow. The flow estimator SpyNet was pre-trained with the end-polt and-point-error (EPE) loss, which does not explicitly encourage smoothness. The non-smooth flow will introduce random noise to VSR and lose sub-pixel information. To study this problem, we investigate the change of the estimated flow while training the VSR Transformer with image alignment using the REDS dataset. We can see that the flow estimated by the fine-tuned SpyNet is getting smoother, which is reflected in the decrease of the average total variation, as shown in Figure 3 Smoother flow maintains the relative relationship of adjacent pixels in the aligned frames, thus facilitating VSR processing.
Although the fine-tuned flow estimator will improve performance, there is still a gap between image alignment with flow fine-tuning and no alignment on the REDS dataset. However, we observe different results on the Vimeo-90K dataset: image alignment with flow fine-tuning is almost identical to no alignment. By observing the distribution of the estimated flow, we are surprised to find that the flow is slowly decreasing to 0 when the final result is almost the same as no alignment. In Figure we show the validation curve of the related experiments and the histogram distribution of the fine-tuned flow. As one can see, there is large gap between the VSR Transformers with and without alignment in the early stage of training. When the training reaches 25,000 iterations, the fine-tuned flow gradually shifts towards the zero
with image alignment using Vimeo-90K. This is why

---

Figure 7: The pipeline of the proposed Patch Alignment method: @ partition the input frames to patches according to the window partition Transformer, calculate the mean motion vector for each patch, @ find the corresponding patches in the supporting frames, and 4 move the entire supporting patches to their corresponding position.
Reference Frame
Image Alignment
Patch Alignment
Figure 8:
The visualization comparison of image alignment and the proposed patch alignment.
direction. When training after 50,000 iterations, the fine-tuned flow vanishes. At this time, the VSR Transformer with alignment equivalent to the one without alignment. This phenomenon does not appear VSR-CNN. This experiment instructive. On the hand, most of the movements in the Vimeo-90K dataset are smaller than the Transformer's 's window size. According to Figure (a), alignment detrimental to training this situation. The fine-tuned flow estimator seems aware of this knowledge and learns to improve performance by forcing flow value to all zeros. On the other hand, the adaptability of the model surprising. The model will choose the policy that maximizes the training objective, even if that policy completely disables part of the network.
We now answer the question posed by this subsection. For VSR Transformers, the smooth flow is better if the alignment is necessary. For situations with small motions, the flow with all zeros the best for VSR Transformers.
4.3 Does Transformer implicitly track the motion between unaligned frames?
We already know that VSR Transformers handle specific range of mis- alignment, but do they track these movements implicitly? Can an alignment- like function be done inside the VSR Transformers? We next pretability tool to visualize the behaviour of the VSR Transformer. Local Attribution Map (LAM) I1 is attribution method aiming find input pixels strongly influencing network output. We first specify a target patch on the output image and then LAM generate the corresponding attribution maps. We will track the information used by the model, and see which parts of pixels in adjacent frames contribute the most. As objects move between frames, an ideal VSR network needs to track those movements and utilize pixels representing the same object. We show some representative results in Figure O It can be observed that even without the alignment module, the VSR Transformer can automatically change its attention to the most relevant pixels. Intuitively, the self-attention operation can detect the associations between pixels in different frames and utilize their additional information for VSR. This similar VSR CNNs with the alignment module. We also show results of VSR CNNs without the alignment module. Since CNN has the inductive bias of locality, can only focus on the area near the current position if no alignment used. Thus, alignment is essential module for CNN-based VSR methods.
HR
LR
inter-
Figure 10: Illustra- tion of sub-pixel in- formation.
4.4
Why do alignment methods have negative effects?
To understand the reasons for the negative impact of alignment, we need to know what sub-pixel information does VSR require. As shown in Figure 10, high-frequency information in HR frames is lost during downsampling, and only aliasing patterns are left in the LR frames. When the HR frames move, different LR frames are generated, and different aliasing patterns are produced. These patterns provide additional constraints for VSR. However, the inaccurate optical flow and the bilinear
7

---

Table The Ablation study of Patch Align- ment. We study the effect of different re- sampling methods (BI and NN) and differ- ent alignment positions (image space and feature space).
Position Img.
Resampling Bi
REDS4
Method
Feat.
NN
PSNR
SSIM
v
31.11
0.8800
Patch Alignment
31.00
0.8781
31.17
0.8810
Nearest
EDVR
BSsicVSR 4
Icon
Frame 013, Clip 011, REDS
VRT 22
BasicVSR++
Ours
GT
Figure 9: Visual comparison of VSR (x4) on REDS dataset.
resampling operation could corrupt these patterns. First, the inaccurate flow be viewed combination of the ground-truth flow and a random error term. Using such flow for aligmment randomly will change the LR patterns and cause information loss. Second, the bilinear resampling operation calculates the weighted average among four adjacent pixels, and the weights are inaccessible for the VSR model. At this time, the interpolation is irreversible. The VSR model can only process the transformed LR patterns and cannot access the original patterns, resulting in information loss. We conduct experiments to demonstrate the negative effects of optical flow and resampling methods. The results are shown in Table 2 Compared with image alignment, feature alignment improves performance by extracting part of the sub-pixel information before is corrupted by alignment. The flow-guides deformable convolution (FGDC) reduces the negative effects of alignment by enabling the network to model geometric transformations. Changing the resampling method to nearest neighbor (NN) also improves performance because the NN method can preserve the relationship between adjacent pixels and ignore the noise of flow estimation to a certain extent. As can seen, feature alignment using the NN resampling method achieves the same performance as the FGDC method but with significantly reduced number of parameters.
5 Patch Alignment
According to our observations, inaccurate flow and the resampling operation will impair the utilization of the inter-frame information. To overcome this problem, simply increasing the Trans- former's s window size is a straightforward solution as Transformers are naturally good at modelling unaligned spatial dependencies within the local window. However, the time and space complexity of increasing the window size is at least On?²). We need more efficient alignment methods to improve performance for large movement pixels while introducing little negative impact and additional com- putational complexity. In this section, we propose a simple yet effective alignment method for VSR Transformers, called Patch Alignment.
5.1 Method
The pipeline of the proposed method is shown in Figure 7 This method does not align individual pixels but treats the image as non-overlapping patches. The partition of patches is consistent with the partition of the Transformer's 's local windows. We treat the patch as whole and perform the same operation on the pixels within the patch. In this way, the relative relationship between pixels kept, and resampling operations will not corrupt the sub-pixel information within the patch. We locate the movement of objects based on optical flow, yet we do NOT pursue precise pixel-level alignment. We calculate the mean motion vector in each patch and find the corresponding patches in the nearest neighbor resampling method to move
the supporting frames for each patch. Next, the entire supporting patches to their corresponding position in the reference frame. The benefits are three-fold. First, the nearest neighbor resampling ignores the fractional part of optical flow estimation and reduces the error caused by inaccurate flow estimation. Second, the entire patch is cropped and moved to the corresponding position, which preserves the relative relationship of the pixels within the patch and thus retains the sub-pixel information.
We show the comparison of image alignment with bilinear resampling and the proposed patch alignment method Figure g As can be seen, image alignment introduces blurry and artifacts to the aligned image that destroy sub-pixel information. Patch alignment retains more details that can

---

Table 4: Quantitative comparison (PSNR) and SSIM1) on the REDS4 [33] dataset, Vid4 177. Vimeo- 90K-T [47 dataset for 4x VSR task. Red indicates the best and blue indicates the second best performance (best view in color) each group of experiments.
Method
Frames
Params (M)
REDS4
Vimeo-90K-T
Vid4
REDS/Vimeo
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
EDVR
+
5/7
20.6
31.09
0.8800
37.61
0.9489
27.35
0.8264
VSR T
5/7
32.6
31.19
0.8815
37.71
0.9494
27.36
0.8258
PSRT-sliding
S/-
14.8
31.32
0.8834
VRT
6/-
30.7
31.60
0.8888
PSRT-recurrent
0.8964
VSR
15/14
31.42
0.8909
0.9450
27.24
0.8251
VSR 9
15/14
8.7
31.67
0.8948
37.47
0.9476
27.39
0.8279
VSR++
30/14
7.3
32.39
0.9069
37.79
0.9500
27.79
0.8400
VRT PSST-rrecureat
16/7
35.6
32.19
0.9006
38.20
0.9530
27.93
0.8425
16/14
13.4
32.72
0,9106
38.27
0.9536
28.07
0.8485
provide additional information for the VSR model. As we do not pursue pixel-level alignment, directly operating on patches will leave discontinuous artifacts along patch borders. But our experiments show that these discontinuities have little effect on VSR Transformers. Because these discontinuities do not appear in the local window of the Transformer, they do not affect the function self-attention. It further illustrates the importance of preserving sub- -ub-pieel information.
5.2 Experimental results
We test the patch alignment method with different alignment positions and resampling methods. The experimental settings and the network configurations are the same with experiments in Table The results are shown in 3 As can be seen, image-level patch alignment is already comparable to FGDC, while the latter uses 25% more parameters. Feature- patch alignment achieves the best performance among all the tested alignment methods. We conduct an ablation experiment using bilinear resampling to study the importance of NN resampling for patch alignment. can be seen that bilinear resampling leads to severe performance drop. It shows that the LR patterns retained by the NN method are critical for VSR.
We use patch alignment to build VSR Transformers based on the sliding window and the recurrent frameworks, namely PSRT-sliding and PSRT-recurrent. We test their performances on REDS and Vimeo-90K datasets. The results are shown in Table VSR Transformers with patch alignment can achieve state-of-the-ar performance with fewer parameters compared to other Transformer- based VSR methods (VSRT [3] and VRT (22 ). Compared to VSs (6). we use fewer frames in training, but achieve a 0.33dB improvement on REDS in terms of PSNR. Visual results of different methods are shown in Figure [
As one can see, in accordance with its significant quantitative improvements, the proposed method can generate visually pleasing images with sharp edges and fine details. By contrast, its competitors suffer from either blurry or lost details.
6 Conclusion
In this paper, we present two important conclusions about using Transformers in the VSR task: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. We also propose a new patch alignment method for VSR Transformers. The proposed method demonstrates the state-of-the-art performance for VSR.
Given the current literature, our results are interesting and inspiring. They challenge our common understanding of using Transformers to process multiple spatially misaligned images. First, the analysis of alignment can provide useful insights for VSR. We need to utilize inter-frame sub sub-pixel information, yet many image pre-process operations interfere with our utilization of this information. Second, these observations hint that Transformer can implicitly make accurate connections for misaligned pixels. Many low-level vision tasks can take advantage of this property, such as video restoration, reference-basee Transformers for these tasks, can no longer explicitly introduce the alignment modules the cost volume modules, but give full play to the powerful modeling capabilities of the Transformer itself.
SR, burst image processing, stereo matching, etc. When designing
This work was supported in part by the National Natural Science Foundation of China (Grant No. 61991450) and the Shenzhen Key Laboratory of Marine IntelliSense and Computation
Acknowledgement
1111660501))

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06433 2016.
arXiv preprint
[2] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE transactions on pattern analysis and machine intelligence, 43(3):933 948, 2019.
[3] Jiezhang Cao. Yawei Li, Kai Zhang, and Lue Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106. 06847, 2021.
Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Basicvsr: The search for essential components video super-resolution and beyond. CVPR, 2021.
[5] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Understanding deformable alignment in video super-resolution. In AAAI, 2021.
[6] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Lay. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In CVPR, 2022.
[7] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. NIPS, 2022.
[9] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[11] Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. In CVPR, 2021.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] Younghyun Jo, Scoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In CVPR, 2018.
[14] Ratheesh Kalarot and Fatih Porikli. Multiboot vsr: Multi-stage multi-reference bootstrapping for vidco super-resolution. In CVPRW, 2019.
[15] Armin Kappeler, Seunghwan Yoo, Qiqin Dai, and Aggelos K Katsaggelos. Video super-resolution with convolutional neural networks. IEEE transactions on computational imaging, 2(2):109-12:, 2016.
[16] Tac Hyun Kim, Mchdi SM Sajjadi, Michael Hirsch, and Bernhard Scholkopf. Spatio-temporal transformer network for video restoration. In ECCV, 2018.
[17] Diederik Kingma and Jimmy Ba. Adam: A methad for stochastic optimization. In ICLR, 2014.
[18] Wei-Sheng Lai, Jia-Bin Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, 2017.
[19] Dingyi Li, Yu Liu, and Zengfu Wang. Video super-resolution using non-simultancous fully recurrent convolutional network. IEEE Transactions on Image Processing, 28(3):1342 1355, 2018.
[20] Sheng Li, Fengxiang He, Bo Du, Lefei Zhang, Yonghao Xu. and Dacheng Tao. Fast spatio-temporal residual network for video super-resolution. In CVPR, 2019.
[21] Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia. Mucan: Multi-correspondence aggregation network for video ruper-reso In ECCV, 2020.
[22] Jingyun Liang, Jiezhang Cao, Yuchen Fan. Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vit: A video restoration transformer. arXiv preprint arXiv:2201.12288, 2022.
10

---

[23] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In CVPR, 2021.
[24] Jingyun Liang. Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy Ilg. Simon Green, Jiezhang Cao, Kai Zhang, Radu Timofte, and Lue Van Gool. Recurrent video restoration transformer with guided deformable attention. arXiv preprint arXiv:2206.02 0246, 2022.
[25] Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma, and Jiaya Jia. Vidco super-resolution via deep craft-c learning. In ICCV, 2015.
[26] Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool. Flow-guided sparse transformer for video deblurring In ICML, 2022.
[27] Ce Liu and Deqing Sun. On bayesian adaptive video super resolution. IEEE transactions pattern analysis and machine intelligence, 36(2):366 360, 2013.
[28] Ding Liu, Zhaowen Wang, Yuchen Fan, Xianming Liu, Zhangyang Wang, Shiyu Chang, and Thomas Huang. Robust video super-resolution with learned temporal dynamics. In ICCV, 2017.
[29] Hongying Liu, Zhubo Ruan, Peng Zhao, Chao Dong, Fanhua Shang, Yuanyuan Liu, Linlin Yang, and Radu Timofte. Video super-resolution based on deep learning: comprehensive survey. Artificial Intelligence Review, pages 1-55, 2022.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei. Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchieal vision transformer using shifted windows. In ICCV, 2021.
[31] Iiya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[32] Alice Lucas, Santiago Lopez-Tapia, Rafael Molina, and Aggelos K Katsaggelos. Generative adversarial networks and perceptual losses for video supcr-resolution. IEEF Transactions on Image Processing, 28(7):3312-3327. 2019.
[33] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on vidco deblurring and super-resolutio: Dataset and study. In CVPRW, 2019.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NIPS, 2019.
[35] Anurag Ranjan and Michael J Black. Optical flow estimation using spatial pyramid network. In CVPR, 2017.
[36] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recurrent video super-resolution, In CVPR, 2018.
[37] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[38] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. Detail-revealing deep video super-resolution In CVPR, 2017.
[39] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu. Tdan: cemporraly-deformaatn alignment network for video super-resolution. In CVPR, 2020.
[40] Du Tran, Lubomir Bourdey, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In /CCV, 2015.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
[42] Hua Wang, Dewei Su, Chuangchuang Liu, Longcun Jin, Xianfang Sun, and Xinyi Peng. Deformable non-local network for video super-resolution. IELE Access, 7:177734-177744 2019.
[43] Xintao Wang. Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In CVPRW, 2019.
11

---

[44] Xintao Wang. Ke Yu, Kelvin C.K. Chan, Chao Dong, and Chen Change Loy. BasicSR: Open source image and video restoration toolbox. http:://ggiauu..c.//itttttttttae http://///./tttt//aaiiii.ocm
2018.
[45] Zhendong Wang. Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu. Uformer: A general u-shaped transformer for image restoration. arXiv preprint arXiv:2 0360,, 2021.
[46] Zhongyuan Wang. Peng Yi, Kui Jiang, Junjun Jiang, Zhen Han, Tao Lu, and Jiayi Ma. Multi-memory con- volutional neural network for video resoluteso IEEE Transactions on Image Processing, 28(5):2530- 2544, 2018.
[47] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127(8):1 106-1 125, 2019.
[48] Bo Yan, Chuming Lin, and Weimin Tan. Frame and feature-context video super-resolution. In AAAI, 2019.
[49] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. CVPR, 2020.
[50] Pong Yi, Zhongyuan Wang. Kui Jiang, Junjun Jiang, and Jiayi Ma. Progressive fusion video super-resolution network via exploiting non-local
correlations. In ICCV, 2019.
[51] Xinyi Ying, Longguang Wang, Yingqian Wang, Weidong Sheng, Wei An, and Yulan Guo. Deformable 3d convolution for video super-resolution. IEEE Signal Processing Letters, 27:1500-1504, 2020.
[52] Syed Waqas Zamir, Aditya Arora. Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hoin King-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
[53] Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. arXiv preprint aXXw:2210001427, 2022.
Checklist
1. For all authors.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
12

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "szaeuSzPZ98", "reviewer": "Reviewer_CC5y", "summary": "This paper studies the alignment problem in video super-resolution. It conducts a series of experiments to show that attention plays a key role for the problem. Using larger window size for unaligned videos can also achieve good performance. Besides, to reduce computation cost, it proposes a patch alignment strategy that aligns different patches. It achieves competitive performance on multiple video super-resolution benchmark datasets.\n", "questions": "See weaknesses.", "limitations": "The authors have discussed the limitations and potential societal impact.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1, interesting finding on the relationship between attention window size and motion magnitude.\n\n2, a patch alignment strategy that crops and aligns patches rather than pixels\n\n\n\n\n\nWeakness:\n\n1, Previous attention-based method [21] proposes to use attention for implicit alignment, along with which feature alignment is used to deal with misalignment outside of the attention window. The key idea seems similar: attention can deal with misalignments.\n\n2, When the motion is small, the feature after alignment is largely still within the attention window. Why no alignment is better for small motions? Is there any more evidence (except for the final PSNR comparison)? Besides, due to the shifted window strategy, one pixel should have direct interactions with its neighbouring 12x12 pixels, why no alignment is worse when pixel movement magnitude is larger than 8 in Fig. 4 (a)?\n\n3, If alignment is unnecessary, global attention (largest window size) should achieve best performance.  Why does the PSNR of [3] (using global attention) is far from good? How about the performance of using global attention for the proposed framework?\n\n4, The patch alignment strategy is very similar to [24], which also crops patches from image features. [24] is even more flexible than this paper, as it samples several patches at a time. Attention is used in a similar way.\n\n5, The patch alignment strategy is also very similar to [1] that samples tokens (patches) for attention according to optical flow.\n\n6, Contradictory experiments results. When using nearest sampling in warping, the optical flow cannot be updated during training as there is no gradient. According to your conclusion, using an updated smooth flow is better than using the original pre-trained flow.\n\n7, If we need to find corresponding patches for each window, do we need to do it again when the windows are shifted?\n\n8, Some conclusions are not new. For example, the superiority of feature warping to image warping is already well-studied in previous works.\n\n9, No runtime and FLOPS comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "gzzvuCpTxq7", "reviewer": "Reviewer_j86q", "summary": "This paper rethinking the alignment in video super-resolution (VSR) Transformer and has two observations. (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. Based on these two observations, the authors remove the alignment in VSR, and propose a patch alignment to reduce the computational burden. Many experiments demonstrate the state-of-the-art performance on REDS.", "questions": "1. The authors state that \"existing alignment methods are sometimes harmful to VSR Transformers\". The importance of the alignment has been proved in most existing VSR methods. The performance of the alignment highly depends on the specific architecture of VSR Transformers. The observation may be only suitable for the proposed Transformer in this paper.\n\n2. The proposed VSR Transformer implicitly conduct the alignment and track the motion between frames. The authors provide visualization of local attribution map to highlight the pixels in each frame. It would better to explain from mechanism why the proposed VSR Transformer implicitly can align the feature in each frame.\n\n3. Recently, some studies demonstrate that large window size has better performance. In Table 4, the performance gain on REDS4 may mainly come from the large window size of 16x16. To verify the effectiveness of the patch alignment. It would better to report results with the window size of 8 in the supplementary. \n\n4. Patch alignment is sensitive to the similarity metric. When the adjacent frames has different illumination and scale, the performance of patch alignment may drop. How to relieve this issue in the experiment.\n\n5. The authors state that the proposed method can del with large motions. However, the performance is worse than VRT on Vimeo-90K-T which has different degrees of motions. Vimeo training set has 7 frames\n\n6. How many days for training PSRT-recurrent on the RED dataset? Besides, it would be better to report the inference time.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors conduct extensive experiments to explore the role of the alignment in VSR and have two observations.\n2. The proposed VSR Transformer method achieves the state-of-the-art performance on multiple datasets.\n\nWeaknesses:\n1. For the first observation, VSR-T and VRT are able to  utilize multi-frame information.\n2. The second observation \"existing alignment methods are sometimes harmful to VSR Transformers\" is two strong.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H0ILTT3guth", "reviewer": "Reviewer_F7UK", "summary": "This work explores the alignment modules in Transformer-based video super-resolution (VSR) methods. The research begins with experiments finding that VSR Transformers can directly utilize the unaligned videos and existing alignment methods can harm VSR Transformer. To solve this problem, this paper proposes a new and efficient alignment method named patch alignment, which aligns image patches instead of pixels. The experiments show the proposed method achieves state-of-the-art performance. ", "questions": "My main concern of this paper is the novelty and contribution. The problem of alignment does not seem broad and relevant, and I'm not sure if this research will contribute enough to the field. Another concern is that I did not find a clear experiment of how much improvement the patch alignment method brings compared to the baseline method, so I’m not sure if it's efficient enough.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper is easy to follow. The motivation is clear and straight, and the main problem is well defined.\n2) The problem of alignment for VSR Transformers is interesting, this paper shows VSR Transformers can utilize the unaligned videos.\n3) The performance is state-of-the-art, the proposed method achieves the first or second best performance on each benchmark.\n4) The method is efficient, there are not many extra parameters and computations introduced.\n\nWeakness:\n1) The main weakness seems to be the limited novelty and contribution. The main contribution of this work is the patch alignment method, but the method of aligning the patch instead of pixel seems too simple. The problems and solutions presented are relatively narrow, and I'm afraid this doesn't contribute much. The problem and solution proposed are relatively narrow, and this may not contribute much.\n2) The experiment is not very clear, I see that the VSR Transformers with patch alignment method achieves good performance, but I cannot find how much improvement the patch alignment brings compared with the baseline method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7clqEYpHnNa", "reviewer": "Reviewer_TWT3", "summary": "This paper rethinks alignment in video Super-Resolution Transformers, demonstrates that existing alignment methods are unnecessary or even harmful to the VSR Transformer, and will cause damage to the sub-pixel information, and proposes a simple but effective alignment method.", "questions": "1.\tIt’s not clear about the explanation of Figure 4,how to get the result of Figure 4,especially Figure 4(c).\n2.\tThere should place one line space before the figure caption and one line space after the figure, the layout of the figure is too crowded.", "limitations": "This paper use bicubic interpolation to produce LR video frames. It could be further discussed whether the experimental results will be affected by different degradation processes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1.\tThere is a certain novelty to this work, reconsidering the alignment in video super-resolution Transformers, while other methods retain the complex alignment module\n2.\tThe paper makes a reasonable analysis of the experimental results, proposes that the existing alignment methods are unnecessary or even harmful to the VSR Transformer, and will cause damage to the sub-pixel information.\n3.\tProposed a effective alignment method --- patch alignment, and experimental results show that this method achieves state-of-the-art VSR performance.\n\nWeaknesses\n1. Details about patch alignment are unclear. Section 5.1 could have further additions, in particular how to find the corresponding patches in the supporting frames for each patch, and how to move the patch to the corresponding position.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept