Recurrent Video Restoration Transformer with Guided Deformable Attention
Authors: Jingyun Liang
Yuchen Fan
Xiaoyu Xiang
Rakesh Ranjan
Eddy Ilg
Simon Green
Jiezhang Cao
Kai Zhang
Radu Timofte
Luc Van Gool
Venue: NeurIPS 2022 Accept (2022)
Keywords: video restoration
video super-resolution
video deblurring
video denoising
video alignment

Abstract: Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability. In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT. RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency. Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature. Within each clip, different frame features are jointly updated with implicit feature aggregation. Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism. Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.

Recurrent Video Restoration Transformer with Guided Deformable Attention
Jingyun Liang', Yuchen Fan", Xiaoyu Xiang', Rakesh Ranjan', Eddy Ilg² Simon Green?, Jiezhang Cao',
Kai Zhang': Radu Timofte Lue Van Gool
CCmmputer Vision Lab, ETH Zurich, Switzerland Meta Inc.
Unniverstty of Wurzburg, Germany
Abstract
Video restoration aims restoring multiple high-quality frames from multiple low- quality frames. Existing video restoration methods generally fall into two extreme cases, iee, they either restore frames in parallel or restore the video frame by frame in recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability. In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT. RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency. Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature. Within each clip, different frame features are jointly updated with implicit feature aggregation. Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism. Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of titte--ffth-aatt the-artt performance on benchmark datasets with balanced model size, testing memory and runtime. The codes are available at https:/ / github. com httm///ignnnnnn//iii..
1 Introduction
Video restoration, such as video super-resolution, deblurring, and denoising, has become a hot topic in recent years. It aims to restore clear and sharp high-quality video from degraded (e-g., downsampled, blurred, or noisy) low-quality video [80
38) has wide applications live streaming 997] video surveillance [49] old film restoration 78 and more.
11
Parallel methods and recurrent methods have been dominant strategies for solving various video restoration problems. Typically, those two kinds of methods have their respective merits and demerits. Parallel methods (2 22 80
99 27 35 38 support distributed deployment and achieve good performance by directly fusing information from multiple frames. but they often have large model size and consume enormous memory for sang-sequence vidcos. In the meanwhile, recurrent the same network block save parameters and predict the new frame feature based on the previously refined frame feature, but the sequential processing strategy inevitably leads information loss and noise amplification [4] for long-range dependency modelling and makes it hard to be parallelized.
72
36
63
models (24 59 21
2
26 28
45
55 98 62
"Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Considering the advantages and disadvantages of parallel and recurrent methods, in this paper, we propose recurrent video restoration transformer (RVRT) that takes the best of both worlds. On the one hand, RVRT' introduces the recurrent design into transformer- parameters and memory usage. On the other hand, it processes neighboring frames together as a clip to reduce video sequence length and alleviate information loss. To be specific, we first divide the video into fixed-length video clips. Then, starting from the first clip, we refine the subsequent clip feature based on the previously inferred clip feature and the old features of the current clip from shallower layers. Within each clip, different frame features are jointly extracted, implicitly aligned and effectively fused by the self-attention mechanism [77] 51 39 Across different clips, information is accumulated clip by clip with a larger hidden state than previous recurrent methods.
models to reduce model
To implement the above RVRT model, one big challenge is how to align different video clips when using the previous clip for feature refinement. Most existing alignment techniques [58] 65
87
15 72 80
11 38 are designed for frame-to-frame alignment. One possible way to apply them to clip-to-clip alignment is by introducing an extra feature fusion stage after aligning all frame pairs. Instead, we propose an one-stage video-to-video alignment method named guided deformable attention (GDA). More specifically, for a reference location in the target clip, we first estimate the coordinates of multiple relevant locations from different frames in the supporting clip under the guidance of optical flow, and then aggregate features of all locations dynamically by the attention mechanism.
GDA has several advantages over previous alignment methods: 1) Compared with optical flow-based warping that only samples one point from one frame [59 87 9.. GDA benefits from multiple relevant locations sampled from the video clip. 2) Unlike mutual attention 38 GDA utilizes features from arbitrary locations without suffering from the small receptive field in local attention or the huge computation burden in global attention. Besides, GDA allows direct attention on non- non-integer locations with bilinear interpolation. 3) In contrast to deformable convolution [15 100 72 80 10 that uses a fixed weight in feature aggregation, GDA generates dynamic weights to aggregate features from different locations. It also supports arbitrary location numbers and allows for both frame-to-frame and video-to-video alignment without any modification.
Our contributions can be summarized as follows:
We propose the recurrent video restoration transformer (RVRT) that extracts features of local neigh- boring frames from one clip in a joint and parallel way, and refines clip features by accumulating information from previous clips and previous layers. By reducing the video sequence length and transmitting information with a larger hidden state, RVRT alleviates information loss and noise amplification in recurrent networks, and also makes it possible to partially parallelize the model.
We propose the guided deformable attention (GDA) for one-stage video clip-to-clip alignment. It dynamically aggregates information of relevant locations from the supporting clip.
Extensive experiments on eight benchmark datasets show that the proposed model achieves state- of thethha performance in three challenging video restoration tasks: video super-resolution, video deblurring, and video denoising, with balanced model size, memory usage and runtime.
2
Related Work
2.1 Video Restoration
Parallel recurrent methods. Most existing video restoration methods can be classified as
parallel or recurrent methods according to their parallelizability. Parallel methods estimate all frames simultancously, as the refinement of one frame feature is not dependent on the update of other frame features. They can be further divided as sliding window-based methods [25 80) 70) 99
[72 79
60 33] and transformer-basee methods 4 38). The former kind of methods typically restore merely the center frame from the neighboring frames and are often tested in a sliding window fashion rather than in parallel. These methods generally consist of four stages: feature extraction, feature alignment, feature fusion, and frame reconstruction. Particularly, in the feature alignment stage, they often align all frames towards the center frame, which leads to quadratic complexity with respect to video length and is hard to be extended for long-sequence videos. Instead, the latter kind of method reconstructs all frames at time based on the transformer architectures. They jointly extract, align, and fuse features for all frames, achieving significant
36
63
99
2

---

Layer
Conv2d
RSTE
RER
OhevOd
RSTB
Upsamppi
Figure 1: The architecture of recurrent video restoration transformer (RVRT). From left to right, it consists of shallow feature extraction, recurrent feature refinement and HQ frame reconstruction. In recurrent fcature refinement (RFR, process frames one clip parallel within globally recurrent framework time. Multiple refinement layers are stacked for better performance.
details in Fig. 2 divide the vidco into N -rame clips (N in this figure) and
performance improvements against previous methods. However, current transformer- basermedr--aaddddd methods are laid up with huge model size and large memory consumption. Different from above parallel
methods, recurrent methods (24 59 23
85
26
28
9
45
62 S
propagate latent
features from one frame to the next frame sequentially. where information of previous frames is accumulated for the restoration of later frames. Basically, they are composed three stages: feature extraction, feature propagation and frame reconstruction. Due to the recurrent nature of feature propagation, recurrent methods suffer from information loss and the inapplicability of distributed deployment.
Alignment in video restoration. Unlike image restoration that mainly focuses feature extrac- tion [16 (94] 9666 42 [40, [1
67 |3] 92], how to align multiple highly-related but misaligned frames is another key problem in video restoration. Traditionally, many methods 43
30
48
68
3 87
first estimate the optical flow between neighbouring frames [19 58 651
and then conduct image
warping for alignment. Other techniques, such deformable convolution
[15 100 72 80
dynamic filter 29 and mutual attention [38] have also heen exploited for implicit feature alignment.
2.2 Vision Transformer
Transformer (77 is the de-facto standard architecture in natural language processing. Recently, has been used in dealing with vision problems by viewing pixels image patches tokens (8] 18), achieving remarkable performance gains in various computer vision tasks, including image
classification [1 37
51 74,
object detection 766 50 84) semantic segmentation 83
17
66
also achieves promising results restoration tasks
39
38 20
90 73 6
In particular, for video restoration, Cao et al. 4 propose the first transformer model for video SR,
while Liang et al. (38 propose an unified framework for video SR, deblurring and denoising.
We note that some transformer-based works [1I 84 have tried to combine the concept of defor- mation [15] 100 with the attention mechanism [77]. Zhu al. [TOI] directly predicts the attention weight from the query feature without considering its feature interaction with supporting locations. Xia et al. (84] place the supporting points uniformly on the image to make use of global informa- tion. Both above two methods are proposed for recognition tasks such as object detection, which is fundamentally different from video alignment in video restoration. Lin et al. [44] use pixel-level
patch-level attention to aggregate information from neighbouring frames under the guidance of
optical flow, but it only samples one supporting pixel or patch from one frame, restricting the model from attending multiple distant locations.
Methodology
3.1 Overall Architecture
Given low-quality video sequence 740 E RTXHXXXC where T, H. W and C are the video length, height, width and channel, respectively, the goal of video restoration to reconstruct the high-quality video He R xeee xaW'x where is the scale factor. To reach this goal, propose recurrent

---

video restoration transformer, as illustrated in Fig. 1 The model consists of three parts: shallow feature extraction, recurrent feature refinement and HQ frame reconstruction. More specifically, in shallow feature extraction, we first use convolution layer to extract features from the LQ video. For deblurring and denoising (i.e., 8 = 1), we additionally add two strided convolution layers to downsample the feature and reduce computation burden in the next layers. After that, several Residual Swin Transformer Blocks (RSTBs) [39] are used to extract the shallow feature. Then, we use recurrent feature refinement modules for temporal correspondence modeling and guided deformable attention for video alignment, which are detailed in Sec. 3.2 and Sec. 3.3 respectively. Lastly, we add several RSTBs to generate the final feature and reconstruct the HQ video I8H2 by pixel shuffle layer [61] For training. the Charbonnier loss
L
v [ 182 2 x2 (c 10 3) is used for all tasks.
3.2 Recurrent Feature Refinement
We stack L recurrent feature refinement modules to refine the video feature by exploiting the temporal correspon- dence between different frames. To make trade-off between recurrent and transformer-based methods, we process N frames locally in parallel on the basis of globally recurrent framework.
GDA
Formally, given the video feature F E RT from the i-th layer, we first reshape it as a 5-dimensional tensor F R XNXHXWX by dividing it into N video clip features: Fí, F2 ...,, F2 ERNxHxWxC Each clip
P1
feature F/ (1 < t S T
N has N neighbouring frame Fi, E " To utilize in- features: Fi, F2.2......
Conv2dl
formation from neighbouring clips, we align the (t- 1)-th clip feature F/_1 towards the t-th clip based on the opti- cal flow 1 clip feature Fi-l and clip feature F!-1.
MRSTBa2
This is formulated as follows:
Fin GDA 1 FE (1)
F
where GDA is the guided deformable attention and F is the aligned clip feature. The details of GDA will be described in Sec.
Figure 2: The illustrations of recurrent feature refinement (RFR). The (t - 1)-th clip feature F7-1 from the i-th layer aligned towards the t-th clip as Fi by guided deformable attention (GDA, see more details in Fig. B Fe. Ft Ft,,.,, FX-1 and F are then refined as F? by several modified residual swin trans- former blocks (MRSTBs), in which different frames are jointly processed in parallel way.
Similar to recurrent neural networks [59 9 11. as shown in Fig. 2 we update the clip feature of each time step follows:
F = RFR(FF FF
(2)
where F. is the output of the shallow feature extraction module and F7 P? ..... Fi-1 are from previous recurrent feature refinement modules. RFR(.) is the recurrent fea- ture refinement module that consists of a convolution layer for feature fusion and several modified residual Swin Transformer blocks (MRSTBs) for feature refinement. In MRSTB, we upgrade the original 2D h X w attention window to the 3D N x h x attention window, so that every frame in the clip can attend to itself and other frames simultaneously, allowing implicit feature aggregation. In addition, in order to accumulate information forward and backward in time, we reverse the video sequence for all even recurrent feature refinement modules (24 I..
The above recurrent feature refinement module is the key component of the proposed RVRT model. Globally, features of different video clips are propagated in a recurrent way. Locally, features of different frames are updated jointly in parallel. For an arbitrary single frame, it can make full use of global information accumulated in time and local information extracted together by the self- attention mechanism. As we can see, RVRT is generalization of both recurrent and transformer models. It becomes a recurrent model when fundamentally different from previous methods that adopt transformer blocks to replace CNN blocks within a recurrent architecture 78 44 is also different from existing attempts in natural language
or a transformer model when = T. This is
processing [82 34].

---

Inpur
Input
K
Warping
(0:: 2-1++
CNN
Inpur
Output
The illustrations of guided deformable attention (GDA). We estimate offsets of multiple relevant locations from different frames based the warped clip, and then aggregate features of different locations dynamically by attention mechanism. Fi-1 is (t 1)-th clip feature from the layer, while F Ff are pre-aligned and aligned features of Ft-1. O-1--6 and 01-1-t denote optical flows and offsets. respectively.
Figure 3:
3.3 Guided Deformable Attention for Video Alignment
Different from previous frameworks, the proposed RVRT needs to align neighboring related but misaligned video clips, as indicated in Eq. ( In this subsection, we propose the guided deformation attention (GDA) for video clip-to-clij alignment.
Given the (t 1) clip feature F; from the i-th layer, goal is to align Fi towards the t-th clip list of features F(.1.:) = (.((1) ((.(2)
1-1 PLNN) where Fi (1 S S N) denotes the aligned clip feature towards the n-th frame feature Fi,, of the 1-th clip, and F. (1 S n' S N) is the aligned frame feature from the n'-th frame in the (t 1)-th clip to the n-th frame in the f-th clip. Inspired by optical flow estimation designs (m)
we first pre-align F with 1- 01-1+t.)) where W denotes the warping operation. For convenience, we summarize the pre-alignment of all "n'-to-n" (1 I n',n n S N) frame pairs between the (t - 1)-th and t-th video clips as follows:
the optical flow (f((n) O2-1--,n as Fi.(n) (-1,n' W(F
(n)(n)
=i(1:N) = W(F (3.(1:))
(3)
After that, we predict the optical flow offsets o from the concatenation of F (<(I:)) and
ofL(1:V) along the channel dimension. A small convolutional neural network (CNN) with several convolutional layers and ReLU layers is used for prediction. This is formulated as
(.(:N) 0-11-1 = CNN((Cnnct((F) -F-1 Fi.(L:)) " .0
(4)
where the current misalignment between the t-th clip feature and the warped (t - 1) clip features
can reflect the offset required for further alignment. In practice, we initialize 0,-1-x the optical
flows estimated from the LQ input video via SpyNet [38], and predict M offsets for each frame (NM offsets in total). The optical flows are updated layer by layer as follows:
M
O (+1(m)
1- 1-
bttn +
(o (o - Jm
(.(n)
(5)
m=11
5

---

where (((())
(1111-t Im denotes the m-th offset in M predictions from the n'-th frame to the n-th frame.
Then, for the n-th frame of the f-th clip, we sample its relevant features from the (t 1)-th clip feature Fi according the predicted locations, which are indicated by the sum of optical flow and offsets, i.e.
1.(n) 1- Oz- 1--1-- according to the chain relationship Fi-1 - Fr.(n)
3.60 2-1-4 (a
58
For simplicity, we define the queries Q, keys K and values V as follows:
Q Fi,m Po.
(6)
K = Sampling( (F: Pk. 0 + 00-1-1)
(7)
V Sampling(Fi Pv,
P, Of + 0z ((x))
01-1-r):
where Q E RIxC is the projected feature from the 1-th frame of t-th clip. K E RNMC and V N are the projected features that are bilinearly sampled from NM locations of P and Fi-1: respectively. Po E RBCxC Px E RCC and Pv E RCXC are the projection matrices. Note that we first project the feature and then do sampling to reduce redundant computation.
Next, similar to the attention mechanism 777. we calculate the attention weights based on the Q and K from the (i - 1)-th layer and then compute the aligned feature F (i(()) as weighted of V from the same i-th layer as follows:
1-1
F-1 SoftMax(QK" (VO)))
(9)
where SoftMax is the softmax operation along the direction and VC is scaling factor.
Lastly, since Eq. (9) only aggregates information spatially, we add multi-layer perception (MLP) with two fully-connected layers and GELU activation function between them to enable channel interaction as follows:
- + MLP(F:
(10)
where a residual connection is used to stabilize training. The hidden and output channel numbers of the MLP are RC (R is the ratio) and C, respectively.
Multi-group multi-head guided deformable attention. We divide the channel into several deformable groups and perform the deformable sampling for different groups in parallel. Besides, in further divide one deformable group into several attention heads and
the attention mechanism, perform the attention operation separately for different heads. All groups and heads are concatenated together before channel interaction.
Connection to deformable convolution. Deformable convolution 15 [00 uses a learned weight for feature aggregation, which can be seen as a special case GDA, i.e., using different projection matrix Py for different locations and then directly averaging the resulting features. Its parameter number and computation complexity are MC2 and OMMC'), respectively. In contrast, GDA uses the same projection matrix for all locations but generates dynamic weights to aggregate them. Its parameter number and computation complexity are (3 + 2²) and O + 2RC + M)C), which are similar to deformable convolution when choosing proper M and R.
4 Experiments
4.1 Experimental Setup
For shallow feature extraction and HQ frame reconstruction, we use 1 RSTB that has 2 swin transformer layers. For recurrent feature refinement, we use 4 refinement modules with a clip size of 2, each of which has 2 MRSTBs with 2 modified swin transformer layers. For both RSTB and MRSTB, spatial attention window size and head number are 8 x 8 and 6, respectively. We use 144 channels for video SR and 192 channels for deblurring and denoising. groups and 12 deformable heads with 9 candidate locations. We empirically project the query to a higher-dimensional space (e.g., 2C') because we found it can improve the performance slightly and the parameter number of GDA is not bottleneck. In training, we randomly crop 256 x 256 HQ patches and use different video lengths for different datasets: 30 frames for REDS (53) 14
GDA, we use 12 deformable
frames for Vimeo-90K (87), and 16 frames for DVD (63). GoPro [54] well DAVIS 31] Adam

---

optimizer [33 with default setting is used to train the model for 600,000 iterations when the batch size The learning rate is initialized as x 10 and deceased with the Cosine Annealing scheme [52]. To stabilize training, we initialize SpyNet [58 [66] with pretrained weights, fix it for the first 30,000
iterations and reduce learning rate by %5%
4.2 Ablation Study
To explore the effectiveness of different components, we conduct ablation studies on REDS [53] for video SR. For efficiency, we reduce the MRSTB blocks by half and use 12 frames in training.
The impact of clip length. In RVRT, we divide the video into N-framme clips. As shown in Table] [] the performance rises when clip length is increased from to 2. However, the performance saturates when N =3 3, possibly due to large within-clip motions and inaccurate optical flow derivation. When directly estimate all optical flows (marked by "), the PSNR hits dB. Besides, to compare the temporal modelling ability, we hack the input LQ video (Clip 000 from REDS, 100 frames in total) by manually setting all pixels of the 50-th frame as zeros. As indicated in Fig. on the one hand, N = has a smaller performance drop and all its frames still have higher PSNR than N = (equals recurrent model) after the attack, showing that RVRT can mitigate the noise amplification from the hacked frame to the rest frames. One the other hand, the hacked frame of N = 2 has an impact on more neighbouring frames than N 1, which means that RVRT can alleviate information loss and utilize more frames than N = 1 for restoration.
The impact of video alignment. The aligmment of video clips plays a key role in our framework. We compare the proposed clip-to-clip guided deformable attention (GDA) with existing frame-to- frame alignment techniques by performing them frame by frame, followed by concatenation and channel reduction. As we can see from Table 2 GDA outperforms all existing methods when is used for Crame-to-frane alignment (denoted as GDA*), and leads further improvement when we aggregate features directly from the whole clip.
The impact of different components in GDA. We further conduct an ablation study on GDA in 3 As we can see, the optical flow guidance is critical for the model, leading to PSNR gain of 1.11dB. The update optical flow in different layers can further improve the result. The channel interaction in MLP also plays an important role, since the attention mechanism only aggregates information spatially.
The impact of deformable group and attention head. We also conduct experiments on different group and head numbers in GDA. As shown in Table when the deformable group rises, the PSNR first rises and then keeps almost unchanged. Besides, double attention heads lead to slightly better results at the expense of higher computation, but using too many heads has adverse impact the head dimension may be too small.
Table 1: Ablation study on clip length.
Table 2: Ablation study different video alignment techniques.
Clip PSNR
3
Alignment
Warping [87]
TMSA [38
DCN 72
GDA
GDA
31.98
32,10
32.07
32.21
PSNR
28.88
30.43
31.93
32.00
32.10
31.0
30.5
(orignaal (hacked)) A 1 (hackes)
Optical Flow Guidance Optical Flow Update
PSWR (18)
30.0
MLP PSNK
29.5
30.99
32.03
31.83
32.10
Table 3: Ablation study on different GDA components.
28.5
28.0
20
40
60
80
100
Frame Index
Detormable Group
12
12
Figure 4: Per-frame PSNR drop when pixels of the 50-th frame is hacked to be all zeros. N is clip length.
Attention Head
1
6
12
24
36
24
PSNR
31.63
32.03
52.10
32.13
32.03
32.11
Table 4: Ablation study on deformable groups and attention heads.
4.3 Video Super-Resolution
For video SR, we consider two settings: bicubic (BI) and blur-downsampling (BD) degradation. For BI degradation, we train the model on two different datasets: REDS (53) and Vimeo-90K (87).
7

---

Table 5: Quantitative comparison (average PSNR/SSIM) with state-of-the-art methods for video super-
resolution (x4) REDS4 1
Vimeo-90K-T
87 Vid4 46
and UDM10 89
degradation
dogradation
Mothod
REDS4 53
Vimeo 87
Vid4 46
UDMIO 89)
Vimeo MOK 87
Vid4 [46]
channel)
(Y channel)
channel)
(Y channel)
channel)
(Y channel)
Bicubic
26. 76.14
31.3
23.78/0.3377
28.47/0.8253
8687
200.0255246
SwinIR
29.05/0.8269
35.6770.9287
23.68/077491
$5.42/0.93380
34. 14.12/0.1167
25. 2500.262
SwinlR-Fi
29.24/0.3119
35.890093011
25. 25.6900.7488
36.76/0.9467
35.7000.9293
25.6200.7498
TOFlow
8
27.9880.799 7990
33,0880.9054
25, 8900766755
36.26/0.9453
34.6280.9212
25,85007655
FRVSR
37,09/0.9522
35.6400.3319
26.69
DUF 29
28.63318221
27.33/0.8319
38.48/0.9660
36.87/0.9447
27. 83299
PFNT.
89
29.63/0.850
36. 140393363
26.73/0.8029
38. 9/2779667
27. 660.835
23
30.09/0.8900
27.12/0.180)
38.66/0.9596
37.2000.9558
27.1 87.1750
MuCAN
36
30.88/18850
37.32X0.9465
RLSP
38.48/0 9606
36, 39,49/0.00 9403
27.48/0.838 8388
TGA 27
38.74/0 962
37. 9715
27. 97..330.8423
RSDN 26
19.35/0.99533
37.23/0.9471
27. 850
RRN
38,96/0 9644
(7.69990..8 8488
FDAN 45
39. /0 /0.916099666 9686
37.75/0.988 9522
27. 88/0.8508
EDVR
80
31.09/0.8800
37,6110.9489
27.35/0.8264
39,89/0 9686
37.81
15.8850.88 8503
GOVSR 88
40. 14/0
37.63/0.9503
28. 172.8224
BSR
8909
37. 87..9400
27.24/0.8255
39.96/0 9694
37.53/0. 9498
1...960.8555 8553
VSR 9
1L.6770.8944
37.4
27.3% 8279
00.0330.9694
37.84K0.9524
85700400
VRT 38
32.19/0. 90006
38,2000, 95.30
27. 27.93/0.8425
41.05/0.9737
38.72/0.9588
29. 87/0
BSsicVSR ++ I
$2.39/00.0000 90069
37.79/0. 9500
37479/0.8400
40.72/0.9722
38.21 90.9550
8755
RVRT (ours)
32.75/0.9113
38. 1510.9527
27.99/0.8466
40.90/0.9729
38.59/0.9576
29. 29.54/0.8810
Table 6: Comparison of model size, testing memory and runtime for an I.Q input of 320 X 180.
Memory
Runtime (ms))
(dB)
VS++ 11
7.3
22.3
77
32.39
VSR ++ [
+RSTB
9.3
1021
201
32.61
EDVR 80
20.6
3535
378
31,09
VSRT [
32.6
27487
328
31.19
38
35.6
2149
RVRT (ours)
10.8
1056
183
32.75
and then test the model on their corresponding testsets: REDS4 and Vimeo-90K-T. We additionally test Vid4 [46) along with Vimeo-90K. For BD degradation, we train it on Vimeo-90K and test on Vimeo-90K-T, Vid4, and UDMI0 [89]. The comparisons with existing methods are shown in Table 5 As we can see, RVRT achieves the best performance on REDS4 and Vid4 for both degradations. Compared with the representative recurrent model Basic M. RVRT improves the PSNR by significant margins of 0.2-0.5dB. Compared with the recent transformer based model VRT (38), RVRT outperforms VRT on REDS4 and Vid4 by up 0.36dB. The visual comparisons of different methods are shown in Fig. 5 It is clear that RVRT generates sharp and clear HQ frames, while other methods fail restore fine textures and details.
We compare the model size, testing memory consumption and runtime of different models in Table 6 Compared with representative parallel methods EDVR (80) VSRT 4 and VST (38), RVRT achieves significant performance gains with less than at least 50 % of model parameters and testing memory usage. also reduces the runtime by at least 25 %. Compared the recurrent model VSs+ (II. RVRT brings a PSNR improvement of 0.26dB. As for the inferiority of testing memory and runtime, we argue that it is mainly because the CNN layers are highly optimized on existing deep learning frameworks. To prove it, we use the cansformer-baseee RSTB blocks in RVRT to replace the CNN blocks in VSR++ in which case it has similar memory usage and more runtime than our model.
In addition, to better understand how guided deformable attention works, we visualize the predicted offsets the LQ frames and show the attention weight in Fig. 6 As we can see, multiple offsets are predicted to select multiple sampled locations in the neighbourhood of the corresponding pixel. According to the feature similarity between the query feature and the sampled features, features of different locations are aggregated by calculating a dynamic attention weight.
4.4 Video Deblurring
For video deblurring, the model is trained and tested on two different datasets, DVD 63 and GoPro [54], with their official training/testing splits. As shown in Table 7 and g RVRT shows its
8

---

(x4)
Clip DE1, REDS
L. (x4)
EDVR RA
VSRT 4
Busie's VSR
x
Franse 012, Clip city. Vidl 3is
VSrkVVR++ LB]
VKT
RYRT (curs)
GT
Figure 5: Visual comparison of video super-resolution (x4) methods REDS
and Vid4 46
Prodicted Offict Locatioes (red points)
Frzuee 10
Frome
Zoom
Aggregate
000
0.00 001
0.00
0.02
Aligned Feature (red point)
0
0.14
0.0
0/8
03
0.11
0.15
0.08
0.12
0.06
0.09
Fram 12
Frame 13
Predicted Attention Weight
Figure 6: The visualization of predicted offsets and attention weight predicted in guided deformable attention. Although guided deformable attention conducted features, plot illustrations I.Q input frames for better understanding. Best viewed by zooming.
superiority over most methods with huge improvements of 1.40~2227dB on two datasets. Even though the performance gain over VRT is relatively small, RVRT has smaller model size and much less runtime. In detail, the model size and runtime of RVRT are 13.6M and 0.3s, while VRT has 18.3M parameters and the runtime of 2.2s on 1280 720 LQ input. The visual comparison is provided in the supplementary material due to the space limit.
4.5
Video Denoising
For video denoising, we train the model on the training set of DAVIS and test it on corre- sponding testset and Set8 (70). For fairness of comparison, following 770 71. train non-blind additive white Gaussian denoising model for noise level a ~ (0, 50). Similar to the case of video deblurring, there is huge gap (0.60~2.37dB) between RVRT and most methods. Compared with VRT, RVRT has slightly better performance on large noise levels, with smaller model size (12.8M v.s.18.4M) and less runtime (0.2s v.s. 1.5s) 1280 720 LQ input. The visual comparison is provided in the supplementary material due to the space limit.
5 Conclusion
In this paper, proposed recurrent video restoration transformer with guided deformable attention. a globally recurrent model with locally parallel designs, which benefits from the advantages of both parallel methods and recurrent methods. We also propose the guided deformable attention

---

Table 7: Quantitative comparison (average RGB channel PSNR/SSIM) with statc-of-the-art methods for video
deblurring on DVD
63
DBN
STIN
EDV
PSNR
30,01
31.24
31.61
31.71
31.82
32.13
SSIM
0.8877
0.9340
0.9160
0.9160
0.9160
0.9268
PVDNet
(64
[35]
(44
(ours)
PSNR
32.51
32.53
32.80
33.36
34,24
34.30
SSIM
0.9260
0,9468
0.9352
0.9500
0.9651
0.9635
Table Quantitative comparison (average RGB channel PSNR/SSIM) with state-of-the-art methods video deblurring on GoPro [54]
Method
SRN [69]
MPRNer [9]
MAXIM [73
IFT-RNN [53]
ESTRNN [8]
EDVR 80
PSNR
30.25
32.66
32,86
31.05
31.07
31.54
SSIM
0.9342
0,9590
0,9610
0,9110
0,9023
0,9260
Method
ISP 137
PVDNet 662
GSTA [64
FGST [44]
VRT [38]
RVRT (ours)
PSNR
31.67
31.98
32.10
32.90
34.81
34,92
SSIM
0.9279
0.9280
0.9600
0.9610
0,9724
0.9738
Table 9: Quantitative comparison (average RGB channel PSNR) with state-of-the-art methods for video
demoising DAVIS 31 and Set8 77]
Dataset
VLNB I
DVDNet
FastDVDNet
PaCNet 775
VRT
RVRT (ours)
10
38.85
38.13
38.71
39.97
40.82
40.57
DAVIS
30
33,73
34.08
34.04
34,79
36.52
36.57
32.32
32.86
32.82
33.34
35.32
35.47
31.13
31.85
31.86
32,20
34,36
34.57
10
37.26
36.08
36.44
37.06
37.88
37.53
33.49
33.94
35.02
Se18
30
31.74
31.79
31.68
32.05
33.35
33.30
30.39
30.55
30.46
30.70
32.15
32.21
29,24
29.56
29.53
29.66
31,22
31.33
module for our special case of video clip-to-clip alignment. Under the guidance of optical flow, it aggregates information from multiple neighboring locations adaptively with the attention mechanism. Extensive experiments on video super-resolution, video deblurring, and video denoising demonstrated the effectiveness of the proposed method.
Limitations and Societal Impacts
Although RVRT achieves mate-of-the-an performance in video restoration, still has some limitations. For example, the complexity of pre-alignment by optical flow increases quadratically with respect to the clip length. One possible solution is to develop a tideo-to-video optical flow estimation model that directly predicts all optical flows. As for societal impacts, similar to other restoration methods, RVRT may bring privacy concerns after restoring blurry videos and lead to misjudgments used for medical diagnosis.
Acknowledgments and Disclosure of Funding
This work was partially supported by the ETH Zurich Fund (OK), Huawei Technologies Oy (Finland) project, the China Scholarship Council and an Amazon AWS grant. Special thanks goes to Yijue Chen.
References
[1] Pablo Arias and Jean-Michel Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal Mathematical Imaging and Vision, 60(1):70-93, 2018.
[2] Jose Caballero, Christian Ledig. Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real-time video super-resolution with spatio-temporal networks and motion compensation. IEFF Conference on Computer Vision and Pattern Recognition, pages 2017.
10

---

[3] Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real-time video super-resolution with spatio-temporal networks and motion compensation. In IEEE Conference on Computer Vision and Pattern Recognition,
pages 4787- 2017.
[4] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool, Video super-resolution transformer. arXiv preprint arXiv: 202.000..00. 06847, 2021.
[5] Jiezhang Cao, Jingyun Liang, Kai Zhang, Wenguan Wang, Qin Wang, Yulun Zhang, Hao Tang. and Luc Van Gool. Towards interpretable video super-resolution via alternating optimization. arXiv preprint arXiv:2207. 10765, 2022.
[6] Jiezhang Cao, Qin Wang. Jingyun Liang, Yulun Zhang, Kai Zhang. and Luc Van Gool. Practical real video denoising with realistic degradation model. arXiv preprins arXiv:2208.J /103, 2022.
V [7] Mingdeng Cao, Yanbo Fan, Yong Zhang, Jue Wang, and Yujiu Yang. Vdtr: Video deblurring with transformer. arXiv preprint arXiv:2204.0800
2022.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference
on Computer Vision, pages 213 229, 2020.
[9] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Basievsr: The search for essential components in video super-resolution and beyond. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4947 -44744956,.4 2021.
[10] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Understanding deformable alignment in video super-resolution. In AAAI Conference Artificial Intelligence, pages 973-981, 2021.
[11] Kelvin CK Chan, Shangehen Zhou, Xiangyu Xu, and Chen Change Loy. Basicvsr++: Im- proving video super-resolution with enhanced propagation and alignment. arXiv preprint arXiv:2104.1337/ 2021.
[12] Pierre Charbonnier, Laure Glanc-Feraude Gilles Aubert, and Michel Barlaud. Two deterministic half-quadratic regularization algorithms for computed imaging. In International Conference on Image Processing, pages 168-172, 1994.
[13] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12299-12310, 2021.
[14] Mengyu Chu, You Xie, Jonas Mayer, Laura Taixé, and Nils Thuerey. Learning temporal coherence via self-supervision for gan-based video generation. ACM Transactions Graphies (TOG),39(4).))))) 3954) 2020.
[15] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In IEEE International Conference on Computer Vision, pages 764-773, 2017.
[16] Chao Dong. Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages
184-199, 2014.
[17] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2107.0065 2021.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[19] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision pages 2758-228800 276822776,222 2015.
[20] Dario Fuoli, Martin Danelljan, Radu Timofte, and Lue Van Gool. Fast online video super- resolution with deformable attention pyramid. arXiv preprint arXiv:2202.01733 2022.
[21] Dario Fuoli, Shuhang Gu, and Radu Timofte. Efficient video super-resolution through recurrent latent space propagation. In IEEE International Conference on Computer Vision Workshop. pages 3476-3485, 2019.
[22] Zhicheng Geng, Luming Liang, Tianyu Ding, and Ilya Zharkov. Rstt: Real-time spatial temporal transformer for space-time video super-resolution. arXiv preprint 2xX. 2022.
[23] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back projeccinn network for video super-resolution. In IEEE Conference on Computer Vision and Pattern
11

---

Recognition, pages 3897-3906, 2019.
[24] Yan Huang, Wei Wang, and Liang Wang. Bidirectional recurrent convolutional networks for multi-frame super-resolution. Advances in Neural Information Processing Systems, 28:235-
243, 2015.
[25] Yan Huang, Wei Wang, and Liang Wang. Video super-resolution via bidirectional recurrent convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence,
1014):1015-1028. 2017.
[26] Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang Li. Shengjin Wang, and Qi Tian. Video super- resolution with recurrent structure-dctai network. In European Conference Computer Vision. pages 645-660, 2020.
[27] Takashi Isobe, Songjiang Li, Xu Jia, Shanxin Yuan, Gregory Slabaugh, Chunjing Xu, Ya-Li Li, Shengjin Wang. and Qi Tian. Video super-resolution with temporal group attention. In IEEE Conference on Computer Vision and Pattern Recognition, pages
2020. [28] Takashi Isobe. Fang Zhu, Xu Jia, and Shengjin Wang. Revisiting temporal modeling for video 2020.
super-resolution. arXiv preprint arXiv:2008.
[29] Younghyun Jo. Seoung Wug Oh, Jacyeon Kang, and Seon Joo Kim. Deep video super- resolution network using dynamic upsampling filters without explicit motion compensation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3224-3233 2018.
[30] Armin Kappeler, Seunghwan Yoo, Qiqin Dai, and Aggelos K Katsaggelos. Video super- resolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2(2):109-122, 2016.
[31] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Asian Conference on Computer Vision, pages 123-141, 2018.
[32] Tae Hyun Kim, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Scholkopf. Spatio-temporal transformer network for video restoration. In European Conference on Computer Vision, pages
106-122, 2018
[33] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.0 6980, 2014.
[34] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L Berg, and Mohit Bansal. Mart: Memory-augmmentd recurrent transformer for coherent video paragraph captioning. arXiv preprint arXiv:2005.0544
2020.
[35] Dongxu Li, Chenchen Xu, Kaihao Zhang, Xin Yu, Yiran Zhong, Wenqi Ren, Hanna Suominen, and Hongdong Li. Arvo: Learning all-range volumetric correspondence for video deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7721-7731, 2021. [36] Wenbo Li, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia. Mucan: colii-correspoddece aggregation network for video super-resolution. In European Conference on Computer Vision,
pages 335-351, 2020.
[37] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers, arXiv preprint arXiv:2104.00 100.02000 2021.
[38] Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer. arXiv preprint arXiv: 200.22221.288..
2022.
[39] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In IEEE Conference on International Conference on Computer Vision Workshops, 2021.
[40] Jingyun Liang, Andreas Lugmayr. Kai Zhang, Martin Danelljan, Luc Van Gool, and Radu Timofte. Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In IEEE Conference on International Conference Computer Vision, 2021.
[41] Jingyun Liang, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Mutual affine network for spatially variant kernel estimation in blind image super-resolution. In IEEE Conference on International Conference on Computer Vision. 2021.
[42] Jingyun Liang, Kai Zhang, Shuhang Gu, Luc Van Gool, and Radu Timofte. Flow-based kernel prior with application to blind super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 10601-10610, 2021.
[43] Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma, and Jiaya Jia. Video super-resolution via deep draft-ensemble learning. In IEEE International Conference on Computer Vision, pages 531-
539, 2015.
[44] Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Lue Van Gool. Flow-guided sparse transformer for video deblurring. arXiv preprint arXiv:2201.01893 2022.
12

---

[45] Jiayi Lin, Yan Huang, and Liang Wang. Fdan: Flow-guides deformable alignment network for video super-resolution. arXiv preprint arXiv:2105. 10w::200500060... 2021.
[46] Ce Liu and Deqing Sun. On bayesian adaptive video super resolution. JEEE Transactions on Pattern Analysis and Machine Intelligence, 36(2):346-360, 2013.
[47] Chengxu Liu, Huan Yang. Jianlong Fu, and Xueming Qian. Learning trajectory-
transformer for video super-resolution. arXiv preprint 0rXiv:2204.042) 2022.
[48] Ding Liu, Zhaowen Wang, Yuchen Fan, Xianming Liu, Zhangyang Wang, Shiyu Chang, and Thomas Huang. Robust video super-resolution with learned temporal dynamics. In IEEE International Conference on Computer Vision, pages 2507-2515, 2017.
[49] Hongying Liu, Zhubo Ruan, Peng Zhao, Chao Dong, Fanhua Shang. Yuanyuan Liu, Linlin Yang, and Radu Timofte. Video super-resolution based on deep learning: a comprehensive survey. Artificial Intelligence Review, pages 1-55, 2022.
[50] Li Liu, Wanli Ouyang, Xiaogang Wang. Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2))26131)
2020.
[51] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14000
2021.
[52] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint orXiv:/608. 0608.03983, 2016.
[53] Seungjun Nah, Sungyong Baik. Scokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1996-2005, 2019.
[54] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In IEEE Conference Computer Vision and Pattern Recognition, pages 3883-3891, 2017.
[55] Seungjun Nah, Sanghyun Son, and Kyoung Mu Lee. Recurrent neural networks with intra- frame iterations for video deblurring. In JEEE Conference on Computer Vision and Pattern Recognition, pages 8102-8111, 2019.
[56] Simon Niklaus. A reimplementation of SPyNet using PyTorch. https://gi thubbbb/ttttuub./. com/
aaiiklluuppyyorrch chrrtorrcchsyneet.... 2018.
[57] Junshan Pan, Haoran Bai, and Jinhui Tang. Cascaded deep video deblurring using temporal sharpness prior. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3043-3051. 2020.
[58] Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4161-4170, 2017. [59] Mchdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recurrent video super- resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 6626- 6634, 2018
[60] Dev Yashpal Sheth, Sreyas Mohan, Joshua L Vincent, Ramon Manzorro, Peter A Crozier, Mitesh M Khapra, Eero Simoncelli, and Carlos Fernandez-Grand Granda. Unsupervised deep video denoising. In IEEE International Conference on Computer Vision, pages 1759-1768, 2021. [61] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1883, 2016.
[62] Hyeongseok Son, Junyong Lee, Jonghyeop Lee, Sunghyun Cho, and Seungyong Lee. Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM Transactions Graphics. 40(5):1-18, 2021.
[63] Shuochen Su, Mauricio Delbracio, Jue Wang. Guillermo Sapiro. Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-held cameras. In JEEE Conference on Computer
Vision and Pattern Recognition. pages 1279-1288, 2017.
[64] Maitreya Suin and AN Rajagopalan. Gated spatio-temporal attention attention-guided video deblurring.
In IEEE Conference Computer Vision and Pattern Recognition, pages 7802-7811 2021. [65] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8934-8943, 2018
[66] Guolei Sun, Yun Liu, Thomas Probst, Danda Pani Paudel, Nikola Popovic, and Luc Van Gool. Boosting crowd counting with transformers. arXiv preprint arX2v:2105.10926, 2021.
13

---

[67] Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang. and Luc Van Gool. Mefnet: Multi-scale event fusion network for motion deblurring, arXiv preprin! 2xXiv2212..0///
2021.
[68] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. Detail-revealing deep video super-resolution. In JEEE International Conference on Computer Vision, pages 4472-4480, 2017.
[69] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In IEEE Conference on Computer Vision and Pattern Recognition, pages $174-8182 2018.
[70] Matias Tassano, Julie Delon, and Thomas Veit. Dvdnet: A fast network for deep video denoising. In IEEE International Conference on Image Processing, pages 1805-1809, 2019. [71] Matias Tassano, Julie Delon, and Thomas Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In IEEE Conference on Computer Vision and Pattern
Recognition, pages 1354-1363, 2020.
[72] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu. Tdan: cemporally-deformabbl alignment network for video super-resolution. In IEÉE Conference Computer Vision and
Pattern Recognition, pages 1360-3369 2020.
[73] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. arXiv preprint arXiv:2201 002973,
2022.
[74] Zhengzhong Tu, Hossein Talebi, Han Zhang. Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204, 01697, 2022.
[75] Gregory Vaksman, Michael Elad, and Peyman Milanfar. Patch craft: Video denoising by deep modeling and patch matching. In IEEE International Conference on Computer Vision, pages
1759-1768, 2021.
[76] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. arXiv
preprint arXiv:2103. 12731. 2021.
[77] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint
arXiv:1706.0m 03762, 2017.
[78] Ziyu Wan, Bo Zhang. Dongdong Chen, and Jing Liao. Bringing old films back to life. arXiv preprint DrXiv:2203.17277
2022.
[79] Longguang Wang. Yulan Guo, Li Liu, Zaiping Lin, Xinpu Deng, and Wei An. Deep video super-resolution using hr optical flow estimation. IEEE Transactions on Image Processing, 01:4323-3336, 2020.
[80] Xintao Wang. Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In IEEE Conference Com- puter Vision and Pattern Recognition Workshops, pages 1954-1963, 2019.
[81] Zhendong Wang, Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu. Uformer: A general u-shaped transformer for image restoration. arXiv preprint arXiv:2 1066, 2021.
[82] Zhiwei Wang. Yao Ma, Zitao Liu, and Jiliang Tang. R-transformer: Recurrent neural network enhanced transformer. arXiv preprint erXiv:1907. 055522 2019.
[83] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transform- ers: based image representation and processing for computer vision. arXiv preprint UrXiv:2006.0367 2020.
[84] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. arXiv preprint arXiv:2201.003. 005200,0000000000 2022.
[85] Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P Allebach, and Chenliang Xu. Zooming slow-mo: Fast and accurate one-stage space-time time video super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3370-3379, 2020. Xinguang Xiang. Hao Wei, and Jinshan Pan. Deep video deblurring using sharpness features from exemplars. IEEE Transactions on Image Processing, 29:8976-8988 2020.
[86]
[87] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhance- ment with task-oriented flow. International Journal of Computer Vision, 127(8): 106-1125, 2019.
[88] Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, Tao Lu, Xin Tian, and Jiayi Ma. Omni- scient video super-resolution. In IEEE International Conference on Computer Vision pages 4439-4438, 2021.
14

---

[89] Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, and Jiayi Ma. Progressive fusion video super-resolution network via exploiting non-local apatio-tempora correlations. In IEEE International Conference on Computer Vision, pages 3106-3115 15, 2019.
[90] Wulian Yun, Mengshi Qi, Chuanming Wang, Huiyuan Fu, and Huadong Ma. Coarse-to- fine video denoising with dual-stage spatial-channel transformer. arXiv preprint 2205v:2205.000 14, 2022.
[91] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In JEEE Conference on Computer Vision and Pattern Recognition, pages 14821-148311 2021.
[92] Kai Zhang. Yawei Li, Jingyun Liang. Jiezhang Cao, Yulun Zhang, Hao Tang, Radu Timofte, and Luc Van Gool. Practical blind denoising via swin-conv-unet and data synthesis. arXiv preprint arXiv:2203. 13278, 2022.
[93] Kai Zhang, Jingyun Liang. Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In IEEE Conference on International Conference on Computer Vision, 2021.
[94] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image
Processing, (11-31142-31555 2017.
[95] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution network for multiple degradations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3262-3271, 2018.
[96] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In European Conference
Computer Vision, pages 286-301, 2018.
[97] Yinjie Zhang, Yuanxing Zhang, Yi Wu, Yu Tao, Kaigui Bian, Pan Zhou, Lingyang Song, and Hu Tuo. Improving quality of experience by adaptive video streaming with super-resolution. In IEEE Conference Computer Communications, pages 1957-1966, 2020.
[98] Zhihang Zhong, Ye Gao, Yinqiang Zheng, and Bo Zheng. Efficient spatio-temporal recurrent neural network for video deblurring. In European Conference on Computer Vision, pages 191-207, 2020.
[99] Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, and Jimmy Ren. Spatio-temporal filter adaptive network for video deblurring. In JEEE International Conference on Computer Vision, pages 2482-2493 1891, 2019.
[100] Xizhou Zhu, Han Hu, Stephen Lin, and lifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2019.
[101] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 2rXiv:2010.04159,
2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A]. - You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline deseription. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
15

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] See S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section 4.
and supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and supplemental material.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "7V7bRaikglL", "reviewer": "Reviewer_zj8L", "summary": "This work proposed a recurrent video restoration transformer. It divides the video into multiple clips and the correlations among clips and inside clips are explored jointly. Within each clip, different frame features (2 frames in this work) are updated with implicit feature aggregation. Across clips, guided deformable attention is proposed to perform clip-to-clip alignment. Experimental results demonstrate that the proposed method achieves SOTA results on video SR, delurring, and denoising. ", "questions": "The authors are suggested to give responses for my concerns about the proposed GDA module. Can it be directly plugged into benchmark video restoration networks (i.e., replacing their original alignment modules) to further improve their performance? If yes, the impact of this work will be great. ", "limitations": "Yes, the authors have addressed their limitations. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe proposed guided deformable attention for clip-to-clip alignment is the main technical contribution of this work. The ablation study also demonstrates the effectiveness of the proposed module. \n2.\tThe proposed method can efficiently utilize long-term correlations \n3.\tThe proposed method works well on multiple video restoration tasks. \n\nWeaknesses:\nMore explanations about the proposed module are expected. For example, from table 3, the MLP module is essential for high performance (0.17 dB gain). Why? If we introduce the MLP module to DCN, will it outperform the proposed GDA?  In other words, the gain of GDA over DCN comes from which part of GDA? In addition, since GDA utilizes flow for guidance, a more fair comparison with GDA should be the flow-guided DCA, which was utilized in  BasicVSR++. \n\nIn Eq. (5), the predicted M offsets are averaged. Why? In DCN, we utilize multiple offsets for the same pixel and it is demonstrated to be more effective than the single-offset-based optical flow (each pixel only has one offset). What is the value of m in experiments?  Are the results sensitive to different settings of m? \n\nAs shown in Table 7, the denoising performance of the proposed method is worse than the compared methods when the sigma is smaller than 20. Why? \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2jogGt8dv29", "reviewer": "Reviewer_Js9S", "summary": "In this paper, the authors propose a general recurrent video restoration model for various video restoration tasks, including video SR, video deblurring and video denoising. It divides the video sequence into multiple video clips and deal with them by two strategies: globally, it propagates clip features in a recurrent way to reuse model parameters and save memory; locally, it jointly updates different frame features from one clip in parallel. Besides, since it processes the video as clips, it proposes the guided deformable attention for video clip-to-clip alignment. Extensive experiments on various benchmark datasets show the effective and the generalizability of the model.", "questions": "See weakness above.", "limitations": "The authors have discussed the limitations and potential societal impact.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed RVRT architecture is novel, effective and technically sound. Unlike previous methods that are either recurrent or parallel, RVRT takes the advantages of both directions and alleviates their corresponding problems. As validated in ablation studies and comparisons with existing methods, RVRT makes a good trade-off. It is much smaller, quicker and more memory efficient than parallel methods, and, still, it achieves state-of-the-art performance on benchmark datasets. RVRT also provides a useful way to tackle the information loss and noise application problems that are inherent to recurrent models, as proved in the supplementary. Furthermore, it proposes a guided deformable attention module that is directly applicable for clip-to-clip alignment, in which relevant features from different frames are aggregated dynamically and efficiently. This module is well illustrated and is proved to be effective in experiments. \n\nOverall, this paper makes a good contribution to video restoration and conducts extensive experiments to support its arguments. It provides an alternative video sequence modelling option, rather than focusing on the design of specific modules. It should also be an interesting paper for the wider community. I think this paper should be accepted.\n\n\nPros:\n\n1, It proposes a novel architecture that extracts features locally in parallel and accumulates information globally in a recurrent way. It has many benefits due to reduced sequence length, larger hidden state and local parallel processing.\n\n2, It proposes a one-stage guided deformable attention module for dynamic and global feature aggregation in clip-to-clip alignment. It fits the overall architecture well.\n\n3, The proposed method is evaluated on most standard video restoration benchmarks (8 different datasets), and the performance is solid and convincing. In particular, on deblurring and denoising, it remains the surprising performance of transformer models and outperforms most of its competitor by up to 2.27~2.37dB. \n\n4, The paper writing is fairly good.\n\nCons:\n\n1, What are the training time and memory cost for different tasks?\n\n2, Typo in the caption of Fig. 2.\n\n3, The comparison of temporal modelling ability is interesting and is suggested to be put in the main paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0rca7LYHs9h", "reviewer": "Reviewer_CJyZ", "summary": "This paper tries to integrate the advantages of parallel methods and recurrent methods, and proposes a recurrent video restoration transformer (RVRT). RVRT first divides the video into fixed-length video clips and refines the subsequent clip feature based on the previously inferred clip feature and the old features of the current clip from shallower layers. Within each clip, the authors use a self-attention mechanism to fuse the feature. Across the clips, the authors propose guided deformable attention (GDA) to achieve video-to-video alignment. The proposed RVRT achieves state-of-the-art performance in video super-resolution, video deblurring, and video denoising. ", "questions": "See questions in Weaknesses", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n\n1. Parallel methods and recurrent methods are dominant strategies in the VSR area, and most researchers only adopt one strategy in their work. Therefore, the attempt of adopting these two strategies in a single framework is valuable.\n\n2. From Tab. 5, Tab. 7, Tab. 8, and Tab. 9, RVRT achieves state-of-the-art performance in video super-resolution (BI-REDS/Vid4, BD-Vid4), video deblurring, and video denoising. \n\n- Weaknesses:\n\n1. The effectiveness of the core idea (which takes advantage of parallel and recurrent methods) is not well verified. \n\n    1.1. From Tab. 1, when N=3, the performance is worse than N=2. When N=3, the frames within each clip can aggregate more information (the information of the other 2 frames) than N=2. Besides, it also aggregates more information from the neighboring clips (by aggregating the features of 3 frames). Therefore, it is unreasonable that the performance of RVRT saturates when N=3. More detailed analyses (visualization or experiments) are supposed to be provided.\n\n    1.2. From Tab. 6, when BasicVSR++ is equipped with RSTB, the performance gains 0.26dB and achieves 32.61dB. Compared to RVTB, its performance is only 0.14dB lower with 1.2M fewer parameters. Therefore, it is hard to distinguish the effectiveness of the core idea. What is the performance of BasicVSR++ + RSTB, whose parameters are the same as RVRT? \n\n2. Some results are confusing. Wait for the authors’ explanations/analyses.\n\n    2.1. From Tab. 5, it is clear that the performance of RVRT is better than VRT on Vid4 and REDS. However, on Vimeo-90K-T and UM10 datasets, RVRT achieves inferior performance. \n\n    2.2. From Tab. 7, when sigma is 10 and 20, the performance of RVRT is inferior to VRT. However, when sigma is larger than 30, the performance of RVRT is comparable to or better than VRT. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept