Learning Truncated Causal History Model for Video Restoration
Authors: Amirhosein Ghasemabadi
Muhammad Kamran Janjua
Mohammad Salameh
Di Niu
Venue: NeurIPS 2024 poster (2024)
Keywords: video restoration
low-level computer vision
motion understanding
Primary area: machine_vision
TL;DR: We present a new video restoration framework to learn and model the history of the video frames.

Abstract: One key challenge to video restoration is to model the transition dynamics of video frames governed by motion. In this work, we propose Turtle to learn the truncated causal history model for efficient and high-performing video restoration. Unlike traditional methods that process a range of contextual frames in parallel, Turtle enhances efficiency by storing and summarizing a truncated history of the input frame latent representation into an evolving historical state. This is achieved through a sophisticated similarity-based retrieval mechanism that implicitly accounts for inter-frame motion and alignment. The causal design in Turtle enables recurrence in inference through state-memorized historical features while allowing parallel training by sampling truncated video clips. We report new state-of-the-art results on a multitude of video restoration benchmark tasks, including video desnowing, nighttime video deraining, video raindrops and rain streak removal, video super-resolution, real-world and synthetic video deblurring, and blind video denoising while reducing the computational cost compared to existing best contextual methods on all these tasks.

Learning Truncated Causal History Model for Video Restoration
Amirhosein Ghasemabadi
Muhammad Kamran Janjua
ECE Department, University of Alberta
Huawei Technologies, Canada kamran, janjuaChuawei.cmmm com
ghasemabQualberta.
Mohammad Salameh
Niu ECE Department, University of Alberta dniu@ualberta.co ca.
Huawei Technologies, Canada mohammad, sal awwh@huawei...... com
Abstract
One key challenge video restoration to model the transition dynamics of video frames governed by motion. In this work, we propose TURTLE to learn the TRUncaTed causaL. history modEl for efficient and high-performing video restoration. Unlike traditional methods that process range of contextual frames in parallel, Turtle enhances efficiency by storing and summarizing a truncated history of the input frame latent representation into evolving historical state. This achieved through a sophisticated similarity- retrieval mechanism that implicitly accounts for inter-frame motion and alignment. The causal design in TURTLE enables recurrence inference through state-memorized historical fea- tures while allowing parallel training by sampling truncated video clips. We report new state-of-the-e results on a multitude of video restoration benchmark tasks, including video desnowing, nighttime video deraining, video raindrops and rain streak removal, video super-resolution, real-world and synthetic video deblurring, and blind video denoising while reducing the computational cost compared existing best contextual methods on all these tasks.
Q https:: battps:/kkjajja/// gi ghtubb... io/turtle/
Introduction
Video restoration aims restore degraded low-quality videos. Degradation in videos occurs due to noise during the acquisition process, camera sensor faults, or external factors such as weather or motion blur [53, 38]. Several methods in the literature process the entire video either in parallel or with recurrence in design. In the former case, multiple contextual frames are processed simultaneously to facilitate information fusion and flow, which leads increased memory consumption and inference cost as the context size increases [63, 4, 69, 86, 58, 28, 26, 5, 34, 62). Methods with recurrence frame sequentially based previously refined ones [54, 14, 21, 25, 6, 7,7,42, 42, 57]. Such sequential processing approaches often result in cumulative errors, leading information loss in long-range temporal dependency modeling [8] and limiting parallelization capabilities.
in design reuse the same network to process
Recently, methods based on state space models (SSMs) have seen applications across several machine vision tasks, including image restoration [19, 56], and video understanding [30]. While Video- Mamba [30] proposes state space model for video understanding, the learned state space does not reason at the pixel level and, hence, can suffer from information collapse in restoration tasks [77].
indicates equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Additionally, the state evolves over time with respect to motion that affects the entire trajectory non- uniformly [51] the pixel level. Therefore, pertinent to learn a model capable of summarizing the history' of the input as it operates on the spatiotemporal structure of the input video.
In this work, present "TURTIFF", a new video restoration framework to learn the TRUncaTed causaL history modEl of a video. TURTLE employs the proposed Causal History Model (CHM) to align and borrow information from previously processed frames, maximizing feature utilization and efficiency by leveraging the frame history to enhance restoration quality. We outline our contributions.
TURTLE'S encoder processes each frame individually, while its decoder, based on the pro- posed Causal History Model (CHM), reuses features from previously restored frames. This structure dynamically propagates features and compensates for lost or obscured information by conditioning the decoder on the frame history. CHM models the evolving state and compensates the history for motion relative to the input. Further, it learns to control the effect of history frames by scoring and aggregating motion-compensated features according to their relevance to the restoration of the current frame.
TURTLE facilitates training parallelism by sampling short clips from the entire video se- quence. In inference, TURTLE's 's recurrent view implicitly maintains the entire trajectory ensuring effective frame restoration.
TURTLE sets new state-of-the-ar results on several benchmark datasets and video restoration tasks, including video desnowing, nighttime video deraining, video raindrops and rain streak removal, video super-resolution, real and synthetic video deblurring, and achieves competitive results on the blind video denoising task.
2 Related Work
Video restoration is studied from several facets, mainly distributed in how the motion is estimated and compensated for in the learning procedure and how the frames are processed. Additional literature review is deferred appendix G.
Motion Compensation in Video Restoration. Motion estimation and compensation are crucial for correcting camera and object movements in video restoration. Several methods employ optical flow to explicitly estimate motion and devise a compensation strategy as part of the learning procedure, such as deformable convolutions [33, 34], or flow refinement [23]. However, optical flow can struggle with degraded inputs [84, 3, 20], often requiring several refinement stages to achieve precise flow estimation. On the other end, methods also rely on the implicit learning of correspondences in the latent space across the temporal resolution of the video; few techniques include temporal shift modules [29], non-local search [64, 32, 85), or deformable convolutions [69, 13, 80].
Video Processing Methods. There is similar distinction in how a video is processed, with several methods opting for either recurrence in design or restoring several frames simultaneously. Parallel methods, also known as sliding window methods, process multiple frames simultaneously. This sliding window approach can lead to inefficiencies in feature utilization and increased computational costs [63, 4, 69, 86, 58, 28, 26, 5, 34, 62, 9]. Although effective in learning joint features from the entire input context, their size and computational demands often render them unsuitable for cnoource-constraingg
devices. Conversely, recurrent methods restore frames sequentially, using multiple stages to propagate latent features [87, 81, 82). These methods are prone to information loss [33]. Furthermore, while typical video restoration methods in the literature often rely context from both past and future neighboring frames [34, 33, 29], TURTLE is causal in design, focuses on using only past frames. This approach allows TURTLE to apply in scenarios like streaming and online video restoration, where future frames are unavailable.
3 Methodology
Consider a low-quality video I E TTxH*WxC where T, H, W. C denote the temporal resolution, height, width, and number of channels, respectively, that has been degraded with some degradation
In this manuscript, the term "history" refers to temporally previous frames with respect to the input frame.

---

Degraded Video Franes
Restored Video Frames
Degraded Video Restored Video Franes
Time
111
Historyless FFN X
Skip Connection
Decoder N
Causal History
Historyless
Skip
y!!!!
Model
Decoder
Transforeer
Historyless FFN X
Middle Block
Decodar,
H
(b) Docoder View
TURTLE's Architecture
Assorted Restoration Results
Figure 1: TURTLE'S Architecture. The overall architecture diagram of the proposed method. TURTLE a U-Net [52] style architecture, wherein the encoder blocks are historyless feedforward blocks, while the decoder couples the causal history model (CHM) to condition the restoration procedure on truncated history of the input. We also present assorted restoration examples on the right-frame taken from video raindrops and rain streak removal [71], night deraining [47], and video deblurring [41] tasks, respectively.
E ID. The goal of video restoration is to learn model Mo parameterized by 0 to restore high-quality video IeQ E Txxxxaxalll XC, where is the scale factor (where > 1 for video super-resolution). To this end, we propose TURTLE, a U-Net style [52] architecture, to process, and restore a single frame at any given timestep conditioned on the truncated history of the given frame. TURTLE'S encoder focuses only on single frame input and does not consider the broader temporal context of the video sequence. In contrast, the decoder, however, utilizes features from previously restored frames. This setup facilitates dynamic propagation of features through time, effectively compensating for information that may be lost or obscured in the input frame. More specifically, we condition a decoder block at the different U-Net stages on the history of the frames. Given frame timestep each block learns to model the causal relationship p(y, F,, Hz), where y: is the output of a decoder block, F, is the input feature map of the decoder block, and H2 is the history of corresponding features maps from the previous frames at the same block. We train the architecture with the standard L, loss function: L = n E |I - 11 for all the restoration tasks. We present the visual illustration of TURTLE'S 's architecture in Figure
3.1 Architecture Design
Given a model Ms. F!!l denote the feature map of a frame at timestep 6, taken from M, at layer 1. We, then, utilize F to construct the causal history states denoted as H E xRT xw" xe where T is the truncation factor (or length of the history). ht w denote spatial resolution of the history, and c denotes the channels. More specifically, H [F
- e F!!
F!ll R x xe! +-++1 2 + f-1
where the concatenation operation. We denote the cootion-ommpenstteed history at timestep
as Ár! which is compensated for motion with respect to the input frame features F!l In this work, the state refers to the representation of a frame of the video. Further, history states (or causal history states) refers to set of certain frame features previous to the input at some timestep.
TURTLE'S encoder learns representation of each frame by downsampling the spatial resolution, while inflating the channel dimensions by a factor 2. At each stage of the encoder, we opt for several stacked convolutional feedforward blocks, termed Historyless FFN,2, 2 The learned representation at the last encoder stage onwards is fed to running history queue Q of length 7.3 We empirically set y for all the tasks, and consider sequence of 5 frames. The entire video sequence is reshaped
We use term "Historyless" because the encoder not conditioned on the history input.
'The space complexity is limited to O(y x h x c) and (7 x x x c) << (T' x H x W x C') because 7 << I', and h << H, < < W. and all enqueue and dequeue operations O in time complexity.

---

softtnax
0.2
Dos Product
Input Frane
0.0
Imput Frame
Frama Attention
History
BA
H
top-k Selection
vill
Notion Compensated History
Figure 2: Causal History Model. The diagrammatic illustration of the proposed Causal History Model (CHM) detailing the internal function. In the initial phase, for each patch in the current frame (denoted by the stars), we identify and implicitly align the top-k similar patches in the history. In the subsequent phase, we score and aggregate features from this aligned history create refined output that blends the input frame features with pertinent history data. We visualize frames in this diagram for exposition, but in practice the procedure operates the feature maps.
into = xxxxC thereby allowing parallelism training while maintaining a dense representation of history states to condition the reconstruction procedure
The decoder takes the feature map of the current frame, F!", and the history states H We propose a motion compensation module that operates on the feature space to implicitly align history states with respect to the input frame. Next, a dynamic router learns control the effect of history frames by scoring and aggregating motion-compensated features based on their relevance to the restoration of the current frame. Such procedure accentuates the aligned history such that the following stages of the decoder can learn to reconstruct the high-quality frame appropriately. Both of these procedures combine form the Causal History Model CHM(F!") : H detailed in section 3.2. Further, multiple CHMs are stacked as black box layers at different stages to construct the decoder of TURTLE.
3.2 Causal History Model
CHM learns to align the history states with respect to the input feature map. Further, there could still exist potential degradation differences at the same feature locations along the entire sequence in the motion-compensated history states. To this end, CHM re-weights the sequence along the temporal dimension to accentuate significant features and suppress irrelevant ones. Let H! e (f++ xwr xe denote the motion-compeessted causal history states, and let input feature map be
F!I 1d xw"xx' Let the transformation on the history states to align the features be denoted by
6,, and let vh, denote the re-weighting scheme. If the output is given by !!] E IRA xw xe! formalize the Causal History Model (CHM) as.
we then,
A M = de
f E (FF)),
||| the F!! (F.")) y
(2)
In eq. (1), B, denotes transformation on the input, and denotes the skip connection, while O the concatenation operation. In practice, we lear di, and the input transformation matrix B following the procedure described in State Align Block, while ehy detailed Frame History Router. We present a visual illustration of Causal History Model (CHM) in fig. 2. We also present special case of (CHM) in appendix D, wherein we consider optimally compensated motion in videos.
State Align Block (o). State Align Block (d) implicitly tracks and aligns the corresponding regions defined as groups pixels (or patches) (p: x p2)-across each frame in the history. State Align Block computes attention scores through dot product between any given patch from the current frame and

---

all the patches from the history. Given the input feature map of frame F.]] E IRA x xe!
we calculate WF!
the patched projections as, QF K. F. F. R P7 (ep1pz)) i.c., Of, K v Fe
v
F.
where W is learnable parameter matrix. For exposition, let the dimensions of projections be Rad x xd we subsequently rearrange the patches to (mpp=))'e xd' Here, n/, = pr and n'..
denote the number of patches along the height and width dimension, and d' (cP1p2)' represents the dimension of each patch. Formally, we define the history states H; as a set of keys and values to facilitate the attention mechanism as,
H = (K H. H. J.
(3)
where K and Vill He are formally written as K H. [|]
[K F... v v.,.) E RTXPAMuxd
K
K
F.
I x and V [l [v]l F.... v P V
H.
We, then, compute the attention, and limit it to the top-k most similar patches in the key vector for each patch in the query vector, and, hence, focus solely on those that align closely. This prevents the inclusion of unrelated patches, which can, potentially, introduce irrelevant correlations, and obscure principal features. We, then, formalize the top-k selection procedure as,
(QF, 11
K E E RRTX(n vi,,) x(n) ml.)
(4)
if x E (et, mL) (A (A(.)))) (9] A:),
A A
(5)
00,
otherwise
where a is learnable parameter scale the dot product, and (s..) denotes the 5th patch along the second dimension. A t computation. In other words, each patch is compensated for with respect to its top-k similar, and salient patches across the trajectory. Such procedure allows for soft alignment, and encourages each patch to borrow information from its most similar temporal neighbors, i.c., a one-to-top-k temporal correspondence is learned. Given the top-k scores, we compute the motion-comppesatte history states Hr!" as follows,
[l] masks the non top-k scores, and replaces with -oo to allow for softmax
A !4 = a
M H.
W + (F!)))
(6)
where a is the softmax operator, @ is the concatenation operator, WH," " is the parameter matrix learned with gradient descent, and B is a transformation on the input F!! realized through self-
attention along the spatial dimensions [39, 66). In eq. (6), di (H p(l) [o(A ( v wh which follows from eq. (1).
Frame History Router (v). Given the motion-compensated history states H. E [ and xe!
and the input features F!"l E Tu'h xw" xe'. Frame History Router (x)) learns to route and aggregate critical features for the restoration of the input frame. To this end, we compute the query vector from
F.4 through the transformation matrix WQ!"" resulting in Q!"! +- F!Il O.!"" Similarly, the key and
value vectors are derived from ( and are parameterized W i.e., K. v!ll A
This configuration enables cross-frame channel attention, where the query from F!!l attends to channels from both (r and F and accentuates temporal history states as necessary in order to restore the given frame. The cross-channel attention map A E [R(r-1eexx xel is then computed through the dot product, i.e., A = (Q K!')/ae E ( xe where a is the scale factor to control the dot product magnitude. Note that, overload the notation A!" for exposition. The output feature map, yt takes the shape A x xc! since the attention matrix takes the shape e [ xe", while v!l is R(++ 1jo xh' x If denotes the softmax operator, and D is the skip connection, we then
'This is because in practice, channels are collapsed in the temporal dimension.
5

---

Table 2: Video Desnowing Results.
Table 1: Night Video Deraining Results.
Method
PSNR
SSIM
Method
PSNR
SSIM
Trans Weather [65]
23.
0.8543
FDM [22]
23.49
0.7657
SnowFormer [12]
24.01
0.8939
DSTFM [46]
17.82
0.6486
S2VD [78]
22.95
0.8590
WeatherDiff [43]
20.98
0.6697
RDDNet [68]
22.97
0.8742
RMFD [75]
16.18
0.6402
EDVR [69]
17.93
0.5790
DLF [74]
15.17
0.6307
Basic VSR [6]
22.46
0.8473
HRIR [31]
16.83
0.6481
IconVSB [6]
22.35
0.8482
MctaRain (Meta) [47]
23.49
0.7171
BasicVSR++ [7]
22.64
0.8618
MetaRain (Sert) [47]
22.21
0.6723
RVRT [33]
20.90
0.7974
NightRain [35]
26.73
0.8647
SVDNet [10]
25.06
0.9210
TURTLE
29.26
0.9250
TURTLE
26.02
0.92.0
Degraded Region
Ground Truth
SVDNet
TURTLE
Desnowing
Night Deraining
Degraded Region
Ground Truth
MetaRain (Meta)
TURTLE
Figure 3: Visual Results on Video Desnowing and Nighttime Video Deraining, We compare video desnowing results with the best published method in literature, SVDNet [10]. The video frame has both snow, and haze. While SVDNet [10] removes snow flakes, TURTLE can remove haze, and snow flakes, and hence is more faithful the ground truth. In nighttime deraining, we compare TURTLE to MetaRain [47]. TURTLE maintains color consistency in the restored result.
compute the output, y!! as,
y: [o(Al")v")] A + DD(FF" e gd xxuxx
(7)
In eq. (7), w/ (ii
((A v!!
wAl" which follows from eq. (2).
Experiments
We follow the standard training setting of architectures in the restoration literature [29, 79, 15] with Adam optimizer [27] (3, 0.9, 82 = 0.999). The initial learning rate is set to 4e -4 and decayed to -e- throughout training following the cosine annealing strategy [40]. All of our models are implemented in the PyTorch library, and are trained on 8 NVIDIA Tesla v100 PCIe 32 GB GPUs for 250k iterations. Each training video is sampled into clips of y = 5 frames, and TURTLE restores frames of each clip with recurrence. The training videos are cropped to 192 x 192 sized patches at random locations, maintaining temporal consistency, while the evaluation is done on the full frames during inference. We assume no prior knowledge of the degradation process for all the tasks. Further, we apply basic data augmentation techniques, including horizontal-verna flips and 90 90-deeeee rotations. Following the video restoration literature, Peak Signal- totall--t-Noiee

---

Table 3: Real-World Video Debturring. Quantitative results (PSNR, and SSIM) the 3ms-24ms BSD
Table 4: Synthetic Video De- blurring Results. Quantita- tive results (PSNR, and SSIM) on the GoPro dataset [41] com- paring of the- meth- ods.
Table 5: Video Raindrop and Rain Streak Removal. Quantitative re- sults (PSNR, and SSIM) the VRDS datasct [71] comparing state-of-the-ar methods.
dataset [83] comparing state-
of-the-art methods.
Method
PSNR
SSIM
Method
PSNR
SSIM
Method
PSNR
SSIM
S2VD [78]
18.95
0,6630
STRCNN [24]
29.42
0.893
IFI-RNN [42]
31.05
0.9110
EDVR [69]
19.19
0,6363
DBN [58]
31.21
0.922
ESTRNN [82]
31.07
0.9023
BasicVSR [6]
28.35
0.8990
SRN [60]
28.92
0.882
EDVR [69]
31.54
0.9260
VRT [34]
27.77
0.8856
IFI-NNN [42]
30.89
0.917
TSP [44]
31.67
0.9260
TTVSR [37]
28.05
0.8998
STEAN [86]
29.47
0.872
GSTA [59]
32.10
0.9600
RVRT [33]
28.24
0.8857
CDVD-TSP [44]
31.58
0.926
FGST [36]
32.90
0.9610
RDDNet [68]
28.38
0.9096
PVDNet [57]
31.35
0.923
BSsicVSR++ [7]
34.01
0.9520
Basic BasicVSR++ [7]
29.75
0.9171
ESTRNN [83]
31.39
0.926
DSTNet [45]
34.16
0.9679
VIMPNet [71]
31.02
0.9283
TURTLE
33.58
0.954
TURTLE
34.50
0,9720
TURTLE
32.01
0.9590
Ratio (PSNR) and Structural Similarity Index (SSIM) [70] distortion metrics to report quantitative performance. For qualitative evaluation, we present visual outputs for each task and compare them with the results obtained from previous best methods in the literature.
4.1 Night Video Deraining
SynNightRain [47] is synthetic video deraining dataset focusing nighttime videos wherein rain streaks get mixed in with significant noise in low-light regions. Therefore, nighttime deraining with heavy rain is generally harder restoration task than other daytime video deraining. We follow the train/test protocol outlined in [47, 35], and train TURTLE on 10 videos from scratch, and evaluate on a held-out test set of 20 videos. We report distortion metrics, PSNR and SSIM, in table 1, and compare them with previous restoration methods. TURTLE achieves PSNR of 29.20 dB, which is notable improvement of +2.53 dB over the next best result, NightRain [35]. Further, we present visual results in fig. 3, and in fig. 12. Our method, TURTLE, maintains color consistency in the restored results.
4.2 Video Desnowing
Realistic Video Desnowing Dataset (RVSD) [10] is a video-first desnowing dataset simulating realistic physical characteristics of snow and haze. The dataset comprises a variety of scenes, and the videos are captured from various angles to capture realistic scenes with different intensities. dataset includes 110 videos, of which S0 are used for training, while 30 are held-out test set to measure desnowing performance. We follow the proposed train/test split in the original work [10] and train TURTLE on the video desnowing dataset. Our scores, 26.02 dB in PSNR, are reported in table 2, and compared to previous methods, TURTLE significantly improves the performance by +0.96 dB in PSNR. Notably, TURTLE is prior-free unlike the previous best result SVDNet [10], which exploits snow-type priors. We present visual results in fig. 3, and in fig. 11 comparing TURTLE to SVDNet [10]. Our method not only removes snowflakes but also removes haze, and the restored frame is visually pleasing.
total, the
4.3 Real Video Deblurring
The work done in [83, 82] introduced a real-world deblurring dataset (BSD) using the Beam-Splitter apparatus. The dataset introduced contains three different variants depending on the blur intensity settings. Each of the three variants has total of 11, 000 blurry/sharp pairs with resolution of 640 x 480. We employ the variant of BSD with the most blur exposure time, i.e., 3ms-24ms. We follow the standard train/test split introduced in [83] with 60 training videos, and 20 test videos. We report the scores in table on the 3ms-24ms variant of BSD and compare with previously published methods. TURTLE scores 33.58 dB in PSNR on the task. observing an increase of +2.0 dB compared to the previous best methods, CDVD-TSP [44], and ESTRNN [83. 82]. We present visual results in fig. 13.
$24ms is the blur exposure time, and 3ms is the sharp exposure time.
7

---

Dagraded Frame
TURTLE
GOPRO Debluring
240 0-12-
240 0212
VRDS
Degraded Frame
Ground Truth
VIMPler
TURTLE
Figure 4: Visual Results on Video Deblurring and Raindrops and Rain Streaks Removal. Qualitative results on video deblurring on the GoPro dataset [41] are in the top row. Our method, TURTLE, restores the frames without any artifacts (see the number plate) unlike DSTNet [45]. On video raindrops and rain streaks removal task, we compare our method with the best method in literature ViMPNet [71]. Notice how the frame restored by ViMPNet [71] has artifacts (see tree region, and the railing gate), while TURTLE's output free of unwanted artifacts.
Table 6: Blind Video Denoising Results. Quan- titative results on blind video denoising task terms of distortion metrics, PSNR and SSIM, on two datasets DAVIS [48], and Set8 [6]].
Table 7: 4x Video Super Resolution. Quantita- tive results on video super resolution task in terms of distortion metrics, PSNR and SSIM.
DAVIS
Set8
Method
PSNR
SSIM
Methods
σ 50
30
TDAN [63]
23.07
0.7492
EDVR [69]
23,51
0.7611
VLNB []]
33.73
31.13
31.74
29.24
BSR [6]
23.38
0.7594
FastDVDNet [62]
34.04
31.86
31.60
29.42
MANA [76]
23.15
0.7513
DVDNet [61]
34.08
31.85
31.79
29.56
TTVSR [37]
23.60
0.7686
UDVD [55]
33.92
31.70
32.01
29.89
VSs [7]
23.70
0.7713
ReMoNet [72]
33,93
31.65
31.59
29.44
EAVSR [67]
23.61
0.7618
BSVD-32 32 [49]
34.46
32.25
31.71
29.62
EAVSR+ [67]
23.94
0.7726
BSVD-64 [49]
34.91
32.72
32.02
29.95
TURTLE
25.30
0.8272
TURTLE
34.48
32.38
32.22
30.29
4.4 Synthetic Video Deblurring
GoPro dataset [41] is established video deblurring benchmark dataset in the literature. The dataset is prepared with videos taken from a GOPRO4 Hero consumer camera, and the videos are captured at 240fps. Blurs of varying strength introduced in the dataset by averaging several successive frames; hence, the dataset is a synthetic blur dataset. We follow the standard train/test split of the dataset [41], and train our proposed method. TURTLE scores 34.50 dB in PSNR on the task, with an increase of +0.34 dB compared to the previous best method in comparable computational budget, DSTNet [45] (see table 4). We also present visual results on the GoPro dataset [41] comparing TURTLE to DSTNet [45] in fig. 4, and fig. 9. Our method restores frames free of artifacts (see the number plate on the car) in fig. 4.
4.5 Video Raindrops and Rain Streak Removal
The work done in [71] introduced a synthesized video dataset of 102 videos, VRDS, wherein the videos contain both raindrops and rain streaks degradations since both rain streaks and raindrops corrupt the videos captured in rainy weather." 6 We split the dataset in train and held-out test sets as outlined in the original work [71]. We present TURTLE'S scores table 5, and compare it with several methods in the literature. TURTLE scores 32.01 dB in PSNR on the task, with an increase of +0.99 dB compared to the previous best method, ViMPNet [71]. We present visual results on the task fig. 4, and fig. 10, and compare our method with ViMPNet [71]. TURTLE restores the frames that are pleasing to the human eye and are faithful to the ground truth.
'The dataset comprises videos captured in diverse scenarios in both daytime and nighttime settings.
8

---

Frame
BSVD-64
TuRn
Dencising
HORHEX
Super Resolation
TTVSR
BasicVSR1
EAVSR
TURTLE
Ground Truth
Figure 5: Blind Video Denoising and Video Super- Super-Resolution Visual Results. Qualitative compar- ison of previous methods with TURTLE on a test frame from Set8 dataset for blind video denoising (a 50), and MVSR4x dataset [71] for video super resolution. In video denoising, TURTLE restores details, while BSVD-64 [49] smudges textures (text and the dinosaur on the biker's jacket). VSR, previous methods such TTVSR [37], BasicVSR++ [7], EAVSR [71] tend to introduce blur in results, while TURTLE's restored results are sharper, and crisper.
Table 8: MACs (G) Comparison. We report MACs (G) of TURTLE, and compare with previous methods in literature. We also extensively profile TURTLE with varying input resolutions on single GPU, and compare it with previous restoration methods in appendix F.
Method
Venue
Task
MAC's (G)
RVRT [33]
NeurlPS'22
Restoration
1182.16
VRT [34]
TIP'24
Restoration
1631.67
RDDNet [68]
ECCV'22
Deraining
362.36
DSTNet [4S]
CVPR
Deblurring
720.28
EDVR [69]
CVPR'19
Deraining
527.5
Bas [6, 7]
CVPR'21
Super Resolution
240.17
TURTLE
NeurlPS'24 24
Restoration
181.06
4.6 Video Super-Resolution
MVSR4x is real-world paired video super-resolution dataset [67] collected by mobile phone's dual cameras. We train TURTLE following dataset split in the official work [67] and test on provided held-out test set. We report distortion metrics in table 7 and compare it with several methods in the literature. TURTLE scores 25.30 dB in PSNR the task, with significant increase of +1.36 dB compared to the previous best method, EAVSR+ [71]. We present visual results on the task in fig. 5. Other methods such TTVSR [37], SSR [7], EAVSR [71] tend to introduce blur in up-scaled results, while TURTLE'S restored results are sharper.
4.7 Blind Video Denoising
We assume no degradation prior, and consider blind video denoising task [49, 55]. We train our model on DAVIS [48] dataset, and test on DAVIS held-out testset, and a generalization set Set8 [61]. We add white Gaussian noise to the dataset with noise level 2(330 50] train TURTLE, and
on two noise levels o = 30, and o 50; scores are reported in table 6. TURTLE observes gain of +0.31 dB on 30, and +0.34 dB on 50 on Set8 testset, scoring 32.22 dB, and 30.29 dB, respectively, while observes an average drop of -0.3 dB to BSVD-64 [49] on the DAVIS testset. Further, we present qualitative results in fig. 5 comparing TURTLE, and previous best method BSVD [49].
4.8
Computational Cost Comparison
In table 8, we compare TURTLE with previous methods in the literature terms of multiply- accumulate operations (MACs). The results are computed for the input size 256 x 256. We measure the performance on the number of frames the original works utilized to report their performance, as
or frames if the actual number of frames did not fit into single 32GB GPU

---

Table 9: State Align Block.
Table 10: Truncation Factor.
Table 11: Value of topk.
Block
Truncation
Value of k in topk
PSNR f
PSNR t
PSNR 1
Configuration
Factor 7
No CHM
31.84
1
32.15
32.18
No
32.07
5
32.26
20
32.10
TURTLE
32.26
TURTLE (7 = =3)
32.26
TURTLE (k = 5)
32.26
reported in their manuscript or code bases. In TURTLE's case, we report MACs (G) on a single frame since TURTLE only considers a single frame at a time but adjust for history features utilized in CHM as part TURTLE'S decoder. In comparison to parallel methods, EDVR [69], VRT [34], TURTLE is computationally efficient, as it is lower in MACs (G). Although the MACs are approximately similar to recurrent methods, Basic [6], TURTLE scores significantly higher in PSNR/SSIM metrics (see table 7, and table 5). In comparison contemporary methods such as RVRT [33], which combines recurrence and parallelism in design, TURTLE is significantly lower on MACs (G) and
performs better (see table 5, and table 2) thanks to its ability to memorize previous frames.
Ablation Study
We ablate TURTLE to understand what components necessitate efficiency and performance gains. All experiments are conducted on synthetic video deblurring task, GoPro dataset [41], using smaller variant of model. Our smaller models operate within computational budget of approximately MACs (G), while the remaining settings are the same as those of the main model. In all the cases, the combinations we adopt for TURTLE are highlighted Additional ablation studies are deferred to appendix A, and discuss the limitations of the proposed method in appendix C.
Block Configuration. We ablate the Causal History Model (CHM) to understand if learning from history benefits the restoration performance. We compare TURTLE with two settings: baseline (no CHM block) and TURTLE without State Align Block (#). In baseline (no CHM), no history states are considered, and two frames are concatenated and fed to the network directly. Further, in No ø, the state align block is removed from CHM. We detail the results in table 9, and find that both State Align Block, and CHM are important to the observed performance gains.
and past frames and found no PSNR improvement when increasing the context length beyond three frames. Results in table 10 confirm that extending beyond three frames does not benefit performance. This is because. as in most cases, the missing information in the current frame is typically covered within the three- fraee-ffame span, and additional explicit frame information fails to provide additional relevant details.
Truncation Factor T. We evaluate context lengths of T
Value of k in topk. We investigate the effects of different & values in topk attention. Our experiments, detailed in table 11, show that l crucially affects restoration quality. Utilizing a larger number of patches, h 20, leads by adding unnecessary noise. Further, selecting only 1 patch is also sub-optimal as the degraded nature of inputs can lead to inaccuracies in identifying the most similar patch, missing vital contextual information. The optimal balance was found empirically with & = 5, which effectively minimizes noise while ensuring the inclusion of key information.
accumulation of irrelevant information, negatively impacting performance
Conclusion
In this work, we introduced novel framework, TURTLE, for video restoration. TURTLE learns to restore any given frame by conditioning the restoration procedure on the frame history. Further, compensates the history for motion with respect to the input and accentuates key information to benefit from temporal redundancies in the sequence. TURTLE enjoys training parallelism and maintains the entire frame history implicitly during inference. We evaluated the effectiveness of the proposed method and reported state-of-the-a results seven video restoration tasks.
10

---

References
[1] Arias and J.-M. Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60:70 93,9009---000 2018.
[2] A. Bardes, Q. Garrido, Ponce, X. Chen, M. Rabbat, LeCun, M. Assran, and N. Ballas. Revisiting feature prediction for learning visual representations from vidco. arXiv preprint arXiv:2404. 08471, 2024.
[3] L. Barron, D. J. Fleet, and S. S. Beauchemin. Performance of optical flow techniques. International journal of computer vision, 12:43-77, 1994.
[4] Caballero, C. Ledig, A. Aitken, A. Acosta, Totz, Z. Wang, and W. Shi. Real-time video super-resolution with spatio-temporal networks and motion compensation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4778-4787, 2017.
[5] Cao, Li, K. Zhang, and L. Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106.06847, 2021.
[6] K. C. Chan, X. Wang, K. Yu, C. Dong, and C. C. Loy. Basievsr: components in video super-resolution and beyond. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4947-4956, 2021.
The search for essential
[7] K. C. Chan, S. Zhou, X. Xu, and C. C. Loy. Basicvsr++: enhanced propagation and alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5972-5981, 2022.
Improving video super-resolution with
[8] K. C. Chan, S. Zhou, X. Xu, and C. C. Loy. Investigating tradeoffs in real-world video super- resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5962-5971, 2022.
[9] H. Chen, Y. Jin, K. Xu, Y. Chen, and C. Zhu. Multifrane-to-mullfrfaaee network for video denoising. IEEE Transactions on Multimedia, 24:2164 21778 2178, 2021.
[10] H. Chen, J. Ren, J. Gu, H. Wu, X. Lu, H. Cai, and I.. Zhu. Snow removal in video: A new dataset and novel method. In 2023 IEEF/CVF International Conference Computer Vision (ICCV), pages 13165-13176. IEEE, 2023.
[11] I.. Chen, X. Chu, X. Zhang, and J. Sun. Simple baselines for image restoration. In European conference on computer vision, pages 17-33. Springer, 2022.
[12] S. Chen, T. Ye, Y. Liu, E. Chen, J. Shi, and J. Zhou. Snowformer: Scale-aware transformer via context interaction for single image desnowing. arXiv preprint arX1v:2208.09703, 2, 2022.
[13] Deng, L. Wang. S. Pu, and C. Zhuo. Spatio-temporal deformable convolution for compressed video quality enhancement. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10696-10705 10703, 2020.
[14] D. Fuoli, S. Gu, and R. Timofte. Efficient video super-resolution through recurrent latent space propagation. In 2019 IEEE/CVF International Conference Computer Vision Workshop (ICCVW), pages 3476 3485. IEEE, 2019.
[15] A. Ghasemabadi, M. K. Janjua, M. Salameh, C. ZHOU, F. Sun, and D. Niu. Cascadedgaze: Effi- ciency global context extraction for image restoration. Transactions on Machine Learning Re- search, 2024. ISSN 1835-8856 URL https://openreviiie wwwpiiewweee/forro net wwt.:neeennnniieeooo C3FXHxMVuq.
[16] A. Gu. Modeling Sequences with Structured State Spaces. Stanford University, 2023.
[17] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
[18] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal
polynomial projections. Advances neural information processing systems, 33:1474-1487, 2020.
11

---

[19] H. Guo, J. Li. T. Dai, Z. Ouyang, X. Ren, and S.-T. Xia. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint arXiv:2402.15648, 2024.
[20] Harguess, C. Barngrover, and A. Rahimi. An analysis of optical flow on real and simulated data with degradations. In Geospatial Informatics, Fusion, and Motion Video Analytics VII, volume 10199, pages 23-39. SPIE, 2017.
[21] M. Haris, G. Shakhnarovich, and N. Ukita. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3897-3906, 2019.
[22] W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35:27953-27965, 2022.
[23] C. Huang, J. Li, B. Li, D. Liu, and Y. Lu. Neural compression- sradeeessoo-b feature learning for video restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5872-5881, 2022.
[24] T. Hyun Kim, K. Mu Lee, B. Scholkopf, and M. Hirsch. Online video deblurring via dynamic temporal blending network. In Proceedings of the IEEE international conference on computer vision, pages 4038 4047, 2017.
[25] T. Isobe, X. Jia, S. Gu. S. Li, S. Wang, and Q. Tian. Vidco super-resolution with recurrent structure-dete detail network. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 645-660. Springer, 2020.
[26] T. Isobe, S. Li, X. Jia, S. Yuan, G. Slabaugh, C. Xu, Y.-L. Li, S. Wang, and Q. Tian. Video super-resolution with temporal group attention. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8008-8017, 2020.
[27] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 2014.
[28] D. Li, C. Xu, K. Zhang, X. Yu, Y. Zhong, W. Ren, H. Suominen, and H. Li. Arvo: Learning all-range volumetric correspondence for video deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7721-7731, 2021.
[29] D. L. X. Shi, Y. Zhang, K. C. Cheung, S. See, X. Wang. H. Qin, and H. Li. A simple baseline for video restoration with grouped apatial-temporee shift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[30] K. Li, X. Li, Y. Wang, Y. He, Y. Wang, L. Wang, and Y. Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977, 2024
[31] R. Li, L.-F. Cheong, and R. T. Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1633-1642, 2019.
[32] W. Li, X. Tao, T. Guo, I. Qi, J. Lu, and J. Jia. Mucan: Multi-correspondence aggregation net- work for video super-resolution. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pages 335-351. Springer, 2020.
[33] J. Liang, Y. Fan, X. Xiang, R. Ranjan, E. Ilg, S. Green, J. Cao, K. Zhang, R. Timofte, and L. V. Gool. Recurrent video restoration transformer with guided deformable attention. Advances in Neural Information Processing Systems, 35:378-393, 2022.
[34] J. Liang, J. Cao, Y. Fan, K. Zhang, R. Ranjan, Y. Li, R. Timofte, and L. Van Gool. Vrt: A video restoration transformer. IEEE Transactions on Image Processing, 2024.
[35] B. Lin, Y. Jin, W. Yan, W. Ye, Y. Yuan, S. Zhang, and R. T. Tan. Nightrain: Nighttime video deraining via adaptive-rain-remova and adaptive-correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3378-3385, 2024.
[36] Lin, Cai, X. Hu, H. Wang, Yan, X. Zou, H. Ding, Zhangg R. Timofte, and L. Van Gool. Flow-guided sparse transformer for video deblurring. In ICML, 2022.
12

---

[37] Liu, H. Yang. J. Fu, and X. Qian. Learning trajectory aware transformer for video super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5687-5696, 2022.
[38] H. Liu, Z. Ruan, P. Zhao, C. Dong, F. Shang, Y. Liu, L. Yang, and R. Timofte. Video super- resolution based on deep learning: comprehensive survey. Artificial Intelligence Review, 55 (8):5981-608) 2022.
[39] 7. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[40] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint http:16088039.33 2016.
[41] S. Nah, T. Hyun Kim, and K. Mu Lec. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3883-3891, 2017.
[42] S. Nah, S. Son, and K. M. Lee. Recurrent neural networks with intra-frame iterations for video deblurring. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1102-81111 2019.
[43] O. Özdenizci and R. Legenstein. Restoring vision in adverse weather conditions with patch- based denoising diffusion models. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[44] J. Pan, H. Bai, and J. Tang. Cascaded deep video deblurring using temporal sharpness prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3043-3051, 2020.
[45] J. Pan, B. Xu, J. Dong, J. Ge, and J. Tang. Deep discriminative spatial and temporal network for efficient video deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22191-22200, 2023.
[46] W. Patil, S. Gupta, Rana, and S. Venkatesh. Dual-frame spatio-temporal feature modulation for video enhancement. Pattern Recognition, 130: 2022.
[47] W. Patil, S. Gupta, S. Rana, and S. Venkatesh. Video restoration framework and its meta- adaptations to data-poor conditions. In European Conference on Computer Vision, pages 143-160. Springer, 2022.
Sorkine-Homunn, [48] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine- Hornung, and L. Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.0065 2017.
[49] C. Qi, J. Chen, X. Yang, and Q. Chen. Real-time streaming video denoising with bidirectional buffers. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2758 2788-2766,22 2022.
[50] R. Qian, X. Dong, P. Zhang, Y. Zang, S. Ding, D. Lin, and J. Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405.16000 2024.
[51] A. Rajagopalan and R. Chellappa. Motion deblurring: Algorithms and systems. Cambridge University Press, 2014.
[52] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicul image computing and computer-assisted intervention-MICCY 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241, Springer, 2015.
[53] C. Rota, M. Buzzelli, S. Bianco, and R. Schettini. Video restoration based on deep learning: comprehensive survey. Artificial Intelligence Review, 56(6):5317-534 2023,
13

---

[54] M. S. Sajjadi. R. Vemulapalli, and M. Brown. Frame- recurreet video super-resolttion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6626- 6634, 2018.
[55] D. Y. Sheth, S. Mohan, J. Vincent, R. Manzorro, P. A. Crozier, M. M. Khapra, E. P. Simoncelli, and C. Fernandez--Granda. Unsupervised deep video denoising. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2021.
[56] Y. Shi, B. Xia, X. Jin, X. Wang, Zhao, X. Xia, X. Xiao, and W. Yang. Vmambair: Visual state space model for image restoration. arXiv preprint arXiv:2403. 11033.1142,,,, 2024.
[57] H. Son, Lee, Lee, S. Cho, and S. Lee. Recurrent video deblurring with blur -lvarinvariant motion estimation and pixel volumes. ACM Transactions on Graphics (TOG), 40(5):1-18, 2021.
[58] S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang. Deep video deblurring for hand-held cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[59] M. Suin and Rajagopalan. Gated spatio-temporal aatentiio-guided video deblurring. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7802-7811, 2021.
[60] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8 8174-818 1182, 2018.
[61] M. Tassano, J. Delon, and T. Veit. Dvdnet: A fast network for deep video denoising. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1805-1809. IEEE, 2019.
[62] M. Tassano, J. Delon, and T. Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 1354-133 1363, 2020.
[63] Y. Tian, Y. Zhang, Y. Fu, and C. Xu. Tdan: deeporally-ddeformbble alignment network for video super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3360-3369, 2020.
[64] G. Vaksman, M. Elad, and P. Milanfar. Patch craft: Video denoising by deep modeling and patch matching, In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2157-2166, 2021.
[65] J. M. J. Valanarasu, R. Yasarla, and V. V.M. Patel. Transweather: Transformer-base restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2353- 2363, 2022.
[66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[67] R. Wang, X. Liu. Z. Zhang, X. Wu, C.-M. Feng. L. Zhang, and W. Zuo. Benchmark dataset and effective inter- frame alignment for real-world video super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1168-1177, 2023.
[68] S. Wang, L. Zhu, H. Fu, J. Qin, C.-B. Schönlieb, W. Feng, and S. Wang. Rethinking video rain streak removal: A new synthesis model and deraining network with video rain prior. In European Conference on Computer Vision, pages 565-582 Springer, 2022.
[69] X. Wang, K. C. Chan, K. Yu, C. Dong, and C. Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[70] Z. Wang, A. C. Bovik, H. R. Sheikh, and F. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612 -12, 2004.
14

---

[71] H. Wu, Y. Yang, H. Chen, J. Ren, and L. Zhu. Mask busk-guided progressive network for joint raindrop and rain streak removal in videos. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7216-7225, 2023.
[72] L. Xiang, J. Zhou, J. Liu, Z. Wang. H. Huang, J. Hu, J. Han, Y. Guo, and G. Ding. Remonet: Recurrent multi-output network for efficient video denoising. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 2786-2794, 2022.
[73] S. Yang, J. Walker, J. Harker-Holder, Y. Du, J. Bruce, A. Barreto, P. Abbeel, and D. Schuurmans. Video as the new language for real-world decision making. arXiv preprint 24Xiv:2402.17139, 2024.
[74] W. Yang. J. Liu, and J. Feng. Frame-consistent recurrent video deraining with dual-level flow. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1661-1670, 2019.
[75] W. Yang. I. Tan, J. Feng, S. Wang. B. Cheng, and Liu. Recurrent multi-frame deraining: Combining physics guidance and adversarial learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(11))8569-8383 2021.
[76] Yu, J. Liu, L. Bo, and T. Mei. Memory-augmented non-local attention for video super- resolution. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 17843, 2022.
[77] W. Yu and X. Wang. Mambaout: Do we really need mamba for vision?, 2024.
[78] Z. Yue, Xie, Q Zhao, and D. Meng. Semi-supervised video deraining with dynamical rain generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 642-652, 2021.
[79] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728-5732 5039, 2022.
[80] H. Zhang, H. Xie, and H. Yao. Spatio-temporal deformable attention network for video deblurring. In European Conference on Computer Vision, pages 581-596. Springer, 2022.
[81] M. Zhao, Y. Xu, and Zhou. Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction. In Proceedings of the 29th ACM international conference on multimedia, pages 5646-5654, 2021.
[82] Z. Zhong, Y. Gao, Y. Zheng, and B. Zheng. Efficient spatio-temporal recurrent neural network for video deblurring. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt VI 16, pages 191-207. Springer, 2020.
[83] Z. Zhong, Y. Gao, Y. Zheng, B. Zheng, and I. Sato. Real-world video deblurring: A benchmark dataset and efficient recurrent neural network. International Journal of Computer Vision, 13 131())2284330) 2023.
[84] H. Zhou, Y. Chang, G. Chen, and L. Yan. Unsupervised hierarchical domain adaptation for adverse weather optical flow. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 3778-3786, 2023.
[85] K. Zhou, W. Li, X. Han, and J. Lu. Exploring motion ambiguity and alignment for high-quality video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22169-22177 19, 2023.
[86] S. Zhou, J. Zhang. J. Pan, H. Xie, W. Zuo, and J. Ren. Spatio-temporal filter adaptive network for video deblurring. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2482-249) 2019.
[87] C. Zhu, H. Dong. J. Pan, B. Liang, Y. Huang, L. Fu, and F. Wang. Deep recurrent neural network with scati bi-directional propagation for video deblurring. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 3598-3607 2022.
15

---

Technical Appendices
Contents
Additional Ablation Studies
16
TURTLe's Specifiitioons & Details
18
B.1 Motivation: Causal History Model
18
B.2 Historyless FFN.
18
C Limitations & Discussion
18
C.1 Societal Impact
19
D Relationship to State Space Models
19
Further Visual Comparisons
20
E.1 Synthetic Video Deblurring
20
E,2 Video Raindrops and Rain Streaks Removal
20
E,3 Video Desnowing.
21
E,4
Nighttime Video Deraining
21
E.5
Real-World Video Deblurring..
22
E,6
Real-World Weather Degradations
22
Computational Profile of TURTLE
22
Additional Literature Review
23
H Dataset Information & Summary
24
In appendices, we discuss additional details about the proposed method TURTLE, and provide additional ablation studies in appendix A, motivate the need for learning to model the history of input for video restoration in appendix B, discuss limitations of the proposed approach appendix C, discuss theoretical relationship to state-space models in appendix D, present visual results in appendix E, discuss related work appendix G, and computationally profile the proposed method TURTLE in appendix F.
Table 12: CHM Placement.
Table 13: Softmax Ablation.
CHM
Ablating
PSNR t
PSNR 1
Placement
Softmax
in Latent
32.05
Softmax
32.04
in Latent & Decoder
32.26
lopk (k=
32.26
A Additional Ablation Studies
We ablate two more aspects of the proposed method TURTLE. Mainly, we empirically verify the rationale behind placing CHM in both latent, and decoder stages. Further, we ablate if selecting topk regions compared to plain softmax is beneficial for restoration.
16

---

Degraded Frame
LQ Frame
No CHM
TURTLE
GT Frame
Ground Truth
Do we need history? We present visual results TURTLE and TURTLE without CHM to motivate the need for summarizing the history and conditioning the restoration on the history of the input. Other than efficiency, also brings perceptual benefits. Notice how "no CHM" introduces smudges and blemishes in place of the guard railing in the balcony of the building since the region is obscured in the degraded input.
Figure 6:
Frome att -3 & Similar Point
Frame at - & Similar Point
Frame at - & Similar Point
Input Frame and Query
Figure 7: CHM Tracking, Visual illustration of CHM tracking query points in the frame history (frames previous to the input frame). the top row, we plot the correctly tracked points, while the bottom row visualizes the limitations in the case of redundant patterns. We plot the query and most similar points on input frames for ease of exposition, but in practice, they function on feature maps.
CHM Placement. Our experiment, in table 12, indicates that having CHM in both the latent and decoder stages is necessary for optimal performance. In the latent stage, the spatial resolution is minimal, and CHM provides greater benefit in the following decoder stages as the spatial resolution increases.
Softmax Ablation. In table 13, we verify that topk selection is necessary to allow the restoration procedure to only consider relevant information from history. Since softmax does not bound the information flow we observe non-trivial performance drop when topk is replaced with softmax. We
17

---

Historyless FFN
Trans Block
LayerNorm
313. degth comv
1x 1x2, co.c
LayerNorm
ChannelMlLe
LayerNorm
1n1. moint-com
degth-comm
Channel
Attention
LayerNorm
politt
Gelu
363,
Figure 8: Illustration of Historyless FFN. Transformer block is similar in spirit to the block introduced in [79]. while the Historyless FFN's design takes inspiration from the blocks [15, 11].
argue that topk prevents the inclusion of unrelated patches, which can, potentially, introduce irrelevant correlations, and obscure principal features.
B TURTLE'S Specifications & Details
We motivate TURTLE's design and present empirical results to pinpoint the benefits modeling the history in the case of video restoration. Further, we present additional details of the proposed method, TURTLE, and expand on the construction of the architecture.
B.1 Motivation: Causal History Model
Recall that the Causal History Model (CHM) is designed to model the state and compensate for motion across the entire history relative the input. It then learns to control the effect of history frames by scoring and aggregating motion-compenn cetpon-comeensttedd features based on their relevance to the restoration of the current frame. Such procedure allows for borrowing information from the preceding temporal neighbors the input frame. In table 9, we ablate CHM indeed provides the performance boost. Moreover, in fig. 6, present visual results the video raindrops and rain streaks removal task to motivate the need for summarizing the frame history as part of the restoration method. We train TURTLE without the CHM block, referred to "no CHM",, following TURTLE's experimental setup, and for fair comparison, we keep the model size consistent. TURTLE, equipped with CHM, maintains the spatial integrity of the input without introducing faux textures blur even though the region (see guard railing in the balcony of the building) is entirely obscured by raindrops and streaks the degraded input. However, without CHM block, unwanted artifacts (such holes and blemishes in place the guard railing in the balcony) are introduced to fill in the missing information since no information is borrowed from preceding frames. Note that in the case of no CHM experiment, feed two concatenated frames (one frame previous to the input and the input frame) to the architecture.
B.2 Historyless FFN.
Recall that TURTLE's encoder is historyless, i.e., it employs no information about the history of the input. Further, we opt for a feedforward style design the encoder with convolutional layers. This because shallow representations at this stage are not sufficiently compressed and are riddled with degradation. Thus, expensive attention-baun based operators provide no significant performance benefit but add to the computational complexity. The diagrammatic illustration of Historyless FFN and the Transformer block [79] used in CHM presented in fig. 8.
C Limitations & Discussion
This section discusses the limitations of our proposed CHM. While CHM is adept tracking similar patches across the history of frames, encounters challenges in certain scenarios. For instance, as demonstrated in the zebra example in fig. 7, CHM identifies similar patches on different parts of the zebra's ss body due their redundant patterns, even though these patches are not located in the same area. Moreover, in case of severe input degradation, CHM's capacity to accurately identify and utilize similar patches may diminish due to spurious correlations, which could affect its ability to use history effectively for restoring the current frame.
18

---

Degraded
DSTNet
TURTIF
Figure 9: Visual Results on Synthetic Video Deblurring. We present additional qualitative analysis on synthetic video deblurring on the GoPro dataset [41]. We compare TURTLE with DSTNet [45] on two frames taken from two different videos of the testset.
C.1 Societal Impact
This work presents an efficient method to advance the study of machine learning research for video restoration. While the proposed method effectively restores the degraded videos, we recommend expert supervision in medical, forensic, or other similar sensitive applications.
D Relationship to State Space Models
We present special case of the proposed Causal History Model, CHM, wherein the videos degraded, and each frame is optimally compensated for motion with respect to the input. Lemma D.1. (Special Case of Causal History Model) In the absence of degradation and optimally compensated motion through optical flow, the state history H."" then, only depends on the input F. and the previous state !!!! 1-1- Under this assumption. eq. (1), and eq. (2) can be rewritten
i [] (fi + (F
(8)
y | = CAA.").
where A,, B,, C₁ parameters learned through gradient descent, and F! is the input feature map at timestep t. In this case, eq. (8) is realizable and not flawed, given the motion-compensatte input. This assumption allows the model to be learned in similar fashion to HiPPO [18] or Mamba [17]. More specifically, the Causal History Model (CHM) reduces to an equivalent time-variant and input-dependent flavor of the State Space Model (SSM).
Proof 1. Consider a state space model (SSM) [16] that maps the input signal F, to the output signal yt through implicit state H,, i.e.,
H, (H, + B(F,),
(10)
y: C(H.).
(11)
In the above equations, we abuse the SSM notation for exposition. Recall that we consider the special case wherein the motion is compensated for, and there is no degradation the input video. this
case, we can say that history and motionncompenssted history are equal i.c., A, H,. Now, consider
the first frame of the video at timestep = 0. Let the initial condition be denoted by Fo, then we can write the eq. (10), and eq. (11) as,
Hg = (Fo), because H,-1= 0
(12)
C (H)):
(13)
where Ag. Bo. Co are learnable parameters. Then for the next timestep i 1, we can write that
Å = (f,)) Bi (F.),
(14)
y1 C (
(15)
19

---

From eq. (12), we know that Ho Bu (F)), then we can re-write eq. (14) as
H1 A Bu Fo)) + (F
(16)
A Bu Fo) (1)..
The output y1 then be written as,
y1 C₁ (H))
(17)
At every timestep i, the output can be written in terms of the history (previous timestep) and the input (at current timestep). Now, consider the case 2, and let the frame features be denoted by F2, then we can write,
H2 = A₂ [ (22
(18)
A A B (Fo))) + (1(F + (2 because eq. (16)
therefore,
H A (F())) + B (F)) + (F2).
(19)
History
Input
Notice, how in eq. (19) H2 is written in terms of the input frame F2, and the previous frames Fo. and Fi. The output y2 then computed as,
Y2 C2 (H2).
(20)
We can then generalize eq. (18), and eq. (20) to any timestep t, and we arrive at eq. (8), and eq. (9), i...
A. Yr (H.))
+ B.F,,),
(21)
(22)
Therefore, the model can be learned in Mamba [17], or HiPPO [18] style, and the parameters A, B, C can be learned with gradient descent since the linear relationship between the input and the output tractable this case.
In practice, however, the degradation assumption does not hold. Therefore, naive state up- date, eq. (8), renders sub-optimal results. This is because, in video processing tasks, motion governs the transition dynamics, i.e., the state evolves over time due motion, and therefore, any linear relationship between the output and the input is intractable unless the motion is compensated for.
E Further Visual Comparisons
We present additional visual results, comparing our method with previously available methods in the video restoration literature.
E.1 Synthetic Video Deblurring
We present further results on synthetic video deblurring task on the GoPro dataset [41] in fig. 9. We compare TURTLE with the previous method in the literature that is computationally similar TURTLE, DSTNet [45]. Turtle avoids unnecessary artifacts in the restored results (see the tire undergoing rotation in the frame in the top row in fig. 9). Further, the restored results not smudged, and textures are restored faithfully to the ground truth (see feet of the person in the frame in the bottom row).
E.2 Video Raindrops and Rain Streaks Removal
Unlike just rain streaks, raindrops often pose more complex challenge for restoration algorithms. This is because several video/image restoration methods often induce blurriness in the results. Raindrops get mixed with the background textures; therefore, minute details such numbers or text are blurred in restored results. However, since TURTLE utilizes the neighboring information, learns to restore these details better. We observe this fig. 10, where the previous best method ViMPNet [71] blurs the number plate on the car (in last row), or introduces faux texture on the building (in second row). On the contrary, TURTLE better restores the results and avoids undesired artifacts or blur.
20

---

Deg rededded Frame
Truth
VIMPNet
TURTLE
Figure 10: Additional Visual Results on VRDS. We present additional qualitative analysis on video raindrops and rain streaks removal (VRDS) tasks. We compare TURTLE with ViMPNet [71], the best method literature, on three frames taken from the testset. TURTLE'S results are artifacts-free as it can effectively remove both the streaks and drops. However, ViMPNet [71] tends to mix in the raindrops with the background, introducing smudge (see the building) and blur (see number plate on the car).
Degraded
Ground
SVDNet
Figure 11: Additional Visual Results on RSVD. We present additional qualitative analysis on the desnowing task. We compare TURTLE with SVDNet [10], the best method in literature, two frames taken from the testset. TURTLE removes even smaller snowflakes (flecks snow on the underside of orange roof) and differentiates between textures and snow (see white spots on sheep).
E.3
Video Desnowing
We present additional visual results on the video desnowing task in fig. 11. We compare TURTLE with SVDNet [10], the previous best method the literature on task. Our method removes snowflakes effectively and differentiates between them and similar-sized texture regions in the background (see white spots on the sheep wool in the top row) without requiring any snow prior like SVDNet [10]. Although SVDNet [10] removes snowflakes considerable extent, fails to remove smaller flecks of snow comprchensively. However, TURTLE's restored results are visually pleasing and faithful to the ground truth.
E.4 Nighttime Video Deraining
In fig. 3, we presented the visual results comparison MetaRain [47]. For fair comparison, we resized the outputs to 256 x 256 following the work in [47, 35]. However, in fig. 12, we present TURTLE'S results on full-sized frames taken from two different testset videos. Our method preserves the true colors of the frames and removes rain streaks from the input without introducing
21

---

Degraded Frame
Ground Truth
TURTLE
Figure 12: Additional Visual Results on Nighttime Deraining. We present additional visual results on the nighttime deraining dataset [47]. TURTLE maintains color consistency, is artifact-free, and is more faithful to the ground truth.
unwanted textures discoloration. Note that in table 1, we compute PSNR in Y-Channel following MetaRain [47] since NightRain [35] did not release their code, and their manuscript does not clarify if the scores computed in RGB color space or in Y-Channel. Nonetheless, we report PSNR score in RGB color space to allow comparison with NightRain [35] regardless: TURTLE scores 27.68 dB in RGB color space.
E.5 Real-World Video Deblurring
Different from synthetic video deblurring (see fig. 9), in real-world deblurring, the blur is induced through real motion, both of camera, and object. In fig. 13, we present visual results on three frames taken from three videos from the testset of the BSD dataset (3ms-24ms configuration) to complement the quantitative performance of TURTLE in table 3. TURTLE restores the video frames with high perceptual quality, and the resultant frames are faithful to the ground-trume and are visually pleasing.
E.6 Real-World Weather Degradations
In fig. 14, present qualitative results of TURTLE real-world weather degradations. The purpose of these results to verify generalizability of TURTLE on non-synthetic degradations. Therefore, in addition to real-world video superresolution (see fig. 5), and real-world deblurring (see fig. 13), we also consider real-world weather degradations. We download four videos randomly chosen from a free stock video website. First two videos (in first two columns) while the last two are by rain. Notice how in the last video (in the last column), there is also haze that affects the video. TURTLE removes snow, and rain (including haze) and the restored frames visually pleasing. Given the lack of ground-truth in these videos, we do not report any quantitative performance metric.
afflicted by snow degradation,
F Computational Profile of TURTLE
In table 14, we report the runtime analysis of the proposed method TURTLE on a single 32GB V100 GPU, and compare with three representative general video restoration methods in the literature, namely ShiftNet [29]. VRT [34], and RVRT [33]. We consider four different input resolutions varying from (256 x 256 x 3) 1080p. Prior methods exhibit exponential growth GPU memory requirements as the resolution increases, TURTLE, however, features linear scaling in GPU memory usage, underscoring its computational efficiency advantage. As the resolution increases beyond 480p, all of the previous methods throw 00M requirement exceeded the total available memory (of 32GB). On the flip side, TURTLE can process the videos even at a resolution of 1080p on the same GPU.
errors indicating that the memory
22

---

Degraded Frame
Ground Truth
F
Visual Results on World Video Deblurring, We present visual results of TURTLE on the real-world video deblurring task BSD 3ms-2ims dataset [83, 82] on three frames taken from three different videos in the testset, Our method restores the frame with high perceptual quality.
Figure 13:
G Additional Literature Review
We further the discussion on prior art in the literature in terms of temporal modeling of videos, and causal learning.
Temporal Modeling. In video restoration, temporal modeling mainly focuses on how the neighbor- ing frames (either in history or in the future) can be utilized to restore the current frame better. For such a procedure, the first step usually involves compensating for motion either through explicit meth- ods (such as using optical flow [33, 34, 36, 7J), or implicitly (such as deformable convolutions [63], search approaches [64], or temporal shift [29]). A few works in the literature focus on reasoning at the trajectory level (i.e., considering the entire frame sequence of video) [37] through learning to form trajectories pixel can borrow information from the entire trajectory instead of focusing a limited context. The second step is then aggregating such information, where in the case of Transformers, attention is employed, while MLPs are also used in other cases.
each pixel (or some group of pixels). The motivation is that in this case, each
Causal Learning in Videos. In videos, causal learning is generally explored in the context of self- supervised learning to learn representations from long- onn--coteext videos with downstream applications various video tasks such as action recognition, activity understanding, etc [50, 73]. In [2], causal masking of several frames at various explored. the best of our knowledge, other than one streaming video denoising method [49], almost all of the state-of-the-art video restoration methods are not causal by design since they rely on forward and backward feature propagation (i.e., they consider both frames history and in the future) either aligned with the optical flow otherwise [6, 7, 33, 29]. However, there is significant
regions as a strategy to leam the representations
23

---

Degraced Frame
oot fee sadd
TURTLE
Do bofffe shop
Figure 14: Real-World Weather Degradations. We present visual results on real-world weather degradations. The samples taken from four different videos downloaded from free stock video website (www.pexels com). The first two columns contain frames from videos afflicted by snow degradation, while the last two are afflicted by rain degradations. TURTLE restores the frames reliably, and the resultant frames are pleasing to the eye.
Table 14: Profiling TURTLE. We profile the proposed method, TURTLE, on single 32 GB V100 GPU, and compare with 3 recent video restoration methods, namely ShiftNet [29]. VRT [34], and RVRT [33]. We consider different input resolutions and compute the per-frame inference time (ms), total MACs (G), FLOPs (G), and the GPU memory usage of the model. ODM denotes Out-Of Out-Of-Memory error i... the memory requirement exceeded the total available memory of 32GB.
Frame
Per Frame
GPU Memory Usage (MBs)
Methods
MACs (G)
Resolution
Inference Time (ms)
256 256 3
190
989
2752
ShiftNet [29]
640 x 480 x 3
510
5630
7068
1280 720 x 3
0OM
0OM
00M
1920 x 1080 x 3
DOM
DOM
00M
256 256 3
455
1631
3546
VRT
640 x 480 x 3
2090
7648
11964
[34]
1280 720 3
DOM
00M
1920 x 1080 x 3
DOM
DOM
DOM
256 256
252
1182
5480
RVRT
640 x 480 x 3
1240
10588
21456
[33]
1280 720
DOM
0OM
00M
1920 x 1080 x 3
0OM
OOM
00M
256 x 256
95
181
2004
640 X 480 x 3
380
812
4826
TURTLE
1280 X 720
1180
2490
11994
1920 X 1080 3
2690
5527
24938
amount of work on causal representation learning where the aim to recover the process generating the data from the observation to learn the disentangled latent representation. Note that this is out of the scope of this work.
H Dataset Information & Summary
All of the experiments presented in this manuscript employ publicly available datasets that are disseminated for the purpose of scientific research on vidco/image restoration. All of the datasets

---

employed are cited wherever they are referred to in the manuseript, and we summarize the details here.
Video Desnowing: We utilize the video desnowing dataset introduced in [10]. The dataset is made available by the authors at the link: Video Desnowing
Video Nighttime Deraining: We utilize the nighttime video deraining dataset introduced in [47]. The dataset is made available by the authors at the link: Nighttime Deraining
Video Raindrops and Rain Streaks Removal: We utilize the video raindrops and rain streaks removal (VRDS) dataset introduced in [71]. The dataset made available by the authors the link: VRDS
Synthetic Video Deblurring: We employ the GOPRO dataset introduced in [41]. The dataset is made available by the authors at the link: GOPRO
Real Video Deblurring: We employ the BSD dataset introduced in [82, 83]. The dataset is made available by the authors the link: BSD
Real-Worde Video Super Resolution: We utilize the MVSR dataset introduced in [67]. The dataset is made available by the authors at the link: MVSR
Video Denoising: employ DAVIS [48], and Set8 [61] datasets video denoising. The datasets are available at: DAVIS-2015 Set8 [4 sequences], and Set8 [4 sequences]
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The introduction (section 1) lists the contributions of this work, and we back the claims up empirically in section 4, followed by ablations in section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitation of the proposed method in appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
This is mainly an empirical work, however we do discuss a theoretical inter- pretation and provide complete proof, and list the set of assumptions in appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the proposed architecture detail, and provide all the experimen- tal details necessary to reproduce the results in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We provide readme file, along with the code, that lists the steps to run the experiments. Further, the links to download the datasets used the experiments reported in this manuscript are summarized in appendix H.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //wpsscc... wwtpccpccpgguupiieeecceeecooceee gwwd//////////gg /Cons//bmddeeacacac
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail the hyperparameters, train/test splits of the data used in section 4, and also ablate the necessary hyperparameters section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We fix the seed in all experiments for reproducibility following the standard in restoration literature.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail the hardware, along with the software, stack we used to run the proposed method in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: //neurip///uris..... cctppcciccccccccccccccccccccccct ctwpeiieeeeiichhieeeeeeeeeeeeeee
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
The research conforms with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader impacts of our work, specifically societal implications in appendix C.I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: To the best of our knowledge, we believe that the work done in this manuscript poses such risk. However, highlight societal impact in appendix C.I.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In all of the experiments reported in this manuscript, use publicly available datasets released by their respective authors for research purposes. Although we cite the original work wherever the dataset is discussed, we also summarize the details in appendix H.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This work does not create new datasets. However, the code released is accompanied by README that details the steps necessary re-produce the results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "xWQAGMdPan", "reviewer": "Reviewer_KLsv", "summary": "The paper presents a novel framework TURTLE for video restoration. TURTLE aims to improve video restoration tasks by modeling and utilizing truncated historical data of input video frames to enhance the restoration quality while maintaining computational efficiency. The proposed method demonstrates state-of-the-art results across various video restoration benchmarks, including desnowing, deraining, deblurring, and super-resolution.", "questions": "-  Line 24, the author claims recurrent designs often result in cumulative errors. I think it would be better to explain why a truncated causal history model can avoid this phenomenon.\n- As shown in Fig. 7, how many frames are affected by incorrect tracking query points? Will this reduce the restoration quality of subsequent frames?\n- I noticed a latent cache block in your code that is barely mentioned in the paper. Could you explain its function and impact on the model's performance?\n- Please refer to the weaknesses above.", "limitations": "The authors have largely addressed their limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The introduction of a truncated causal history model is novel and addresses the limitations of both parallel and recurrent video processing methods.\n- The paper thoroughly explains the TURTLE architecture and its components, including the encoder, decoder, and Causal History Model (CHM).\n- The method achieves superior results on multiple video restoration benchmarks, showing clear improvements over existing techniques. The method is evaluated across various video restoration tasks quantitatively and qualitatively.", "weaknesses": "- This paper did not discuss the parameters and FLOPs of this model.\n- Lack of comparison with  Restormer: Efficient transformer for high-resolution image restoration. (CVPR2022), A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift.  (CVPR2023)\n- Lack of visualization results of the effectiveness of truncation factor and value of topk.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reviewing our work and providing constructive comments.\n\n**W1.** FLOPs of Turtle.\n\nWe have provided the Turtle's MACs (G) in Table 8 of the main paper and discussed its comparison to several previously published video restoration methods in Section 4.8. Additionally, during rebuttal, we have profiled the proposed method, Turtle, and compared it with previous methods, including ShiftNet [28], VRT [32], and RVRT [33], as reported in Table 1 of the global rebuttal. We computed per-frame inference time (ms), MACs (G), FLOPs (G), and GPU memory usage (MB).\n\nFrom these comparisons, we can see Turtle is significantly more computationally efficient. ShiftNet considers a context of 50 frames, VRT considers 16 frames, and RVRT also uses 16 frames. Both VRT and RVRT rely on fine-tuning an optical flow model (SpyNet) to restore a video. In contrast, Turtle only considers a single frame and conditions its restoration on the recent history of the current frame up to a truncation factor (τ = 3), which reduces computational costs and enhances efficiency.\n\nThis computational efficiency is achieved by the proposed CHM which contains a state align block (SAB) to retrieve only top-k relevant patches from recent frames to help restore each patch, and a Frame History Router (FHR) to further route attention to features in the most relevant historical frame. Both SAB and FHR together achieve efficient temporal cross-frame attention calculation.  \n\n\n**W2.** Lack of comparison with Restormer, and ShiftNet.\n\nRestormer [76] is designed for Image Restoration and does not account for the temporal nature of video frames. Despite this difference, here we provide a comparison with Restormer on the GoPro video deblurring task. Unlike video restoration methods, Restormer treats video frames as individual images. Consequently, it is uncommon in the literature to compare video restoration methods with image restoration methods as it is not a fair comparison.\n\n| **Methods** | **Task**          | **PSNR** | **SSIM** |\n|-------------|-------------------|----------|----------|\n| Restormer   | Image Restoration | 32.92    | 0.961    |\n| Turtle      | Video Restoration | **34.50**    | **0.972**    |\n\n**Table 2**: _Comparison of Turtle with Restormer on the GoPro deblurring task in terms of PSNR and SSIM metrics._\n    \nWe have not compared our model with ShiftNet due to the immense difference in computational cost, where a major benefit of Turtle is its low computation cost. In Table 1 (refer to the global rebuttal), we provide a detailed comparison of Turtle with ShiftNet [28] in terms of FLOPs, GPU memory consumption, and inference time for a single frame on a 32 GB GPU. To compute the numbers in Table 1, we considered an 8-frame context for ShiftNet, as it was not feasible to run even the smallest ShiftNet model with its full 50-frame context length, which was used to report the performance in their paper, on a single 32 GB GPU. Additionally, it is important to note that ShiftNet considers a 50-frame context, both in the future and in history. Therefore, unlike Turtle, ShiftNet is not suitable for online video restoration (a typical use case in streaming scenarios) since it relies on future frames.\n\n\n**W3.** Lack of visualization results of the effectiveness of truncation factor and value of top-k.\n\nDuring rebuttal, we have added the visualization results for the value of top-k (with k = 5 and k = 20); refer to Fig. 2 in the one-page PDF. As for the effect of the truncation factor, increasing the truncation factor will increase computational costs. Our experiments, presented in Table 10, revealed that raising the truncation factor from 3 to 5 did not enhance PSNR scores or visual quality and merely increased computational expenses. Conversely, increasing the truncation factor from 1 to 3 led to visual improvements and higher PSNR scores, which justifies the slight rise in computational cost. Thus, we have chosen a truncation factor of 3 to be used in Turtle.\n\n**Q1.** How truncated causal history model can avoid cumulative errors?\n\nThis can be seen through Eq (1) and (2). Although like RNN models, in Turtle, the causal history state $\\hat{H_t}$ given by (1) will also recursively depend on prior causal history state $\\hat{H_{t-1}}$, yet we notice through Eq(1) and (2) that when decoding $y_t$, Turtle only needs to rely on the current frame $F_t$ and the recent truncated history of $H_t$, where $\\hat{H_t}$ (although recursive itself) is extracted by retrieving topk patches from $H_t$ to align with each patch in $F_t$ as additional channels. Therefore, the errors only depend on $F_t$ and $H_t$ with a small truncation factor and do not accumulate.  \n\n**Q2.** As shown in Fig. 7, how many frames are affected by incorrect tracking query points? Will this reduce the restoration quality of subsequent frames?\n\nIn some cases, where the input frames are extremely degraded or occluded (for instance, with snow covering the point of interest), the points found in the previous frames can be slightly wrong. In Fig. 7, we see this phenomenon with Zebra's front part of the torso. In frames at time t-1, t-2, etc., the region of interest is blurred and occluded due to snow and haze. Therefore, in such cases, some mismatch can occur. However, notice that the most similar points are still somewhere on the Zebra's torso, and not on other random regions (like grass).\n\n**Q3.** Latent Cache Block in code.\n\nThe latent block (or latent cache block as in the code, or Middle Block in Figure 1 in the main paper) is no different than the decoder blocks in principle. The only difference is in the number of CHM blocks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsJ1sRwPGT", "reviewer": "Reviewer_SsXx", "summary": "This work presents a video restoration framework named TURTLE, which stands for truncated casual history model. The key innovation of TURTLE is its ability to efficiently model the transition dynamics of video frames governed by motion, a critical challenge in video restoration. Unlike traditional methods that process multiple contextual frames simultaneously, TURTLE enhances efficiency by storing and summarizing a truncated history of the input frame's latent representation into an evolving historical state.", "questions": "Why is it necessary to use a Casual History Model in each decoder? I believe applying CHM in the latent space should be sufficient. This raises concerns about the actual inference efficiency of the model.\nThere is a lack of comparison regarding actual inference time, as well as a crucial study on the actual VRAM usage at high resolutions, both of which are very important; the reference value of GMacs alone is limited.", "limitations": "Please refer to the weaknesses.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper introduces a novel video restoration technique that leverages a truncated causal history model, which is a unique approach to handling the transition dynamics of video frames influenced by motion. This represents a significant advancement in the field of video processing.\n2.\tThe TURTLE framework achieves state-of-the-art results across a wide range of video restoration tasks, including desnowing, deraining, raindrop and rain streak removal, super-resolution, deblurring, and blind video denoising. The consistent high performance across different tasks underscores the effectiveness of the proposed method.\n3.\tTURTLE is designed to be computationally efficient, which is a critical consideration for practical applications. The framework reduces computational costs compared to existing methods while maintaining high performance, making it suitable for resource-constrained environments.", "weaknesses": "1. The section on related work is seriously lacking in content; the author should revise this section to include a more detailed explanation of related work, especially in the areas of temporal modeling and Causal Learning.\n\n2. There is a scarcity of Visual Comparisons, and many tasks lack real-world sample comparisons. This is a serious issue, and I hope the author can provide a complete set of comparisons.\n\n3. Why is it necessary to use a Casual History Model in each decoder? I believe applying CHM in the latent space should be sufficient. This raises concerns about the actual inference efficiency of the model.\n\n4. There is a lack of comparison regarding actual inference time, as well as a crucial study on the actual VRAM usage at high resolutions, both of which are very important; the reference value of GMacs alone is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and reading our work.\n\n**W1.** The section on related work is seriously lacking in content.\n\nDue to space constraints, we had to be selective in related work. In the rebuttal, here we would like to add a literature review on temporal modeling and causal learning for videos as suggested.\n\n**Temporal Modelling:** In video restoration, temporal modeling mainly focuses on how the neighboring frames (either in history or in the future) can be utilized to better restore the current frame.\nFor such a procedure, the first step usually involves compensating for motion either through explicit methods (such as using optical flow [32], [33], [35], [5]), or implicitly (such as deformable convolutions [61], search approaches [62], or temporal shift [28]). A few works in the literature focus on reasoning at the trajectory level (i.e., considering the entire frame sequence of a video) [36] through learning to form trajectories of each pixel (or some group of pixels). The motivation is that in this case, each pixel can borrow information from the entire trajectory instead of focusing on a limited context. The second step is then aggregating such information, where in the case of Transformers, self-attention is employed, while MLPs are also used in other cases.\n\n**Causal Learning for Videos:** In videos, causal learning is generally explored in the context of self-supervised learning to learn representations from long-context videos with downstream applications to various video tasks such as action recognition, activity understanding, etc. In [85], causal masking of several frames at various spatio-temporal regions as a strategy to learn the representations is explored. To the best of our knowledge, almost all of the state-of-the-art video restoration methods are not causal by design since they rely on forward and backward feature propagation (i.e., they consider both frames in history and in the future) either aligned with the optical flow or otherwise [32], [33], [5].\n\n[85] Bardes, Adrien, et al. \"Revisiting feature prediction for learning visual representations from video.\" arXiv preprint arXiv:2404.08471 (2024).\n\n**W2.** Visual Comparisons.\n\nWe have already provided extensive visual comparisons between Turtle and state-of-the-art baseline methods in the main paper for a range of tasks on respective benchmark datasets, including desnowing in Figures 3 and 11, night deraining in Figure 3, deblurring in Figures 4 and 9, raindrop removal in Figure 4, and blind video denoising in Figure 5. These visual comparisons on done on the benchmark dataset where the methods are evaluated in order to align with the reported numerical performance comparisons in tables, which is following standard practice in the literature. We also have included a visual comparison on real-world video super-resolution in Figure 5.\n\nDuring the rebuttal, we added more real-world Visual Effect results, including real-world deblurring from the BSD 3ms-24ms dataset and removing snow from real snowy videos taken from www.pexels.com (a free stock videos website). Please refer to Figure 1 in the one-page PDF for the visual results.\n\nThese results when put together show the competence of Turtle not only numerically but also in terms of visual effects.  \n\n**W3.** Is CHM in Latent sufficient?\n\nIn rebuttal, we add an ablation study for the potential benefits of adding CHM at the decoder stage compared to adding it only at the latent stage. The ablation results are included in Table 1 of the one-page PDF. This table is also shown below:\n\n| **Ablations**                    | **PSNR** |\n|----------------------------------|----------|\n| No CHM                           | 31.84    |\n| CHM in Latent                    | 32.05    |\n| CHM in Latent & Decoder (Turtle) | **32.26**    |\n\n**Table 1**: _Ablation experiments on CHM placement in latent, and latent and decoder._\n\nOur experiment indicates that having CHM in both the latent and decoder stages is necessary for optimal performance. In the latent stage, the spatial resolution is minimal, and CHM provides greater benefit in the following decoder stages as the spatial resolution increases. \nFurthermore, we also calculated the computational overhead of CHM, and show that out of 181 MACs (G) of Turtle, all CHM blocks collectively contribute 11.8 MACs (G), which accounts for 7% of the overall computation cost to achieve temporal modeling. In contrast, processing and restoring multiple frames in parallel with ShiftNet [28], learning trajectories in TTVSR [26], or deploying an additional optical flow network [32,33] entail considerably higher costs.\n \n**W4.** Lack of comparison at different resolutions.\n\nWe profiled the proposed method Turtle and compared it with previous video methods ShiftNet, VRT, and RVRT. We compute per-frame inference time (ms), MACs (G), FLOPs (G), and GPU memory usage (MBs) at varying input resolutions (refer to Table 1 in global rebuttal).\n\nIn practice, note that ShiftNet considers a context of 50 frames, VRT considers a context of 16 frames, and RVRT considers a context of 16 frames for denoising and deblurring. For superresolution, a total of 30 frames are fed. Note that both VRT and RVRT also rely on an optical flow architecture (SpyNet) and fine-tune it during training for the restoration procedure. These factors significantly increase their computational costs and limit their deployability.\n\nHowever, Turtle stands out by only considering a single frame and conditioning the restoration on the history of the current frame up to a truncation factor (τ =3). This setup significantly enhances efficiency, allowing Turtle to perform inference at varying spatial resolutions on a single GPU. This capability is crucial for the successful deployment of video restoration models on hardware-constrained devices, where the availability of multiple GPUs is often limited or impractical.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xloFKw5rOr", "reviewer": "Reviewer_BXMC", "summary": "This paper proposes a truncated causal history model (TURTLE) for video restoration. TURTLE is in a U-Net manner with a historyless encoder and a history-based decoder. The decoder has a causal history model (CHM), which is the core part of the TURTLE. The CHM injects history frames into a hidden state $\\mathbf{H}_t$ with the state align block, which is based on a cross-attention mechanism and a top-k selection strategy. The current frame is then fused with the hidden state with cross-frame attention. Experiments on several tasks demonstrate that the proposed methods can outperform the comparable methods.", "questions": "1. The organization of ablation studies is inappropriate.  \n1.1 It's improper to put the tables of ablation studies into the appendix.  \nThe ablations of some parts are missing.  \n1.2 The ablation study of top-k fusion in the state align block is missing. Most of the time, the information fusion operation in the top-k selection position of CHM is implemented by a softmax. How about using a softmax to replace top-k, i.e., remove the top-k selection strategy?  \n1.3 The implementation of the ''No CHM'' in the ablations is unclear. It states ''two frames are concatenated and fed to the network directly'', which is unclear. What network are the two frames fed to?  \n1.4 What is the patch size $p$ in the state align block? How about using different $p$?\n\n2. There are important video restoration methods absent in the comparable tables.  \n[1] K. Zhou, et al., Revisiting Temporal Alignment for Video Restoration. CVPR 2022.  \n(Link of codes: https://github.com/redrock303/Revisiting-Temporal-Alignment-for-Video-Restoration)  \n[2] D. Li, et al., A Simple Baseline for Video Restoration with Grouped Spatial-Temporal Shift. CVPR 2023.  \n(Link of codes: https://github.com/dasongli1/Shift-Net)\n\n3. Recurrent structures in neural networks can bring considerable costs. How does the CHM? In other words, the comparisons of running time and GPU memory cost are absent.\n\n4. Some of the datasets are uncommon. For example, in the video super-resolution experiments, this paper selects MVSR4X as the dataset. However, the recent comparable methods (such as BasicVSR and VRT)  use REDS and Vimeo-90k as benchmarks. It seems that in the Table. 7, the TURTLE outperforms BasicVSR++ a lot, but is the BasicVSR++ model used in this paper trained for being evaluated on MVSR4X?\n\n5. Do all the models for each task in the paper use the same number of input frames? If not, how to explain the rationality of the experiments?\n\n6. What are the training datasets for each task? Are all the methods in the same comparable table training in the same dataset (or under the same settings)?\n\n7. It looks like the hidden state $\\mathbf{H}$ in the CHM needs some frames to start. How to handle the boundary frames of videos？\n\n8. This paper emphasizes the proposed casual history model, but in the method part, it looks like the core part is a module for alignment (state align block). For extracting features from hidden states for restoration, what is design motivation? It would be better to introduce the components in CHM except the state align block with more details.", "limitations": "Please see the Questions part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The hidden state strategy provides a new thought for video processing tasks.\n2. The proposed method outperforms comparable methods on several video restoration tasks.", "weaknesses": "1. The organization of ablation studies is inappropriate.\n2. Some of the datasets are uncommon.\n3. Some key information is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments.\n\n**Q1.1.**\nWe placed ablation studies in appendix given space limit, but can move them to the main paper.\n\n**Q1.2.**\nDuring rebuttal, we have added ablation experiments comparing softmax and topk (k=5), as shown in Table 2 of the one-page PDF. The main paper included experiments on the effect of k in topk patch selection in Table 11. Softmax is a special case of topk method with k equal to the total number of patches in the frame (retrieving all patches). We can see that topk is better than Softmax, as it prevents the inclusion of unrelated patches and contributes critically to performance and computation efficiency.\n\n**Q1.3.**\nTurtle takes a single frame from the video at time $t$ as input; the encoder processes this frame, while the middle and decoder stages condition restoration on historical frames. In `No CHM` setting, we concatenate frames at times $t$ and $t-1$, which are fed into the same architecture as presented in Fig. 1(a) and Sec. 3.1, without CHM blocks in middle or decoder stages—only Transformer blocks are used.  We used the same-sized models by adjusting the dimensions/blocks in the `No CHM` setting to match the size of Turtle.\n\n**Q1.4.**\nWe use a patch size of 3x3. CHM is effective as long as the patch size is reasonable. If the patch size is too small, attention computations shift to a pixel level, making it harder to find similar patches since each pixel holds limited information. Conversely, if the patch size is too large, top-k filtering becomes less effective due to excessive similarities across the frame's spatial resolution. Therefore, a patch size of 3x3 or 4x4 yields optimal results.\n\n**Q2.**\nWe added comparisons to RTA (CVPR'22) for video denoising at two noise levels, sigma = 30, 50, in Table 3 in one-page PDF. Unlike RTA that requires both the frame and an additional noise map as input, Turtle is completely blind to noise levels when processing a degraded input frame, making the task harder. We did not include ShiftNet as it is significantly more costly than Turtle, although runtimes and other comparisons are presented in Table 1 for a reduced ShiftNet on a single GPU in Global Rebuttal.\n\n**Q3.**\nWe profile Turtle and compare it with 3 state-of-the-art video restoration methods, ShiftNet, VRT, and RVRT on varying spatial resolutions, in Table 1 of Global Rebuttal. Turtle can process videos at varying spatial resolutions on a single 32GB GPU, while others report out-of-memory errors as the resolution increases. A major benefit of Turtle is that it processes a single frame and uses the truncated history, instead of processing several frames in parallel in multiple branches, and thus is significantly faster and more efficient.\n\n**Q4.**\nOur focus is on video restoration, while we used MVSR4x to show Turtle's generalizability to more tasks. In fact, MVSR4x is a more recent real-world SR dataset from CVPR 2023, while VRT and BasicVSR were introduced earlier in 2022 and 2021 and thus didn't evaluate on MVSR4x. MVSR4x is also a challenging dataset, featuring lower resolution frames from phone cameras, resulting in lower PSNR scores compared to REDS and Vimeo90K.\n\nYes, BasicVSR++ is also trained on the MVSR4x dataset and tested on the heldout testset of MVSR4x. We have carefully ensured all comparisons are fair. In all tables, all the methods are directly trained on the same dataset and follow the same procedure as Turtle. We either take results from their original work or the work that introduced the dataset and retrained these methods on the proposed dataset after carefully tuning the methods (the case of MVSR4x paper, which retrained BasicVSR++ when introducing the dataset). \n\n**Q5.**\nThey do not. ShiftNet uses a context length of 50 frames, while VRT and RVRT utilize 16 frames. RTA uses 5 context frames. These methods compare against each other according to standard practices in video restoration literature. Although Turtle uses a truncated history of 3 frames, yet it achieves state-of-the-art results across different tasks, demonstrating its efficiency.\n\n**Q6.**\nYes, to ensure complete fairness in comparisons, all methods we compare are trained on the same dataset and follow the same train/test splits recommended in the literature. The datasets used are listed in Appendix F of the main paper.\n\n**Q7.**\nBoundary conditions occur in the first $\\tau$ frames. The first frame is restored without conditioning, second frame using the first frame as history, third frame using the first two frames and so on. Frame $\\tau+1$ will use history up to the set truncation factor $\\tau$.\n\n**Q8.** As Sec. 3.2 describes, overall rationale of CHM is best explained by Eqs. (1)(2), where (1) retrieves the relevant information from the prior frames $H_t$, transforms it into channels of (succinct) hidden features $\\hat{H_t}$ via $\\phi_t$, while (2) aims to attend current frame $F_t$ to this succinct historical feature to restore $F_t$ via $\\psi_t$. In detail, Eq (1) is implemented by State Align Block (SAB) $\\phi_t$, which for each patch in $F_t$, retrieves and only keeps topk relevant patches from a prior frame (in the truncated history) and “moves” those spatially to align with this patch as additional channels. This ensures that each patch in $F_t$ is not to be attending to all patches in a prior frame or to a patch at the same position in prior frame (since pixels have moved), but only to most relevant ones. Eq (2) is implemented by Frame History Router (FHR) $\\psi_t$, which achieves temporal correlation across frames through efficient channel attention with succinct historical features (instead of full attention between frames). For each patch in $F_t$, FHR “routes” attention to prior historical features that are the most helpful in restoring this patch. Therefore, both (1) and (2), i.e., both SAB and FHR are core to this design and work together to achieve said feature reduction and computational efficiency of Turtle.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "faa25QHSOI", "reviewer": "Reviewer_PswG", "summary": "This paper proposes a TURTLE to learn the truncated causal history model for video restoration tasks.\nThe proposed turtle's causal history model consists of two sub-modules: a State Align Block and a Frame History Router.\nThe state align block has a similarity-based retrieval mechanism that implicitly accounts for inter-frame motion and alignment from previous frames, and those retrieved representations are summarized and stored into a truncated history.\nThen the frame history router generates output frames by cross-frame channel attention with the motion-compensated history states.\nIn experiments, the proposed method shows a promising result in a number of video restoration tasks including deraining, desnowing, deblurring, and super-resolution.", "questions": "- Does the size of p1 and p2 affect the results?", "limitations": "It is surprising that the proposed method shows excellent performance in all various experiments.\nBut one thing may be complemented is that there is no strong guidance for temporal consistency, and I found there is a background flickering in the deraining scene in attached video.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed methods outperforms others in a number of video restoration tasks and benchmarks.\n\n- The proposed causal history model has a capability to consider spatio-temporal-channel correlation by the state align block (spatio-temporal) and the frame history router (channel-temporal).\n\n- It theoretically shows the link with the space state model.\n\n- It seems to increase efficiency by designing the encoder to process single frame only while the decoder to process multiple frames.", "weaknesses": "- As in the state align block, find similarity from history frames have previously proposed [1] and it seems the frame history router (channel-wise attention) is a new one proposed in this paper. However, there is no relevant ablation study with and without frame history router, so it is hard to see the improvement caused by it.\n[1] Learning Trajectory-Aware Transformer for Video Super-Resolution (CVPR22)\n\n- There is no detailed model architectures and model parameter (and runtime) comparisons with others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work important and for their insightful and constructive comments.\n\n**W1** Frame history router ablation study.\n\nIn the main paper, we conducted ablation experiments to analyze the effects of different components in CHM. We investigated three setups: No CHM, No State Align Block (only Frame History Router), and Turtle (both FHR and SAB), as shown in Table 9 (in the main paper). For clarity, we have reproduced the table below. The results demonstrate that using only CHM results in a +0.23 dB increase in PSNR and adding SAB on top of FHR provides an additional +0.19 dB increase in the ablation experiment.\n\n**Methods**| **PSNR** |\n------------------------------------------------------------|----------|\n| No CHM  | 31.84    |\n| No State Align Block (Only Frame History Router)           | 32.07    |\n| Turtle (Both State Align Block, and  Frame History Router) | **32.26**    |\n\n**Table 1**: _Ablating the CHM block to understand if both State Align Block, and Frame History Router are necessary._\n\nThe paper \"Learning Trajectory-Aware Transformer for Video Super-Resolution (TTVSR)\" [36] processes the current frame and multiple neighboring frames together, which is inefficient. Both TTVSR and Turtle use attention to find similar patches, but Turtle has distinct advantages. Turtle uses a history state $H_t$ and reuses features from past frames, reducing computational inefficiencies. While TTVSR learns the entire trajectory of each pixel (or group of pixels) in the video, Turtle limits the history and relevant patches. This is beneficial because the entire trajectory is often not useful in video restoration due to rapid changes caused by degradations like snow, rain, and raindrops.\n\nAdditionally, Turtle's Frame History Router (FHR) efficiently performs channel attention to correlate each patch in the current frame with the most relevant historical features, avoiding the overhead of full attention between frames while maintaining high restoration quality. The superiority of our proposed method is supported by the results in Table 7 and Table 5 in the main paper, which show that Turtle significantly outperforms TTVSR. Specifically, Turtle achieves a +3.96 dB increase in PSNR for Video Raindrop and Rain Streak Removal and a +1.7 dB increase in video super-resolution.\n\n**W2.** Runtime comparisons with other methods.\n\nWe profile Turtle and compare it with three previous state-of-the-art video restoration methods, ShiftNet, VRT, and RVRT on varying spatial resolution sizes in terms of runtime per frame (ms), GPU memory usage, FLOPs (G), and MACs (G), refer to Table 1 in global rebuttal. Turtle can process videos at varying spatial resolutions on a single 32 GB GPU, while all three other methods throw out-of-memory errors since the memory requirements exceed the total available memory.\n\n**Q1.** Effect of patch size.\n\nCHM (specifically SAB) is effective with reasonable patch sizes. If the patch size is too small, attention computations shift to a pixel-to-pixel level, increasing the computational cost and making it harder to find similar patches since each pixel holds limited information. Conversely, if the patch size is too large, topk filtering suffers due to excessive similarities across the frame's spatial resolution. Based on our experiments, a patch size of 3x3 or 4x4 yields optimal results.\n\n**L1.** No strong temporal guidance.\n\nFor the purpose of this work, we opted for a simple L1 loss function without any additional auxiliary losses. However, coupling some temporal consistency-based loss functions with the restoration loss can potentially alleviate such flickering [86] in very high frame-rate videos.\n\n[86] Dai, Peng, et al. \"Video demoireing with relation-based temporal consistency.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Profiling Turtle**

We profile the proposed method, Turtle, in terms of per-frame inference time (in ms), MACs (G), FLOPs (G), and GPU memory usage (in MBs) on a single 32GB Nvidia V100 GPU. ShiftNet uses a context length of 50 frames and restores all 50 frames together, while VRT uses (a context length of) 16 frames, RVRT uses 16 frames for deblurring and denoising, and 30 frames for super-resolution. However, these settings are not runnable on a single V100 GPU, which reports out of memory (OOM). Thus, for the purpose of generating this table, we chose a context length of 8 frames for ShiftNet and 2 frames for RVRT and VRT.

|  Methods |  Resolution | Per Frame Inference Time (ms) | MACs (G) | FLOPs (G) | GPU Memory Usage (MBs) |
|:--------:|:-----------:|:------------------------:|:--------:|:---------:|:----------------------:|
| ShiftNet |  256x256x3  |            190           |    989   |    1978   |          2752          |
|          |  640x480x3  |            510           |   5630   |   11260   |          7068          |
|          |  1280x720x3 |            OOM           |    OOM   |    OOM    |           OOM          |
|          | 1920x1080x3 |            OOM           |    OOM   |    OOM    |           OOM          |
|    VRT   |  256x256x3  |            455           |   1631   |    3262   |          3546          |
|          |  640x480x3  |           2090           |   7648   |   15296   |          11964         |
|          |  1280x720x3 |            OOM           |    OOM   |    OOM    |           OOM          |
|          | 1920x1080x3 |            OOM           |    OOM   |    OOM    |           OOM          |
|   RVRT   |  256x256x3  |            252           |   1182   |    2364   |          5480          |
|          |  640x480x3  |           1240           |   5294   |   10588   |          21456         |
|          |  1280x720x3 |            OOM           |    OOM   |    OOM    |           OOM          |
|          | 1920x1080x3 |            OOM           |    OOM   |    OOM    |           OOM          |
|  Turtle  |  256x256x3  |            95            |    181   |    362    |          2004          |
|          |  640x480x3  |            380           |    812   |    1624   |          4826          |
|          |  1280x720x3 |           1180           |   2490   |    4980   |          11994         |
|          | 1920x1080x3 |           2690           |   5527   |   11054   |          24938         |

**Table 1**: _We profile the proposed method, Turtle, on a single 32 GB V100 GPU, and compare with 3 recent video restoration methods, namely ShiftNet [28], VRT [33], and RVRT [32]. We consider different input resolutions and compute the per-frame inference time (ms), total MACs (G), FLOPs (G), and the GPU memory usage of the model. OOM denotes Out-Of-Memory error i.e., the memory requirement exceeded the total available memory of 32GB._

**Note on References in Reviewer-specific Rebuttals**

We use same the reference numbering as in the main paper. For example, ShiftNet is the reference [28] in the main paper. For papers that we cite additionally, we provide the references in their specific replies.

Meta-review: The reviewers were unanimous in accepting this paper, and rated it with an Accept, a Weak Accept, and 2 Borderline Accept.
They found the paper novel, its performance and evaluation convincing and the design computationally efficient. They raised 
some concerns but many of these were addressed in the rebuttal and in the direct messages to the reviewers, and reviewers 
have correspondingly adjusted their ratings.
The authors are encouraged to integrate the answers to these concerns in their paper, especially regarding the paper presentation.
The AC has read all the reviews, the rebuttal and the messages exchanged between the reviewers and the authors
and agrees with the reviewers that this paper should be accepted.
Decision: Accept (poster)