Learning Truncated Causal History Model for Video Restoration

Authors: mohammad  salameh
di  niu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 20:55:46

Abstract: One key challenge to video restoration is to model the transition dynamics of video frames governed by motion. In this work, we propose Turtle to learn the truncated causal history model for efficient and high-performing video restoration. Unlike traditional methods that process a range of contextual frames in parallel, Turtle enhances efficiency by storing and summarizing a truncated history of the input frame latent representation into an evolving historical state. This is achieved through a sophisticated similarity-based retrieval mechanism that implicitly accounts for inter-frame motion and alignment. The causal design in Turtle enables recurrence in inference through state-memorized historical features while allowing parallel training by sampling truncated video clips. We report new state-of-the-art results on a multitude of video restoration benchmark tasks, including video desnowing, nighttime video deraining, video raindrops and rain streak removal, video super-resolution, real-world and synthetic video deblurring, and blind video denoising while reducing the computational cost compared to existing best contextual methods on all these tasks.

Main paper: 
 introduction: 
 Video restoration aims to restore degraded low-quality videos. Degradation in videos occurs due to noise during the acquisition process, camera sensor faults, or external factors such as weather or motion blur [53, 38] . Several methods in the literature process the entire video either in parallel or with recurrence in design. In the former case, multiple contextual frames are processed simultaneously to facilitate information fusion and flow, which leads to increased memory consumption and inference cost as the context size increases [63, 4, 69, 86, 58, 28, 26, 5, 34, 62] . Methods with recurrence in design reuse the same network to process new frame sequentially based on previously refined ones [54, 14, 21, 25, 6, 7, 42, 57] . Such sequential processing approaches often result in cumulative errors, leading to information loss in long-range temporal dependency modeling [8] and limiting parallelization capabilities.Recently, methods based on state space models (SSMs) have seen applications across several machine vision tasks, including image restoration [19, 56] , and video understanding [30] . While Video-Mamba [30] proposes a state space model for video understanding, the learned state space does not reason at the pixel level and, hence, can suffer from information collapse in restoration tasks [77] .♣ indicates equal contribution.Additionally, the state evolves over time with respect to motion that affects the entire trajectory nonuniformly [51] at the pixel level. Therefore, it is pertinent to learn a model capable of summarizing the history 1 of the input as it operates on the spatiotemporal structure of the input video.In this work, we present "TURTLE", a new video restoration framework to learn the TRUncaTed causaL history modEl of a video. TURTLE employs the proposed Causal History Model (CHM) to align and borrow information from previously processed frames, maximizing feature utilization and efficiency by leveraging the frame history to enhance restoration quality. We outline our contributions.• TURTLE's encoder processes each frame individually, while its decoder, based on the proposed Causal History Model (CHM), reuses features from previously restored frames. This structure dynamically propagates features and compensates for lost or obscured information by conditioning the decoder on the frame history. CHM models the evolving state and compensates the history for motion relative to the input. Further, it learns to control the effect of history frames by scoring and aggregating motion-compensated features according to their relevance to the restoration of the current frame. • TURTLE facilitates training parallelism by sampling short clips from the entire video sequence. In inference, TURTLE's recurrent view implicitly maintains the entire trajectory ensuring effective frame restoration. • TURTLE sets new state-of-the-art results on several benchmark datasets and video restoration tasks, including video desnowing, nighttime video deraining, video raindrops and rain streak removal, video super-resolution, real and synthetic video deblurring, and achieves competitive results on the blind video denoising task. 
related work: 
 Video restoration is studied from several facets, mainly distributed in how the motion is estimated and compensated for in the learning procedure and how the frames are processed. Additional literature review is deferred to appendix G.Motion Compensation in Video Restoration. Motion estimation and compensation are crucial for correcting camera and object movements in video restoration. Several methods employ optical flow to explicitly estimate motion and devise a compensation strategy as part of the learning procedure, such as deformable convolutions [33, 34] , or flow refinement [23] . However, optical flow can struggle with degraded inputs [84, 3, 20] , often requiring several refinement stages to achieve precise flow estimation. On the other end, methods also rely on the implicit learning of correspondences in the latent space across the temporal resolution of the video; a few techniques include temporal shift modules [29] , non-local search [64, 32, 85] , or deformable convolutions [69, 13, 80] .Video Processing Methods. There is a similar distinction in how a video is processed, with several methods opting for either recurrence in design or restoring several frames simultaneously. Parallel methods, also known as sliding window methods, process multiple frames simultaneously. This sliding window approach can lead to inefficiencies in feature utilization and increased computational costs [63, 4, 69, 86, 58, 28, 26, 5, 34, 62, 9] . Although effective in learning joint features from the entire input context, their size and computational demands often render them unsuitable for resource-constrained devices. Conversely, recurrent methods restore frames sequentially, using multiple stages to propagate latent features [87, 81, 82] . These methods are prone to information loss [33] . Furthermore, while typical video restoration methods in the literature often rely on context from both past and future neighboring frames [34, 33, 29] , TURTLE is causal in design, focuses on using only past frames. This approach allows TURTLE to apply in scenarios like streaming and online video restoration, where future frames are unavailable.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss limitation of the proposed method in appendix C. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 Consider a low-quality video I LQ ∈ R T ×H×W ×C , where T , H, W , C denote the temporal resolution, height, width, and number of channels, respectively, that has been degraded with some degradationGiven a model M θ , letF [l]t denote the feature map of a frame at timestep t, taken from M θ at layer l. We, then, utilizeF [l]t to construct the causal history states denoted asH [l] t ∈ R τ ×h l ×w l ×c l, where τ is the truncation factor (or length of the history), h, w denote spatial resolution of the history, and c denotes the channels. More specifically, H[l] t = {F [l] t-τ ⊕ F [l] t-τ +1 ⊕ . . . ⊕ F [l] t-1 } ∈ R τ ×h l ×w l ×c l, where ⊕ is the concatenation operation. We denote the motion-compensated history at timestep t as Ĥ[l] t , which is compensated for motion with respect to the input frame features F[l]t . In this work, the state refers to the representation of a frame of the video. Further, history states (or causal history states) refers to a set of certain frame features previous to the input at some timestep. TURTLE's encoder learns a representation of each frame by downsampling the spatial resolution, while inflating the channel dimensions by a factor of 2. At each stage of the encoder, we opt for several stacked convolutional feedforward blocks, termed as Historyless FFN,2 . The learned representation at the last encoder stage onwards is fed to a running history queue Q of length γ. 3 We empirically set γ = 5 for all the tasks, and consider sequence of 5 frames. The entire video sequence is reshaped t . We propose a motion compensation module that operates on the feature space to implicitly align history states with respect to the input frame. Next, a dynamic router learns to control the effect of history frames by scoring and aggregating motion-compensated features based on their relevance to the restoration of the current frame. Such a procedure accentuates the aligned history such that the following stages of the decoder can learn to reconstruct the high-quality frame appropriately. Both of these procedures combine to form the Causal History Model CHM(F[l] t , H [l]t ), detailed in section 3.2. Further, multiple CHMs are stacked as black box layers at different stages to construct the decoder of TURTLE.CHM learns to align the history states with respect to the input feature map. Further, there could still exist potential degradation differences at the same feature locations along the entire sequence in the motion-compensated history states. To this end, CHM re-weights the sequence along the temporal dimension to accentuate significant features and suppress irrelevant ones. Let Ĥ[l] t ∈ R (τ +1)×h l ×w l ×c l denote the motion-compensated causal history states, and let input feature map be F[l] t ∈ R h l ×w l ×c l. Let the transformation on the history states to align the features be denoted by ϕ t , and let ψ t denote the re-weighting scheme. If the output is given by y[l] t ∈ R h l ×w l ×c l, we then, formalize the Causal History Model (CHM) as,EQUATIONEQUATIONIn eq. ( 1), B t denotes transformation on the input, and D denotes the skip connection, while ⊕ is the concatenation operation. In practice, we learn ϕ t , and the input transformation matrix B following the procedure described in State Align Block, while ψ t is detailed in Frame History Router. We present a visual illustration of Causal History Model (CHM) in fig. 2 . We also present a special case of (CHM) in appendix D, wherein we consider optimally compensated motion in videos.State Align Block (ϕ). State Align Block (ϕ) implicitly tracks and aligns the corresponding regions defined as groups of pixels (or patches) (p 1 × p 2 )-across each frame in the history. State Align Block computes attention scores through a dot product between any given patch from the current frame and all the patches from the history. Given the input feature map of a frame F[l]t ∈ R h l ×w l ×c l , we calculate the patched projections as, Q[l] Ft , K [l] Ft , V [l] Ft ∈ R h l p 1 × w l p 2 ×(cp1p2) l , i.e., Q [l] Ft , K [l] Ft , V [l] Ft ← F [l] t W F [l] t where W F [l]t is a learnable parameter matrix. For exposition, let the dimensions of projections be R n l h ×n l w ×d l , we subsequently rearrange the patches to R (n h nw) l ×d l . Here, n l h = h l p1 , and n l w = w l p2 denote the number of patches along the height and width dimension, and d l = (cp 1 p 2 ) l represents the dimension of each patch. Formally, we define the history statesH [l]t as a set of keys and values to facilitate the attention mechanism as,EQUATIONwhereK [l]Ht , andV [l]Ht are formally written as K[l]Ht = {K [l] Ft-τ , K [l]Ft-τ+1 , . . . , KFt-1 } ∈ R τ ×n h nw×d , andV [l] Ht = {V [l] Ft-τ , V [l] Ft-τ+1 , . . . , V [l]Ft-1 } ∈ R τ ×n h nw×d . We, then, compute the attention, and limit it to the top-k most similar patches in the key vector for each patch in the query vector, and, hence, focus solely on those that align closely. This prevents the inclusion of unrelated patches, which can, potentially, introduce irrelevant correlations, and obscure principal features. We, then, formalize the top-k selection procedure as,EQUATIONEQUATIONwhere α is a learnable parameter to scale the dot product, and A (:,:,i) denotes the i th patch along the second dimension. A * [l] t masks the non top-k scores, and replaces with -∞ to allow for softmax computation. In other words, each patch is compensated for with respect to its top-k similar, and salient patches across the trajectory. Such a procedure allows for soft alignment, and encourages each patch to borrow information from its most similar temporal neighbors, i.e., a one-to-top-k temporal correspondence is learned. Given the top-k scores, we compute the motion-compensated history states Ĥ[l] t as follows,EQUATIONwhere σ is the softmax operator, ⊕ is the concatenation operator, Wt is the parameter matrix learned with gradient descent, and B is a transformation on the input F[l] t realized through selfattention along the spatial dimensions [39, 66] . In eq. ( 6), ϕ t (H[l] t , F [l] t ) = σ(A * [l] t )V [l] Ht W Ĥ[l]t which follows from eq. ( 1).Frame History Router (ψ). Given the motion-compensated history states Ĥ[l] t ∈ R (τ +1)×h l ×w l ×c l and the input featuresF [l] t ∈ R h l ×w l ×c l, Frame History Router (ψ) learns to route and aggregate critical features for the restoration of the input frame. To this end, we compute the query vector fromF [l] t through the transformation matrix W Q [l] t , resulting in Q [l] t ← F [l] t W Q [l]t . Similarly, the key and value vectors are derived from Ĥ[l] t , and are parameterized WĤ[l] t , i.e., K [l] t , V [l] t ← Ĥ[l] t W Ĥ[l] t .This configuration enables cross-frame channel attention, where the query from F[l] t attends to channels from both Ĥ[l] t and F[l]t , and accentuates temporal history states as necessary in order to restore the given frame. The cross-channel attention map A ∈ R (τ +1)c l ×c l is then computed through the dot product, i.e., A[l] t = (Q [l] t • K [l] t )/α ∈ R (τ +1)c l ×c l, where α is the scale factor to control the dot product magnitude. Note that, we overload the notation A[l] t for exposition. The output feature map, y[l] t takes the shape R h l ×w l ×c l since the attention matrix takes the shape ∈ R (τ +1)c l ×c l , whileV [l] t is ∈ R (τ +1)c l ×h l ×w l. 4 If σ denotes the softmax operator, and D is the skip connection, we then We compare video desnowing results with the best published method in literature, SVDNet [10] . The video frame has both snow, and haze. While SVDNet [10] removes snow flakes, TURTLE can remove haze, and snow flakes, and hence is more faithful to the ground truth. In nighttime deraining, we compare TURTLE to MetaRain [47] . TURTLE maintains color consistency in the restored result.compute the output, y[l] t , as,EQUATIONIn eq. ( 7),ψ t ( Ĥ[l] t , F [l] t ) = σ(A [l] t )V [l] t W Ĥ[l]t which follows from eq. ( 2).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide a readme file, along with the code, that lists the steps to run the experiments. Further, the links to download the datasets used in the experiments reported in this manuscript are summarized in appendix H.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We follow the standard training setting of architectures in the restoration literature [29, 79, 15] with Adam optimizer [27] (β 1 = 0.9, β 2 = 0.999). The initial learning rate is set to 4e -4 , and is decayed to 1e -7 throughout training following the cosine annealing strategy [40] . All of our models are implemented in the PyTorch library, and are trained on 8 NVIDIA Tesla v100 PCIe 32 GB GPUs for 250k iterations. Each training video is sampled into clips of γ = 5 frames, and TURTLE restores frames of each clip with recurrence. The training videos are cropped to 192 × 192 sized patches at random locations, maintaining temporal consistency, while the evaluation is done on the full frames during inference. We assume no prior knowledge of the degradation process for all the tasks. Further, we apply basic data augmentation techniques, including horizontal-vertical flips and 90-degree rotations. Following the video restoration literature, we use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) [70] distortion metrics to report quantitative performance. For qualitative evaluation, we present visual outputs for each task and compare them with the results obtained from previous best methods in the literature.SynNightRain [47] is a synthetic video deraining dataset focusing on nighttime videos wherein rain streaks get mixed in with significant noise in low-light regions. Therefore, nighttime deraining with heavy rain is generally a harder restoration task than other daytime video deraining. We follow the train/test protocol outlined in [47, 35] , and train TURTLE on 10 videos from scratch, and evaluate on a held-out test set of 20 videos. We report distortion metrics, PSNR and SSIM, in table 1, and compare them with previous restoration methods. TURTLE achieves a PSNR of 29.26 dB, which is a notable improvement of +2.53 dB over the next best result, NightRain [35] . Further, we present visual results in fig. 3 , and in fig. 12. Our method, TURTLE, maintains color consistency in the restored results.Realistic Video Desnowing Dataset (RVSD) [10] is a video-first desnowing dataset simulating realistic physical characteristics of snow and haze. The dataset comprises a variety of scenes, and the videos are captured from various angles to capture realistic scenes with different intensities. In total, the dataset includes 110 videos, of which 80 are used for training, while 30 are held-out test set to measure desnowing performance. We follow the proposed train/test split in the original work [10] and train TURTLE on the video desnowing dataset. Our scores, 26.02 dB in PSNR, are reported in table 2 , and compared to previous methods, TURTLE significantly improves the performance by +0.96 dB in PSNR. Notably, TURTLE is prior-free, unlike the previous best result SVDNet [10] , which exploits snow-type priors. We present visual results in fig. 3 , and in fig. 11 comparing TURTLE to SVDNet [10] . Our method not only removes snowflakes but also removes haze, and the restored frame is visually pleasing.The work done in [83, 82] introduced a real-world deblurring dataset (BSD) using the Beam-Splitter apparatus. The dataset introduced contains three different variants depending on the blur intensity settings. Each of the three variants has a total of 11, 000 blurry/sharp pairs with a resolution of 640 × 480. We employ the variant of BSD with the most blur exposure time, i.e., 3ms-24ms. 5 We follow the standard train/test split introduced in [83] with 60 training videos, and 20 test videos. We report the scores in table 3 on the 3ms-24ms variant of BSD and compare with previously published methods. TURTLE scores 33.58 dB in PSNR on the task, observing an increase of +2.0 dB compared to the previous best methods, CDVD-TSP [44] , and ESTRNN [83, 82] . We present visual results in fig. 13 . Qualitative results on video deblurring on the GoPro dataset [41] are in the top row. Our method, TURTLE, restores the frames without any artifacts (see the number plate) unlike DSTNet [45] . On video raindrops and rain streaks removal task, we compare our method with the best method in literature ViMPNet [71] . Notice how the frame restored by ViMPNet [71] has artifacts (see tree region, and the railing gate), while TURTLE's output is free of unwanted artifacts. GoPro dataset [41] is an established video deblurring benchmark dataset in the literature. The dataset is prepared with videos taken from a GOPRO4 Hero consumer camera, and the videos are captured at 240fps. Blurs of varying strength are introduced in the dataset by averaging several successive frames; hence, the dataset is a synthetic blur dataset. We follow the standard train/test split of the dataset [41] , and train our proposed method. TURTLE scores 34.50 dB in PSNR on the task, with an increase of +0.34 dB compared to the previous best method in a comparable computational budget, DSTNet [45] (see table 4 ). We also present visual results on the GoPro dataset [41] comparing TURTLE to DSTNet [45] in fig. 4 , and fig. 9. Our method restores frames free of artifacts (see the number plate on the car) in fig. 4 .The work done in [71] introduced a synthesized video dataset of 102 videos, VRDS, wherein the videos contain both raindrops and rain streaks degradations since both rain streaks and raindrops corrupt the videos captured in rainy weather. 6 We split the dataset in train and held-out test sets as outlined in the original work [71] . We present TURTLE's scores in table 5 , and compare it with several methods in the literature. TURTLE scores 32.01 dB in PSNR on the task, with an increase of +0.99 dB compared to the previous best method, ViMPNet [71] . We present visual results on the task in fig. 4 , and fig. 10 , and compare our method with ViMPNet [71] . TURTLE restores the frames that are pleasing to the human eye and are faithful to the ground truth. [71] for video super resolution. In video denoising, TURTLE restores details, while BSVD-64 [49] smudges textures (text and the dinosaur on the biker's jacket). In VSR, previous methods such as TTVSR [37] , BasicVSR++ [7] , or EAVSR [71] tend to introduce blur in results, while TURTLE's restored results are sharper, and crisper. MVSR4× is a real-world paired video super-resolution dataset [67] collected by mobile phone's dual cameras. We train TURTLE following the dataset split in the official work [67] and test on the provided held-out test set. We report distortion metrics in table 7 and compare it with several methods in the literature. TURTLE scores 25.30 dB in PSNR on the task, with a significant increase of +1.36 dB compared to the previous best method, EAVSR+ [71] . We present visual results on the task in fig. 5 . Other methods such as TTVSR [37] , BasicVSR [7] , or EAVSR [71] tend to introduce blur in up-scaled results, while TURTLE's restored results are sharper.We assume no degradation prior, and consider blind video denoising task [49, 55] . We train our model on DAVIS [48] dataset, and test on DAVIS held-out testset, and a generalization set Set8 [61] . We add white Gaussian noise to the dataset with noise level σ ∈ U [30, 50] to train TURTLE, and test on two noise levels σ = 30, and σ = 50; scores are reported in table 6 . TURTLE observes a gain of +0.31 dB on σ = 30, and +0.34 dB on σ = 50 on Set8 testset, scoring 32.22 dB, and 30.29 dB, respectively, while it observes an average drop of -0.3 dB to BSVD-64 [49] on the DAVIS testset. Further, we present qualitative results in fig. 5 comparing TURTLE, and previous best method BSVD [49] .In table 8, we compare TURTLE with previous methods in the literature in terms of multiplyaccumulate operations (MACs). The results are computed for the input size 256 × 256. We measure the performance on the number of frames the original works utilized 7 to report their performance, as reported in their manuscript or code bases. In TURTLE's case, we report MACs (G) on a single frame since TURTLE only considers a single frame at a time but adjust for history features utilized in CHM as part of TURTLE's decoder. In comparison to parallel methods, EDVR [69] , VRT [34] , TURTLE is computationally efficient, as it is lower in MACs (G). Although the MACs are approximately similar to recurrent methods, BasicVSR [6] , TURTLE scores significantly higher in PSNR/SSIM metrics (see table 7, and table 5 ). In comparison to contemporary methods such as RVRT [33] , which combines recurrence and parallelism in design, TURTLE is significantly lower on MACs (G) and performs better (see table 5, and table 2) thanks to its ability to memorize previous frames. 
ablation study: 
 We ablate TURTLE to understand what components necessitate efficiency and performance gains. All experiments are conducted on synthetic video deblurring task, GoPro dataset [41] , using a smaller variant of our model. Our smaller models operate within a computational budget of approximately 5 MACs (G), while the remaining settings are the same as those of the main model. In all the cases, the combinations we adopt for TURTLE are highlighted . Additional ablation studies are deferred to appendix A, and we discuss the limitations of the proposed method in appendix C.Block Configuration. We ablate the Causal History Model (CHM) to understand if learning from history benefits the restoration performance. We compare TURTLE with two settings: baseline (no CHM block) and TURTLE without State Align Block (ϕ). In baseline (no CHM), no history states are considered, and two frames are concatenated and fed to the network directly. Further, in No ϕ, the state align block is removed from CHM. We detail the results in table 9, and find that both State Align Block, and CHM are important to the observed performance gains.Truncation Factor τ . We evaluate context lengths of τ = 1, 3, and 5 past frames and found no PSNR improvement when increasing the context length beyond three frames. Results in table 10 confirm that extending beyond three frames does not benefit performance. This is because, as in most cases, the missing information in the current frame is typically covered within the three-frame span, and additional explicit frame information fails to provide additional relevant details.Value of k in topk. We investigate the effects of different k values in topk attention. Our experiments, detailed in table 11, show that k crucially affects restoration quality. Utilizing a larger number of patches, k = 20, leads to an accumulation of irrelevant information, negatively impacting performance by adding unnecessary noise. Further, selecting only 1 patch is also sub-optimal as the degraded nature of inputs can lead to inaccuracies in identifying the most similar patch, missing vital contextual information. The optimal balance was found empirically with k = 5, which effectively minimizes noise while ensuring the inclusion of key information. 
conclusion: 
 In this work, we introduced a novel framework, TURTLE, for video restoration. TURTLE learns to restore any given frame by conditioning the restoration procedure on the frame history. Further, it compensates the history for motion with respect to the input and accentuates key information to benefit from temporal redundancies in the sequence. TURTLE enjoys training parallelism and maintains the entire frame history implicitly during inference. We evaluated the effectiveness of the proposed method and reported state-of-the-art results on seven video restoration tasks. In appendices, we discuss additional details about the proposed method TURTLE, and provide additional ablation studies in appendix A, motivate the need for learning to model the history of input for video restoration in appendix B, discuss limitations of the proposed approach appendix C, discuss theoretical relationship to state-space models in appendix D, present more visual results in appendix E, discuss related work in appendix G, and computationally profile the proposed method TURTLE in appendix F. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We detail the hyperparameters, train/test splits of the data used in section 4, and also ablate the necessary hyperparameters in section 5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We fix the seed in all experiments for reproducibility following the standard in restoration literature.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: To the best of our knowledge, we believe that the work done in this manuscript poses no such risk. However, we highlight societal impact in appendix C.1.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In all of the experiments reported in this manuscript, we use publicly available datasets released by their respective authors for research purposes. Although we cite the original work wherever the dataset is discussed, we also summarize the details in appendix H.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: This work does not create new datasets. However, the code released is accompanied by a README that details the steps necessary to re-produce the results.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We ablate two more aspects of the proposed method TURTLE. Mainly, we empirically verify the rationale behind placing CHM in both latent, and decoder stages. Further, we ablate if selecting topk regions compared to plain softmax is beneficial for restoration. In the top row, we plot the correctly tracked points, while the bottom row visualizes the limitations in the case of redundant patterns. We plot the query and most similar points on input frames for ease of exposition, but in practice, they function on feature maps. CHM Placement. Our experiment, in table 12, indicates that having CHM in both the latent and decoder stages is necessary for optimal performance. In the latent stage, the spatial resolution is minimal, and CHM provides greater benefit in the following decoder stages as the spatial resolution increases.Softmax Ablation. In table 13 , we verify that topk selection is necessary to allow the restoration procedure to only consider relevant information from history. Since softmax does not bound the information flow we observe non-trivial performance drop when topk is replaced with softmax. We Transformer block is similar in spirit to the block introduced in [79] , while the Historyless FFN's design takes inspiration from the blocks in [15, 11] .argue that topk prevents the inclusion of unrelated patches, which can, potentially, introduce irrelevant correlations, and obscure principal features.We motivate TURTLE's design and present empirical results to pinpoint the benefits of modeling the history in the case of video restoration. Further, we present additional details of the proposed method, TURTLE, and expand on the construction of the architecture.Recall that the Causal History Model (CHM) is designed to model the state and compensate for motion across the entire history relative to the input. It then learns to control the effect of history frames by scoring and aggregating motion-compensated features based on their relevance to the restoration of the current frame. Such a procedure allows for borrowing information from the preceding temporal neighbors of the input frame. In table 9, we ablate if CHM indeed provides the performance boost. Moreover, in fig. 6 , we present visual results on the video raindrops and rain streaks removal task to motivate the need for summarizing the frame history as part of the restoration method. We train TURTLE without the CHM block, referred to as "no CHM", following TURTLE's experimental setup, and for fair comparison, we keep the model size consistent. TURTLE, equipped with CHM, maintains the spatial integrity of the input without introducing faux textures or blur even though the region (see guard railing in the balcony of the building) is entirely obscured by raindrops and streaks in the degraded input. However, without CHM block, unwanted artifacts (such as holes and blemishes in place of the guard railing in the balcony) are introduced to fill in the missing information since no information is borrowed from preceding frames. Note that in the case of no CHM experiment, we feed two concatenated frames (one frame previous to the input and the input frame) to the architecture.Recall that TURTLE's encoder is historyless, i.e., it employs no information about the history of the input. Further, we opt for a feedforward style design in the encoder with convolutional layers. This is because shallow representations at this stage are not sufficiently compressed and are riddled with degradation. Thus, expensive attention-based operators provide no significant performance benefit but add to the computational complexity. The diagrammatic illustration of Historyless FFN and the Transformer block [79] used in CHM is presented in fig. 8 .This section discusses the limitations of our proposed CHM. While CHM is adept at tracking similar patches across the history of frames, it encounters challenges in certain scenarios. For instance, as demonstrated in the zebra example in fig. 7 , CHM identifies similar patches on different parts of the zebra's body due to their redundant patterns, even though these patches are not located in the same area. Moreover, in case of severe input degradation, CHM's capacity to accurately identify and utilize similar patches may diminish due to spurious correlations, which could affect its ability to use history effectively for restoring the current frame. We present additional qualitative analysis on synthetic video deblurring on the GoPro dataset [41] . We compare TURTLE with DSTNet [45] on two frames taken from two different videos of the testset.This work presents an efficient method to advance the study of machine learning research for video restoration. While the proposed method effectively restores the degraded videos, we recommend expert supervision in medical, forensic, or other similar sensitive applications.We present a special case of the proposed Causal History Model, CHM, wherein the videos are not degraded, and each frame is optimally compensated for motion with respect to the input. Lemma D.1. (Special Case of Causal History Model) In the absence of degradation and optimally compensated motion through optical flow, the state history Ĥ[l] t , then, only depends on the inputF [l]t , and the previous state Ĥ[l] t-1 . Under this assumption, eq. (1), and eq. ( 2) can be rewritten as,EQUATIONEQUATIONwhere A t , B t , C t are parameters learned through gradient descent, andF [l]t is the input feature map at timestep t. In this case, eq. ( 8) is realizable and not flawed, given the motion-compensated input. This assumption allows the model to be learned in a similar fashion to HiPPO [18] or Mamba [17] . More specifically, the Causal History Model (CHM) reduces to an equivalent time-variant and input-dependent flavor of the State Space Model (SSM).Proof 1. Consider a state space model (SSM) [16] that maps the input signal F t to the output signal y t through an implicit state H t , i.e.,EQUATION)y t = C(H t ).(11) In the above equations, we abuse the SSM notation for exposition. Recall that we consider the special case wherein the motion is compensated for, and there is no degradation in the input video. In this case, we can say that history and motion-compensated history are equal i.e., Ĥt = H t . Now, consider the first frame of the video at timestep t = 0. Let the initial condition be denoted by F 0 , then we can write the eq. ( 10), and eq. ( 11) as,EQUATION)EQUATIONwhere A 0 , B 0 , C 0 are learnable parameters. Then for the next timestep t = 1, we can write thatEQUATIONEQUATION)From eq. ( 12), we know that Ĥ0 = B 0 (F 0 ), then we can re-write eq. ( 14) asĤ1 = A 1 (B 0 (F 0 )) + B 1 (F 1 ) (16) ⇒ A 1 B 0 (F 0 ) + B 1 (F 1 ).The output y 1 can then be written as,y 1 = C 1 ( Ĥ1 ).(17) At every timestep t, the output can be written in terms of the history (previous timestep) and the input (at current timestep). Now, consider the case t = 2, and let the frame features be denoted by F 2 , then we can write,EQUATIONEQUATION)Notice, how in eq. ( 19) Ĥ2 is written in terms of the input frame F 2 , and the previous frames F 0 , and F 1 . The output y 2 is then computed as,EQUATION)We can then generalize eq. ( 18), and eq. ( 20) to any timestep t, and we arrive at eq. ( 8), and eq. ( 9), i.e.,EQUATIONEQUATIONTherefore, the model can be learned in Mamba [17] , or HiPPO [18] style, and the parameters A, B, C can be learned with gradient descent since the linear relationship between the input and the output is tractable in this case.In practice, however, the no degradation assumption does not hold. Therefore, a naive state update, eq. ( 8), renders sub-optimal results. This is because, in video processing tasks, motion governs the transition dynamics, i.e., the state evolves over time due to motion, and therefore, any linear relationship between the output and the input is intractable unless the motion is compensated for.We present additional visual results, comparing our method with previously available methods in the video restoration literature.We present further results on synthetic video deblurring task on the GoPro dataset [41] in fig. 9 . We compare TURTLE with the previous method in the literature that is computationally similar to TURTLE, DSTNet [45] . Turtle avoids unnecessary artifacts in the restored results (see the tire undergoing rotation in the frame in the top row in fig. 9 ). Further, the restored results are not smudged, and textures are restored faithfully to the ground truth (see feet of the person in the frame in the bottom row).Unlike just rain streaks, raindrops often pose a more complex challenge for restoration algorithms. This is because several video/image restoration methods often induce blurriness in the results. Raindrops get mixed in with the background textures; therefore, minute details such as numbers or text are blurred in restored results. However, since TURTLE utilizes the neighboring information, it learns to restore these details better. We observe this in fig. 10 , where the previous best method ViMPNet [71] blurs the number plate on the car (in last row), or introduces faux texture on the building (in second row). On the contrary, TURTLE better restores the results and avoids undesired artifacts or blur. We further the discussion on prior art in the literature in terms of temporal modeling of videos, and causal learning.Temporal Modeling. In video restoration, temporal modeling mainly focuses on how the neighboring frames (either in history or in the future) can be utilized to restore the current frame better. For such a procedure, the first step usually involves compensating for motion either through explicit methods (such as using optical flow [33, 34, 36, 7] ), or implicitly (such as deformable convolutions [63] , search approaches [64] , or temporal shift [29] ). A few works in the literature focus on reasoning at the trajectory level (i.e., considering the entire frame sequence of a video) [37] through learning to form trajectories of each pixel (or some group of pixels). The motivation is that in this case, each pixel can borrow information from the entire trajectory instead of focusing on a limited context. The second step is then aggregating such information, where in the case of Transformers, attention is employed, while MLPs are also used in other cases.Causal Learning in Videos. In videos, causal learning is generally explored in the context of selfsupervised learning to learn representations from long-context videos with downstream applications to various video tasks such as action recognition, activity understanding, etc [50, 73] . In [2] , causal masking of several frames at various spatio-temporal regions as a strategy to learn the representations is explored. To the best of our knowledge, other than one streaming video denoising method [49] , almost all of the state-of-the-art video restoration methods are not causal by design since they rely on forward and backward feature propagation (i.e., they consider both frames in history and in the future) either aligned with the optical flow or otherwise [6, 7, 33, 29] . However, there is significant Table 14 : Profiling TURTLE. We profile the proposed method, TURTLE, on a single 32 GB V100 GPU, and compare with 3 recent video restoration methods, namely ShiftNet [29] , VRT [34] , and RVRT [33] . We consider different input resolutions and compute the per-frame inference time (ms), total MACs (G), FLOPs (G), and the GPU memory usage of the model. OOM denotes Out-Of-Memory error i.e., the memory requirement exceeded the total available memory of 32GB. amount of work on causal representation learning where the aim is to recover the process generating the data from the observation to learn the disentangled latent representation. Note that this is out of the scope of this work.All of the experiments presented in this manuscript employ publicly available datasets that are disseminated for the purpose of scientific research on video/image restoration. All of the datasets employed are cited wherever they are referred to in the manuscript, and we summarize the details here.• Video Desnowing: We utilize the video desnowing dataset introduced in [10] . The dataset is made available by the authors at the link: Video Desnowing • Video Nighttime Deraining: We utilize the nighttime video deraining dataset introduced in [47] . The dataset is made available by the authors at the link: Nighttime Deraining • Video Raindrops and Rain Streaks Removal: We utilize the video raindrops and rain streaks removal (VRDS) dataset introduced in [71] . The dataset is made available by the authors at the link: VRDS • Synthetic Video Deblurring: We employ the GOPRO dataset introduced in [41] . The dataset is made available by the authors at the link: GOPRO • Real Video Deblurring: We employ the BSD dataset introduced in [82, 83] . The dataset is made available by the authors at the link: BSD • Real-World Video Super Resolution: We utilize the MVSR dataset introduced in [67] . The dataset is made available by the authors at the link: MVSR • Video Denoising: We employ DAVIS [48] , and Set8 [61] Justification: The introduction (section 1) lists the contributions of this work, and we back the claims up empirically in section 4, followed by ablations in section 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.In this manuscript, the term "history" refers to temporally previous frames with respect to the input frame.We use the term "Historyless" because the encoder is not conditioned on the history of input.The space complexity is limited toO(γ × h × w × c) and (γ × h × w × c) << (T × H × W × C) because γ << T ,and h << H, w << W , and all enqueue and dequeue operations are O(1) in time complexity.This is because in practice, channels are collapsed in the temporal dimension.24ms is the blur exposure time, and 3ms is the sharp exposure time.The dataset comprises videos captured in diverse scenarios in both daytime and nighttime settings.or 2 frames if the actual number of frames did not fit into a single 32GB GPU 


Decision: The reviewers were unanimous in accepting this paper, and rated it with an Accept, a Weak Accept, and 2 Borderline Accept.
They found the paper novel, its performance and evaluation convincing and the design computationally efficient. They raised 
some concerns but many of these were addressed in the rebuttal and in the direct messages to the reviewers, and reviewers 
have correspondingly adjusted their ratings.
The authors are encouraged to integrate the answers to these concerns in their paper, especially regarding the paper presentation.
The AC has read all the reviews, the rebuttal and the messages exchanged between the reviewers and the authors
and agrees with the reviewers that this paper should be accepted.

Reviews: review: paper_summary: paper_summary: The paper presents a novel framework TURTLE for video restoration. TURTLE aims to improve video restoration tasks by modeling and utilizing truncated historical data of input video frames to enhance the restoration quality while maintaining computational efficiency. The proposed method demonstrates state-of-the-art results across various video restoration benchmarks, including desnowing, deraining, deblurring, and super-resolution.
strength_weakness: strength_weakness: - This paper did not discuss the parameters and FLOPs of this model.
- Lack of comparison with  Restormer: Efficient transformer for high-resolution image restoration. (CVPR2022), A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift.  (CVPR2023)
- Lack of visualization results of the effectiveness of truncation factor and value of topk.
questions: questions: -  Line 24, the author claims recurrent designs often result in cumulative errors. I think it would be better to explain why a truncated causal history model can avoid this phenomenon.
- As shown in Fig. 7, how many frames are affected by incorrect tracking query points? Will this reduce the restoration quality of subsequent frames?
- I noticed a latent cache block in your code that is barely mentioned in the paper. Could you explain its function and impact on the model's performance?
- Please refer to the weaknesses above.
limitations: limitations: The authors have largely addressed their limitations.
paper_summary: paper_summary: This work presents a video restoration framework named TURTLE, which stands for truncated casual history model. The key innovation of TURTLE is its ability to efficiently model the transition dynamics of video frames governed by motion, a critical challenge in video restoration. Unlike traditional methods that process multiple contextual frames simultaneously, TURTLE enhances efficiency by storing and summarizing a truncated history of the input frame's latent representation into an evolving historical state.
strength_weakness: strength_weakness: 1. The section on related work is seriously lacking in content; the author should revise this section to include a more detailed explanation of related work, especially in the areas of temporal modeling and Causal Learning.

2. There is a scarcity of Visual Comparisons, and many tasks lack real-world sample comparisons. This is a serious issue, and I hope the author can provide a complete set of comparisons.

3. Why is it necessary to use a Casual History Model in each decoder? I believe applying CHM in the latent space should be sufficient. This raises concerns about the actual inference efficiency of the model.

4. There is a lack of comparison regarding actual inference time, as well as a crucial study on the actual VRAM usage at high resolutions, both of which are very important; the reference value of GMacs alone is limited.
questions: questions: Why is it necessary to use a Casual History Model in each decoder? I believe applying CHM in the latent space should be sufficient. This raises concerns about the actual inference efficiency of the model.
There is a lack of comparison regarding actual inference time, as well as a crucial study on the actual VRAM usage at high resolutions, both of which are very important; the reference value of GMacs alone is limited.
limitations: limitations: Please refer to the weaknesses.
paper_summary: paper_summary: This paper proposes a truncated causal history model (TURTLE) for video restoration. TURTLE is in a U-Net manner with a historyless encoder and a history-based decoder. The decoder has a causal history model (CHM), which is the core part of the TURTLE. The CHM injects history frames into a hidden state $\mathbf{H}_t$ with the state align block, which is based on a cross-attention mechanism and a top-k selection strategy. The current frame is then fused with the hidden state with cross-frame attention. Experiments on several tasks demonstrate that the proposed methods can outperform the comparable methods.
strength_weakness: strength_weakness: 1. The organization of ablation studies is inappropriate.
2. Some of the datasets are uncommon.
3. Some key information is lacking.
questions: questions: 1. The organization of ablation studies is inappropriate.  
1.1 It's improper to put the tables of ablation studies into the appendix.  
The ablations of some parts are missing.  
1.2 The ablation study of top-k fusion in the state align block is missing. Most of the time, the information fusion operation in the top-k selection position of CHM is implemented by a softmax. How about using a softmax to replace top-k, i.e., remove the top-k selection strategy?  
1.3 The implementation of the ''No CHM'' in the ablations is unclear. It states ''two frames are concatenated and fed to the network directly'', which is unclear. What network are the two frames fed to?  
1.4 What is the patch size $p$ in the state align block? How about using different $p$?

2. There are important video restoration methods absent in the comparable tables.  
[1] K. Zhou, et al., Revisiting Temporal Alignment for Video Restoration. CVPR 2022.  
(Link of codes: https://github.com/redrock303/Revisiting-Temporal-Alignment-for-Video-Restoration)  
[2] D. Li, et al., A Simple Baseline for Video Restoration with Grouped Spatial-Temporal Shift. CVPR 2023.  
(Link of codes: https://github.com/dasongli1/Shift-Net)

3. Recurrent structures in neural networks can bring considerable costs. How does the CHM? In other words, the comparisons of running time and GPU memory cost are absent.

4. Some of the datasets are uncommon. For example, in the video super-resolution experiments, this paper selects MVSR4X as the dataset. However, the recent comparable methods (such as BasicVSR and VRT)  use REDS and Vimeo-90k as benchmarks. It seems that in the Table. 7, the TURTLE outperforms BasicVSR++ a lot, but is the BasicVSR++ model used in this paper trained for being evaluated on MVSR4X?

5. Do all the models for each task in the paper use the same number of input frames? If not, how to explain the rationality of the experiments?

6. What are the training datasets for each task? Are all the methods in the same comparable table training in the same dataset (or under the same settings)?

7. It looks like the hidden state $\mathbf{H}$ in the CHM needs some frames to start. How to handle the boundary frames of videos？

8. This paper emphasizes the proposed casual history model, but in the method part, it looks like the core part is a module for alignment (state align block). For extracting features from hidden states for restoration, what is design motivation? It would be better to introduce the components in CHM except the state align block with more details.
limitations: limitations: Please see the Questions part.
paper_summary: paper_summary: This paper proposes a TURTLE to learn the truncated causal history model for video restoration tasks.
The proposed turtle's causal history model consists of two sub-modules: a State Align Block and a Frame History Router.
The state align block has a similarity-based retrieval mechanism that implicitly accounts for inter-frame motion and alignment from previous frames, and those retrieved representations are summarized and stored into a truncated history.
Then the frame history router generates output frames by cross-frame channel attention with the motion-compensated history states.
In experiments, the proposed method shows a promising result in a number of video restoration tasks including deraining, desnowing, deblurring, and super-resolution.
strength_weakness: strength_weakness: - As in the state align block, find similarity from history frames have previously proposed [1] and it seems the frame history router (channel-wise attention) is a new one proposed in this paper. However, there is no relevant ablation study with and without frame history router, so it is hard to see the improvement caused by it.
[1] Learning Trajectory-Aware Transformer for Video Super-Resolution (CVPR22)

- There is no detailed model architectures and model parameter (and runtime) comparisons with others.
questions: questions: - Does the size of p1 and p2 affect the results?
limitations: limitations: It is surprising that the proposed method shows excellent performance in all various experiments.
But one thing may be complemented is that there is no strong guidance for temporal consistency, and I found there is a background flickering in the deraining scene in attached video.

Comments: title: Addressing Further Questions
Inquiring about additional questions/concerns.
Follow up and Inquiry for Further Questions/Concerns.
Please let us know if you have further questions/concerns
comment: Dear Reviewer BXMC,

Thank you for your valuable input, insight, and feedback that helped improve our work. We appreciate your positive stance of our work, and your decision of raising the score.
Dear authors,
Thank you for your clarification, I've raised my rating to be positive.
Dear Reviewer BXMC,

Thank you for your detailed feedback and questions. We appreciate the opportunity to address your queries and provide further clarification.

**1. Comparison to ShiftNet.**

We had indeed trained ShiftNet with a context of 3 frames to match the input settings of the proposed method, Turtle, on the synthetic video deblurring (GoPro). However, we did not include these results initially in the manuscript because the original ShiftNet was designed for a context length of 50 frames (or even more according to their paper and repo), relying on high-end and multiple GPUs, which is a setting not straightforwardly comparable to our method and other baselines evaluated in this work. However, per the reviewer's request, we now provide these results in comparison to ShiftNet (retrained with 3-frame context on GoPro) in the following table.

| **Method** | **PSNR** | **SSIM** | **MACs (G)** | **Inf. Time (ms)** |
|-|-|-|-|-|
|ShiftNet|33.20| 0.962|399|165|
|Turtle|**34.50**|**0.972**|**181**|**95**|

_**Table 1:**_ Comparison to ShiftNet on the video Deblurring task (GoPro) in terms of PSNR/SSIM, and profiling on input resolution of 256x256x3 on a single 32GB GPU.

We can see that ShiftNet-3frame is almost 1db lower than Turtle and is not even competitive as compared to other baselines in Table 4 runnable on a single v100 GPU. The much lower performance of ShiftNet with a 3-frame context is because its original design needs to perform restoration jointly for a larger number (50 or more) of frames together to incorporate motion-compensated neighboring information. 

Additionally, ShiftNet's high computational demands are notable in the community. Although their paper does not specify the compute requirements, their GitHub repository indicates that training a model with a batch size of 1 and 13 frames required 8x 32GB V100 GPUs. The full 50-frame context model's compute and memory requirements at inference remain unspecified, and there have been multiple comments on the computational challenges faced when running their open-source implementation:

[1] https://github.com/dasongli1/Shift-Net/issues/1 {/7,/9}

Another nuance to be considered is that ShiftNet by design assumes the availability of the future frames and restores all frames in the context jointly, while the proposed method Turtle has the strength that it does not rely on future frames and can be applied in real-time scenarios (e.g., video streaming). 

**2. Cost Comparisons**

In Table 1 of the global rebuttal, we chose to compare the compute costs, inf. time, and memory requirements of Turtle to VRT [33] (2022, published in 2024), ShiftNet [28] (2023), and RVRT [32] (2022), because they are all general video restoration techniques, which is the same as Turtle, rather than designed for accomplishing a specific task. Methods that are designed for specific restoration tasks are optimized for those tasks and, therefore, are not intended to perform competitively across a broad range of restoration tasks as Turtle does. 

We can see from Table 1 in global rebuttal that while methods like ShiftNet, VRT, and RVRT exhibit exponential growth in GPU memory requirements as resolutions increase, Turtle features linear scaling in GPU memory usage for higher resolutions, underscoring its computational efficiency advantage.

However, in Table 8 of the main paper, we have also provided MACs comparisons to more different video restoration methods other than the ones listed in global rebuttal, including some task-specific ones. 

The methods for cost comparison in Table 8 are selected as follows. First, we included two general methods, RVRT and VRT, due to their high citation counts and popularity, recent publication, and competitive performance. Second, for the task-specific methods, our selection was based on their respective performances (focusing on those that ranked second or third best in respective tables), while also depending on the availability of open-source implementations (e.g., for SVDNet [9] and MetaRain [46], code is not available, which are also worse than Turtle on task performance). Furthermore, if the performance gap between a method and Turtle was larger than 1dB PSNR (which is a substantially large gap in log scale), we would not consider those methods for cost comparison, since their task performance is not even close. 

Table 8 and the extensive task results presented in Tables 1-7 suggest that Turtle either substantially outperforms the baselines or achieves the leading results on all the major video restoration tasks evaluated while still being more computationally efficient. For example, EDVR and BasicVSR, RDDNet have relatively lower GMACs in Table 8, although still higher than Turtle, but they achieve much lower performance than Turtle in Table 7 and Table 5. These results have verified the effectiveness and efficiency of Turtle as compared to baselines. 

Thank you again, and we hope these clarifications address your concerns.
Dear Reviewer KLsv,

We thank you for your decision to increase the score and your effort in reviewing our paper. We appreciate your acknowledgement that our rebuttal has addressed your concerns. Your insights have helped improve our paper.
Dear Reviewer PswG,

We are glad that our clarifications have addressed your concerns. Your input has been invaluable throughout the review process. We thank you for your effort in reviewing our paper and for finding it important.
Thank you for your rebuttal. Most of my concerns have been addressed. I've raised my rating.

Best regards.
Dear authors,
Thank you for your reply. Some of my questions are not addressed well.
1. Comparisons: Shiftnet is not compared. Since the re-training is in the same settings as the proposed method, why not modify the number of input frames?
2. Cost: There are many methods in this paper, only the costs of ShiftNet, VRT, RVRT, and Turtle are shown.
Dear Reviewer BXMC,

Thank you for your reply. We appreciate your comments and time spent on reviewing this paper again. In the rebuttal, we have addressed your main questions/concerns raised in your review, including:

1. Ablation Studies: We provided the ablation study for the proposed top-k fusion in the state align block vs. softmax as suggested, and demonstrated the effectiveness of top-k.

2. Comparisons to more baselines: We added comparisons to ShiftNet and RTA as suggested, highlighting Turtle’s advantages on performance and efficiency.

3. Efficiency/cost evaluation: We provided detailed comparisons of running time and GPU memory cost, showing Turtle’s efficiency across varying spatial resolutions.

4. Dataset for SR: We explained our use of the MVSR4X dataset as a recent and challenging benchmark for super-resolution task, as an additional task we evaluate on, and clarified our fair comparison to BasicVSR++ trained on the same dataset.

5. Extensive and fair comparisons with baselines: We ensured fairness by using the same settings, training/testing data splits, and careful tuning across all the methods evaluated. We have done extensive experimental studies and comparisons on a range of tasks and datasets.

6. CHM Design Rationale: We provided a detailed explanation of the design rationale behind the Casual History Model (CHM), and clarified all its critical components.

We hope the above rebuttal has addressed and cleared all your major questions or doubts raised in the review. And we hope these responses can be thoroughly taken into account to reach into conclusions. Please let us know if you have remaining concerns/questions for this work.  Thanks again for your time!
Thanks for your detailed rebuttal and my concerns have been well addressed.
Thanks for your rebuttal. I raised my rating to borderline reject.
Dear Reviewer BXMC,

Thank you very much for your time spent on reviewing our work. In response to your questions, we had provided detailed explanations to clarify the points discussed and addressed the concerns you highlighted. As the deadline for the discussion period is quickly approaching, we are keen to know if our responses have addressed all your concerns. We are committed to answering any further questions that you may have.

Thank you again for your valuable time and expertise.
Dear Reviewer SsXx,

Thank you very much for your time, and insightful comments. We had provided detailed explanations to clarify the points, and questions raised. As the deadline for the discussion period is quickly approaching, we were just wondering if all of your questions have been addressed. We are committed to answering any questions/concerns that you may have.

Thank you again for your valuable time.
Dear Reviewer KLsv,

Thank you very much for your time spent on reviewing our work. 
We had provided detailed explanations to clarify the points raised. As the deadline for the discussion period is quickly approaching, we are just wondering if all your questions have been addressed. We are committed to address any questions/concerns that you may have. 

Thank you again for your valuable time.