AverNet: All-in-one Video Restoration for Time-varying Unknown Degradations
Authors: Haiyu Zhao
Lei Tian
Xinyan Xiao
Peng Hu
Yuanbiao Gou
Xi Peng
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep learning
All-in-one video restoration
Time-varying unknown degradations
Primary area: machine_vision

Abstract: Traditional video restoration approaches were designed to recover clean videos from a specific type of degradation, making them ineffective in handling multiple unknown types of degradation. To address this issue, several studies have been conducted and have shown promising results. However, these studies overlook that the degradations in video usually change over time, dubbed time-varying unknown degradations (TUD). To tackle such a less-touched challenge, we propose an innovative method, termed as All-in-one VidEo Restoration Network (AverNet), which comprises two core modules, i.e., Prompt-Guided Alignment (PGA) module and Prompt-Conditioned Enhancement (PCE) module. Specifically, PGA addresses the issue of pixel shifts caused by time-varying degradations by learning and utilizing prompts to align video frames at the pixel level. To handle multiple unknown degradations, PCE recasts it into a conditional restoration problem by implicitly establishing a conditional map between degradations and ground truths. Thanks to the collaboration between PGA and PCE modules, AverNet empirically demonstrates its effectiveness in recovering videos from TUD. Extensive experiments are carried out on two synthesized datasets featuring seven types of degradations with random corruption levels. The code is available at https://github.com/XLearning-SCU/2024-NeurIPS-AverNet.

AverNet: All-in-one Video Restoration for Time-narying Unknown Degradations
Haiyu Zhao', Lei Tian', Xinyan Xiao², Peng Hu', Yuanbiao Gou'; Xi Peng
Coollege of Computer Science, Sichuan University, China.
Baidu Inc., Beijing, China.
State Key Laboratory of Hydraulics and
Mountain River Engineering, Sichuan University, China.
(haiyuzhao gm, penghu.ml, gouyuanbiao, pengx.gm) @gmail.com (tianlei09, xiaoxinyan] @baidu.com
Abstract
Traditional video restoration approaches designed recover clean videos from a specific type of degradation, making them ineffective in handling multiple unknown types of degradation. To address this issue, several studies have been conducted and have shown promising results. However, these studies overlook that the degradations in video usually change over time, dubbed time- varying unknown degradations (TUD). To tackle such a less-touched challenge, we propose innovative method, termed as All-in-one VidEo Restoration Network (Aver- Net), which comprises two core modules, i.e., Prompt-Guided Alignment (PGA) module and Prompt-Conditinn Conditioned Enhancement (PCE) module. Specifically, PGA addresses the issue of pixel shifts caused by time-varying degradations by learning and utilizing prompts align video frames at the pixel level. To handle multi- ple unknown degradations, PCE recasts it into a conditional restoration problem by implicitly establishing conditional map between degradations and ground truths. Thanks to the collaboration between PGA and PCE modules, AverNet empirically demonstrates its effectiveness recovering videos from TUD. Ex- tensive experiments are carried out two synthesized datasets featuring seven types degradations with random corruption levels. The code available at Aveere
https:///p http://ggtttt.../// com
Introduction
Video restoration aims to recover high-quality video from a low-quality one that is corrupted by degradations such as noise, blur, and compression artifacts. Over the past few years, numerous studies [1 2 3 5 have been conducted, yielding promising performance in video restoration. However, these methods typically require prior knowledge the specific type of degradation to design and train effective models, such denoising, dehazing, and deblurring models [6 7 9 10 [2].
In practice, obtaining such prior knowledge in advance is challenging, especially when the data is affected by multiple unknown types of degradation.
In the field of image restoration, efforts multiple unknown types of degradation using a unified model, which is known as all-in-one image hard to achieve encouraging performance by simply applying existing AIR methods to videos due to the neglect of temporal information, which leads to inferior performance, as verified in our experiments. To favor all-in-one video restoration (AVR), a few
15) have been made to restore images affected by
restoration (AIR) [66.. However,
"Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input Videos
Input Video
Noisy
Blurry
Compressed
Degradation
Degradation 2
Degradation n
?
Specific
Degradation
Specific
Model
Restored
Video
Unknown Degradations
Unified Model
Restored
Video
(a) Classic Vidco Restoration
(b) All-in-onc Vidco Restoration
Figure I: Illustration of classic and all-in-one video restoration. (a) aims to develop specific model for each degradation to handle the corrupted video, assuming that the degradation types and levels are the same and known for all frames. contrast, (b) intends handle videos containing varying unknown degradations through a unified model, which is more practical and challenging.
18 [9] have been conducted very recent, which typically assume that the frames of given video contain unknown but the same type of degradation. In other words, these methods can only handle a single unknown type of degradation and would fail when faced with multiple unknown types of degradation for given video. Clearly, the latter is more practical and challenging because degradations can vary over time in real-world scenarios (Fig. [T. For instance, motion blur when stationary object starts to move, and noise will appear when the scene changes to low-light conditions. Therefore, it is highly desirable to develop an AVR method capable of handling time-varying unknown degradations (TUD).
studies 17
To overcome the aforementioned challenge of TUD, we propose an All-in-one VidEo Restoration Network (AverNet), which consists of two core modules, i.e., Prompt-Guided Alignment (PGA) module and Prompt-Conditioned Enhancement (PCE) module. To recover videos from time-varying degradations, PGA is designed based on the following observation. Specifically, vital step in video restoration is spatial alignment across frames, which aims to climinate frame difference at the pixel level. As shown in Fig. compared to time-invariant degradations. timevirying degradations cause larger and more complex pixel shifts, making frame alignment significantly more challenging. Hence, to align frames pixel level better, PGA optimizes specifie prompt conditioned video clip first and then uses this prompt to align the frames within the clip. To tackle unknown degradations in given frame, PCE is designed to learn prompts corresponding to latent unknown degradations by building the conditional map between the corrupted frame and the ground truth. Through treating the prompts the conditions, PCE transforms the task of video restoration from multiple unknown degradations into a known conditional restoration problem.
To summarize, the contributions of this work are as below:
As far as we know, this work could be the first study on the TUD. Different from previous efforts on time-invariant degradation, our method is suitable for more practical scenarios wherein the types and levels of degradations are both unknown and changing over time.
We show that the TUD challenge could be effectively addressed by our AverNet. In brief, AverNet employs the PGA module address the issue of the large and complex pixel shifts and the PCE module to solve the problem of multiple unknown degradations.
Although the TUD setting is practical, the data with ground-truth is scarce and even impos- sible to collect. To address the data scarcity issue, we develop a data synthesis approach to simulate the data of TUD in real world. Experiments two datasets across seven types of degradations with random levels demonstrate the effectiveness of our AverNet.
Related Work
In this section, we introduce the related works in video restoration and all-in-one image restoration, and elaborate on the differences between our AvenNet and the existing methods.

---

Noisy
Frame
Clean
Noisy
Frames
Frames
Blurry
Frame
Clean
Noise
improved the Time-varying Degradations
Figure 2:
Illustration of the pixel shift issue. We compute the optical flow between two consecutive and lie-varying degradations. Several directional vectors are visualized as
frames with red arrows indicate the estimated pixel alignments between the two frames. One could observe that time-varying degradations lead to less accurate estimations compared to time-invariant degradations, causing larger and more complex pixel shift after alignment.
2.1 Video Restoration
Currently, deep video restoration methods could be roughly divided into two categories according their architectures, i.e., sliding window-based methods I
5
20
2T
2
2
19 and recurrent
methods 224 25 26
The former kind of methods typically take a short sequence of frames as input and restore only the center frame. By leveraging the temporal information from adjacent frames, these methods have achieved promising performance. For example, EDVR [ introduced the pyramid architecture and temporal-spatial attention modules to effectively aggregate information across frames. Shift-Net [5] proposed an efficient framework based on grouped spatial-temporal shift modules to implicitly aggregate inter-frame information. Although the methods have shown impressive performance, they encounter challenges in handling long- seg-sequence videos and suffer from high memory consumption. To alleviate the problem, recurrent methods choose to propagate latent features from one frame to the next frame sequentially, accumulating information from previous frames for the restoration of subsequent frames. For instance, Basic 26 proposed a concise and efficient network employing bidirectional propagation scheme. Basic bidirectional scheme with second-order grid connections to implement an effective recurrent model.
2
Different from the above methods focusing on specific degradation, ViWS-Net [17. CUDN [8] and Diff-TTA [9] explored a more generalized task of AVR, which aims to handle multiple unknown degradations through a unified model. Specifically, ViWS-Net introduces degradation messenger tokens to learn specific degradation information and employs them to guide the restoration. CUDN adaptively estimates the features of unknown degradations and employs the estimation to guide the model to remove diverse degradations. Diff-TTA introduces test-time adaptation techniques [27] to adapt the distribution of test data and recalibrates the parameters of pre-trained models to address unknown degradations. Although there is a similarity between AverNet and these methods in the task of AVR. they are remarkably different in both problem and solution. In problem, ViWS-Net, CUDN and DTf considered the unknown types of degradations, while AverNet considers not only the unknown types and levels of degradations, but also their variations over time, which is more practical and challenging. In solution, they follow the paradigm of sliding window- methods and employ local information for restoration. In contrast, AverNet improves the propagation scheme of the recurrent paradigm to address TUD challenge, enabling it to effectively utilize global temporal information for AVR.
2.2 All-in-one Image Restoration
Some recent works 28 30 31 have been devoted to all-in-one image restoration (AIR),
which aims to recover images from multiple unknown types of degradations using single model. The key to these methods is extracting discriminative information for different degradations, and employing this information to guide a single model for performing AIR. For example, AirNet [16] learns the degradation representations through contrastive learning, and uses the representations to guide the restoration network. PrompuR [3 proposes to use learnable prompts to encode
3

---

Mockel
Maker
oplicll
slgast
Feature
Feature Estraction
Pezttuee lex
10
Sutur
Pmeagation
Proageiiis Black
Propagation llack
Black
Faturn
Intersted
Ranibaa
Propagation
Progagation
DCN aftecb
Beater
Coer
Racks
Prepaaation Blokk
Propggaii Black
Propapetion Black
mabl
futt
Canveaaton Newwck
Blokk
Progegation Block
Propegation Black
Blok
Prompt Crrempp--riidddAAe Aligrment
Promii-Condditonee
Past
Fiue
Chasselaaton
Concacmation
Acaitiiz
Signoid
Elemmmictte
Spatoa
R411
Mullipicatiio
Wagpin
(4) Phomps Generntion Integmation (PGI)
Figure 3: Architecture overview. (a) Overall architecture of our AverNet, which is mainly composed of propagation blocks. Each block consists of (b) PGA module for spatially aligning features across frames with time-varying degradations, and a (c) PCE module for enhancing the features of current frame with unknown degradations. (d) PGI modules endow PGA and PCE with the capacity of conditioning on degradations by means of input-conditioned prompts. For simplicity, the superscripts j in (b) omitted. In (c), the past feature from the last time of propagation, and 7 refers the indices of key frames.
degradation-spectie information first and then guide the restoration of clean images. ProRes 32 introduces additional visual prompts to incorporate task specifci information and utilize the prompts to guide the network for AIR. MPerceiver (33 proposes a multimodal prompt learning approach to exploit the generative priors of Stable Diffusion (34] to achieve high-fidelity all-in-one image restoration. AutoDIR [4] introduces a CLIPP-based module to detect unknown degradations, and instructs datent-diffusion-baate
module with text prompt for restoration. In addition AIR, more recent work TAO 335 has explored a challenging task of open-set image restoration, which aims handle unknown degradations that were unforeseen in the pretraining phase.
Unlike the above methods specialized for image restoration, our AverNet is devoted to all-in-one video restoration, which aims recover videos affected by varying unknown degradations using a single model. In contrast to AIR focusing on unknown degradations, our AVR is more practical and challenging since the degradations are both unknown and time-vare mare-vaynnn..
3 Proposed Method
In this section, we elaborate on our AverNet and two modules, i.e., Prompt-Guided Alignment module (PGA) and Prompt-Conditioned Enhancement module (PCE).
The architecture of our AverNet depicted in Fig. 3 Given an input video, residual blocks are first applicd to extract shallow features from each frame. Then, the features are sequentially propagated four times across the video sequence to incorporate temporal information. Specifically, the first and third times of propagation are backward, in which the features are propagated from the last frame to the first one. The second and fourth times of propagation are forward, in which the features are propagated from the first frame to the last one. Such an alternating bidirectional propagation enables the features of each frame to involve abundant information from the whole video sequence for restoration. After propagation, the features are used to generate the final output for each frame through convolutions and pixel-shufflin (6 blocks. the following, we will detail our core designs, i.e., PGA and PCE, for solving TUD challenge during the propagation process.

---

3.1 Prompt-Guided Alignment
To address the pixel shift issue and obtain aligned features from the video with time time-varying degra- dations, PGA employs deformable convolution network [37] (DCN) to align the spatial contents between frames. The calculation of DCN depends on two parameters of offsets and masks, which indicate spatial coordinates shifts and weights, respectively. Typically, the offsets and masks are esti- mated from the frame features and optical flow (3X] through convolutions. However, the time-varying degradations lead to inaccurate estimations and deteriorate the spatial alignments. Therefore, we further introduce the input-conditioned prompts that encode degradation information to guide the estimation. The graphical illustration is shown in Fig. B)) In the rest of this section, we will detail the alignment procedure in the forward propagation, which is similar in the backward propagation. For simplicity, we omit the superscript which indicates the j-th time of propagation.
Given the shallow features 4 of the i-th frame, the previous features fi-1 from (i 1)-th frame, and the optical flow 1) from the (i 1) to i-th frame, we first warp f:-1 through 1(i-1)->x
where W denotes the spatial warping function. Then, we concatenate the shallow features 4 and the warped features fi-1 into clip fü to estimate the offsets and masks of DCN. Instead of directly estimating them through simple network, we introduce prompts encode degradation-speecfie information conditioned on the clip, and integrate prompts into the clip to guide the alignment of the frames. Specifically, the integrated features I. are generated by a convolution followed by Prompt Generation & Integration (PGI) block (Fig. B))):
PGI (C(nvxxxa3)))))
where f: = ft and Concat(-) refers to channel concatenation. Specifically, PGI obtains the integrated features by generating input-conditioned prompts and fusing them with the input features. Taking x as the input features, PGI first predicts attention- weights W E R. and then applies them to prompt embeddings E N xW obtain input-conditioned prompts P. To be specific, PGI sequentially applies global average pooling (GAP), x 1 convolution, and softmax operation on obtain the weights W:
W = Soft
After that, is calculated as the weighted sum of E followed by x 3 convolution for refinement:
P= Convaxal
Σ Wn E,).
Note that has a fixed spatial size it x W. which may different from the size of frame features. Therefore, we apply the bilinear upsampling operation to upscale P to the same size the input features. Finally, the prompts are integrated into the input features x through:
9x RCConcat(r, P)),
where R(-) denotes residual blocks. After obtaining the integrated features 9s, we combine them with the input f. and refine them using a 3 x 3 convolution:
h Convaxa (Foncat(f..g.))
(6)
where h; is the intermediate features to compute DCN parameters. Finally, the DCN offsets ((11))++ and modulation masks (111))++ are calculated as:
(1-1))+ = "(1-))+ + (h:),
m(j-1)-s = (h:),,
where R(om) denotes a stack of convolutions, and a denotes the sigmoid function. A DCN is then applied to the previous feature fi-1 for spatial alignment:
f
(8)
where D(-) denotes a deformable convolution network.
5

---

3.2 Prompt-Conditioned Enhancement
Here, we detail PCE module in the forward propagation, which is similar in the backward propagation. To handle multiple unknown degradations, PCE employs the prompts corresponding to the latent unknown degradations as the conditions to enhance the current frame features during propagation. However, the prompt extraction and conditional enhancement require additional computations which are non-trivial. To reduce the computational cost, PCE only conducts enhancement at selected key frames and collaterally enhances the features of other frames through propagation. Specifically, key frames are sparsely selected based on a fixed interval T. The indices of key frames can be expressed as list 1t. which is arithmetic sequence with a common difference of T'. During the propagation of key frames, PCE enhances the features based on the aligned features f and the shallow features 4 of key frames.
To obtain the enhanced features f/. PCE first employs PGI to extract the key frame prompt k; based on the shallow features & according to the procedures shown in Fig. d). Then, the key frame prompt is used the condition to enhance aligned features f! through few residual blocks:
F. = R[Conccat()
i Lk.
where R-)) denotes residual blocks, Concat(-) denotes concatenation along channel dimension, and In refers to the list of indices of key frames. After that, the enhanced features are fused with the propagation features of last time for the subsequent propagation:
H = R(Concal "T))) i TR.
(10)
where fl and - are the propagation features of current and last time at the i-th frame, respectively, In the cases without enhancement, i.e., the frame index i & Ig, the aligned features directly concatenated with the propagation features of last time and then refined through residual blocks to obtain propagation features f/:
f = + R[(Comcatt ((), i i Tr- Note that f = 14 for the first time of propagation.
(11)
4 Experiment
In this section, we conduct experiments to evaluate our AverNet. In the following, we introduce the experimental settings first and then show quantitative and qualitative results. Finally, we conduct ablation analyses to demonstrate the effectiveness of our designs.
4.1 Experiment Settings
Video Synthesis Approach. Although the TUD setting is practical, the data with ground-truth is scarce and even impossible to collect. To tackle this issue, we develop a data synthesis approach to simulate such pairs based on the degradation models in [39 40) To be specific, clean video is first cut into multiple clips with a fixed interval Then, for each clip, series of degradations are sampled from seven candidate degradations with a probability of 0.55. Specifically, the candidate degradations include Poisson noise, Gaussian noise, speckle noise, resizing blur, Gaussian blur, JPEG compression, and video compression. The details for each degradation are provided in the appendix. After that, the degradations are shuffled and added to the clip to produce a corrupted one. Finally, the corrupted clips are assembled into corrupted video that contains time-varying unknown degradations.
Video Datasets. We adopt two widely-used video datasets in experiments, i.e., DAVIS 41 and Set8 42). Specifically, DAVIS contains 90 training sequences and 30 test sequences of resolution 854 x 480. Set8 has 8 test video sequences of resolution 960 x 540. We train all models on DAVIS training set, and test them on DAVIS-test and Set8. The training and test pairs are constructed through the above video synthesis approach. For the training set, the interval of degradation variations is set to 6 and the degradations are the above seven degradations including three types of noise, two types of blur, and two types of compression. For the test set, we apply different settings to generate diverse sets for full evaluations, which are detailed in Sec.

---

Implementation Details. We use the same settings for all experiments. To be specific, the number of channels is set to 96, and the embedding length, dimension, and size of prompts are set 5, 96, and 96x9, respectively. For optical flow estimation, we use the pre-trained SPyNet 43 44 parameters and runtime are included in our AverNet.
whose
Training Details. The experiments are conducted in PyTorch [45] framework with four NVIDIA GeForce RTX 3090 GPUs. For training, we use Charbonnier loss 466 and Adam [47 optimizer with 3, = 0.9 and 822 0.999. The initial learning rates of main and optical flow networks are set to and 2.5c respectively, which are gradually decreased to le- through cosine annealing strategy [48]. The number and resolution of input frames are set to 12 and 256 x 256, respectively. We train the networks with the batch size of for 600K iterations, in which the parameters of optical flow network will not be updated for the first 5K iterations.
4.2 Comparison Experiments
In this section, we compare our AverNet with existing state-of-the-arr methods on the video datasets with TUD. To be specific, existing methods include four representative all-in-one image restoration
methods and four conventional video restoration methods. The all-in-one image restoration methods
are WDiffusion [9], Trans Weather [50]. PromptIR [3]. and AirNet [16]. The video restoration
methods are EDVR II, Basic ++ (2) Shift-Net [5]. and RVRT 4 Both the video and image
restoration methods were trained on DAVIS training set from scratch according to the training settings
in their papers. Note that the image restoration methods were trained and tested on each frame of the video sequences. We present the parameters and runtime of our AverNet and the compared methods in Tab. []] which are computed according to their original test settings. Specifically, the runtime was computed on video with 48 frames from DAVIS-test.
Table 1: Comparisons on parameters and runtime. From the table, one could observe that our method is more efficient than most AIR methods.
Video Restoration
All-in-one Image Restoration
AVR
Method
EDVR
VSR+
Shift-Net
RVRT
WDiffusson
Teather
AirNet
PrompuR
Ours
#Param
23.6M
7.4M 5.09s
12.9M
13.6M
80.0M
38.1M 3.32s
8.9M
35.6M
41.3M 6.93s
Runtime
6. 6.8
45.54s
15.07s
3956.50s
12.98s
10.81s
To comprehensively evaluate AverNet on dealing with TUD, we conduct experiments on time-varying unknown degradations with different variation intervals and different degradation combinations. To be specific, the variation interval refers to the interval of video degradation variations over time and degradation combinations refer to various types of degradations in the video.
Evaluation on Different Variation Intervals. To evaluate the effectiveness in handling TUD, we test all models on six test sets that generated through the video synthesis approach, where the variation interval is f. Specifically, six test sets are synthesized with the variation intervals = 6, 12, 24 based on DAVIS-test and Set8, respectively.
As shown in Tab. our method nearly outperforms other methods across all six test sets. For example, our method outperforms RVRT by 0.08dB-0.18dB in PSNR on DAVIS-test and outperforms Shift- Net by up to 0. 0.99dB and 0.65dB on DAVIS-test and Set8, respectively. Additionally, as shown Tab. I. our method achieves the best performance while maintaining the highest efficiency. Specifically, compared with our method, the runtime of RVRT is more than double, and that of Shift-Net is over six times longer. In comparison with the all-in-one image restoration methods, our method yields significantly better results, showing the importance of leveraging temporal information in videos. For example, our AvenNet outperforms AirNet by at least 1.53dB/0.00 0422 and I 1.02dB/0.0345 in PSNR/SSIM on DAVIS-tes. and Set8, respectively, while requiring nearly half the runtime.
Fig. and Fig. 5 present the qualitative results on DAVIS and Set8 datasets with interval = 12, respectively. From the figures, one could observe that all-in-one image restoration methods like Prompt|R and AirNet yield distorted and blurry results. Furthermore, BasicVSR++ and RVRT exhibit residual noise and artifacts. In contrast, our method excels in recovering structures and capturing finer details, resulting in clearer restorations.
7

---

Table 2: Quantitative results compared to state-of-the-an methods on test sets with various variation intervals. i the interval of degradation variations the frame sequences. The best outcomes are highlighted in bold. From the table, one could observe that our method almost outperforms other methods on all test sets.
DAVIS-test
Set8
Method
t=6
t=12
E=24
[=6
1=12
[=24
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
WDiffusion
31.74
0.8768
31.79
0.8784
31.92
0.8809
30.31
0.8784
30.02
0.8716
30.82
0.8746
Trans Weather
31.11
0.8694
31.13
0.8699
31.26
0.8741
29.24
0.8662
28.95
0.8565
29.15
0.8632
AitNet
32.46
0.8873
32.46
0.8887
32.75
0.8928
30.71
0,8874
30.40
0.8806
31.16
0.8825
PromptIR
32.18
0.8843
32.19
0.8867
32.45
0.8900
30.79
0.8903
30.43
0.8821
31.19
0.8847
EDVR
28.70
0.7224
28.37
0.6991
0.7289
0.7259
26.94
0.7382
28.71
0.7675
BS+
33.22
0.9204
33.07
0.9180
33.32
0.9210
30,90
0,90448
30.52
018965
31.35
0.9011
Shift-Net
33.09
0.9096
33.10
0.9113
33.34
0.9133
31.15
0,9027
30.82
0.8947
31.88
0.9000
RVRT
33.99
0,9314
33.98
0.9311
34,10
0.9315
31.73
0.9192
31.39
0,9113
32.47
0.9178
AverNet (Ouns)
34.07
0.9333
34,09
0.9339
34.28
0.9356
31.73
0.9219
31.47
0.9145
32.45
0.9189
Ver RAPIDE
VePi RAPIDE
Reri RAPIDE
Verl VePi LAPIDE
Input
WDiffusion
TeatsWeathrr
AieNet
PromptIR
Verk RAPIDE RAPIDEVe RAPIDE
Verlt RAPIDE VEPIDEVerk RAPIDE RAPIDE
BasicVSR++
Shift-Net
RVRT
AverNet (Ours)
GT
Figure 4: Qualitative results the "tractor" video from DAVIS-test (t 12). from which one could observe that existing methods leave residual noise or artifacts in the results. In contrast, our method obtains clearer results that are closer to GT.
Evaluation on Different Degradation Combinations. To evaluate the robustness in the situation with different degradation combinations, we construct new test sets by removing one type of degradation from the seven candidate degradations in the video synthesis. For example, we construct a test set with new combination of noise and blur degradations by removing JPEG compression and video compression in the seven degradations. As result, we obtain three different combinations, i.e., &bise&blur, noise&comppession, and blur&compression.
From Tab. D and 3 one could observe that our method obtains comparable or better performance while embracing much higher efficiency. For example, our method outperforms RVRT by 0.34dB and 0.24dB PSNR on the DAVIS-ter and Set8. respectively, with the & combination. Note that our method only requires less than half the time of RVRT to process the videos. Besides, our method outperforms Shift-Net by at most 1.0ldB and 0.47dB in PSNR on DAVIS-test and Set8, respectively. Compared with all-in-one image restoration methods, our method has superior performance in every degradation combination on the test sets.
4.3 Ablation Experiments
To investigate the effectiveness our AverNet, we conduct ablation experiments on the core modules, namely, PGA and PCE. All experiments were conducted the test sets with t 12.
To verify the effectiveness of PGA and PCE, we replace each them with conventional propagation and alignment modules, respectively, while excluding the guidance and the condition from prompts. The results are presented in Tab, 4 In the table, is apparent that each component brings considerable improvement, with PSNR gains ranging from 0. to 15BB on the two test sets. Specifically,

---

Table 3: Quantitative results compared to state-of-the-art methods in three degradation combinations, Le., noise&blur, noise&compression, and blur&compression. The best outcomes are highlighted in bold. From the table, one could see that our method outperforms other methods in SSIM, and obtains comparable PSNR values RVRT while requiring only half the runtime.
DAVIS-test
Set8
Method
Noise
Blur
Noise Comp.
Blur & Comp.
Noise & Blur
Noise & Comp.
Blur Comp-
PSNR
SSIM
PSNR
PSNR
PSNR
SSIM
PSNR
SSIM
WDiffusion
32,70
0.8990
33.52
0.9124
33.76
0.9142
31,64
0.8943
30.88
0.8968
31.22
0.8978
TransWeather
31.74
0.8863
32.53
0.9062
32.18
0.9017
29.74
018714
29.93
0.8886
29.61
0.8830
AirNet
33.41
0,9078
0.9184
34.59
32.15
0.9065
31.27
0,9019
31.60
0,9027
PrompiIR
33.69
0.9128
34.18
0.9213
33.87
0.9179
32.10
0,9033
31.54
0.9106
31.53
0.9047
EDVR
28.00
0.6809
29.58
0,7036
34,17
0.9082
27.82
0.7268
27.23
0.7245
32.15
0.8845
+S++
33.89
0.9324
34.72
0.9391
34,82
0.9392
31,88
0.9189
31.42
0.9152
31.35
0.9146
Shift-Net
34.00
0,9277
34,91
0.9390
35.26
0.9376
32.66
0.9159
31,86
0,9184
32.08
0.9164
RVRT
34.67
0.9438
35.69
0.9504
35.94
0.9503
32,70
0.9291
32.40
0.9297
32.38
0.9291
AverNet (Ours)
35.01
0.9468
35.89
0.9531
35.87
0.9506
32.94
0.9326
32.33
0.9309
32.37
0.9306
93
75
93
93
15
93
15
93
Input
WDiffusion
TransWeather
AirNet
PromptR
93
33
93
93
93
BasieVSR=+
Shift-Net
RVRT
AverNet (Ours)
Figure 5: Qualitative results on the "touchdown" video from Set8 12), from which one could see that existing methods yield blurry or distorted results. In contrast, the results of our method have sharper outlines and less artifacts.
the model without PCE suffers from PSNR drops of 1. 1.10dB and 1.67dB on DAVIS-test and Set8, respectively. Similarly, the model without PGA exhibits PSNR drops of 1.50dB and 1.33dB on the two test sets. Notably, as Set8 contains longer videos, the propagation errors caused by multiple unknown degradations are more serious. By enhancing the propagated features of key frames. PCE effectively mitigates these errors and makes significant contributions to the final performance. Additionally, the model without both PGA and PGE shows significant PSNR drops 1.66dB and 3.48dB, highlighting the effectiveness of prompt guidance and prompt conditioning.
Table 4: Ablation studies of the modules. Each module brings improvements in PSNR and SSIM, verifying their effectiveness.
DAVIS-test
Set8
PGA
PCE
PSNR
SSIM
PSNR
SSIM
(A)
32.43
0.8910
27.99
0.8404
B
32.59
0.9157
30.14
0.8958
(C)
32.99
0,9156
29.80
0.8755
(D)
34.09
0.9339
31.47
0.9145
To investigate the influences of intervals between key frames, we change from 6, 12, to 24 and present the results in Tab. One could observe that PCE with varied T obtains similar results

---

DAVIS-test while PCE with larger T suffers from 0,45dB drop in PSNR on Set8. This is because the Set8 contains much longer videos compared to DAVIS-test. Consequently, the accumulated propagation errors on Set8 are more serious. This result further shows the importance of our PCE module. According to the results, we find T is suitable value and set the interval of key frames to 6 in all experiments.
Table 5: Ablation of the key frame interval T PCE. From the table, one could see that larger T, i.e.
fewer key frames result in worse performance on the long videos of Set8.
Datasets
DAVIS test
Set8
Intervals
T=6
T=12
T=24
T=6
T=12
T=24
PSNR
34.09 0.9339
34.15 0.9349
34.14
31.47
31.02
31.02
SSIM
0.9349
0.9145
0.9066
0.9066
Conclusions
In this paper, we study a practical and challenging problem in video restoration, i.e., time-varying unknown degradations. To solve the problem, we propose AverNet which could recover clean video from the corrupted ones with TUD. Different from existing video restoration methods, AverNet assumes the degradations are time-varying and could handle TUD without the prior of degradation. Extensive experimental results show the superiority of AverNet both quantitative and qualitative comparisons.
Acknowledgment
This work was supported in part by the Fundamental Research Funds for the Central Universities under Grant CJ202303; in part by NSFC under Grant U21B2040; in part by Sichuan Science and Technology Planning Project under Grant 2024NSFTD0038.
References
[1] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. Proceedings of the IEEF/VV conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[2] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. Basicvsr++: Improv- ing video super-resolution with enhanced propagation and alignment. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5972-5981, 2022.
[3] Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Lue Van Gool. Vrt: A video restoration transformer. arXiv preprint 120.2228..22 2022.
[4] Jingyun Liang, Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy 1lg, Simon Green, Jiezhang Cao, Kai Zhang, Radu Timofte, and Luc V Gool. Recurrent video restoration transformer with guided deformable attention. Advances in Neural Information Processing Systems, 35:378-393, 2022.
[5] Dasong Li, Xiaoyu Shi, Yi Zhang, Ka Chun Cheung, Simon See, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. simple baseline for video restoration with grouped spatial-temporal shift. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2023.
pages
[6] Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool. Flow-guided sparse transformer for video deblurring. International Conference on Machine Learning, pages 13334-13343. PMLR, 2022.
10

---

[7] Jing Lin, Xiaowan Hu, Yuanhao Cai, Haoqian Wang, Youliang Yan, Xueyi Zou, Yulun Zhang, and Lue Van Gool. Unsupervised flow-aligned eequence-to-sqqueee learning for video restora- tion. In International Conference on Machine Learning, pages 33394-13404 PMLR, 2022.
[8] Zichun Wang, Yulun Zhang, Debing Zhang, and Ying Fu, Recurrent self-supervised video denoising with denser receptive field. In Proceedings of the 31 ACM International Conference on Multimedia, pages 7363-7372 2023.
[9] Jiezhang Cao, Qin Wang, Jingyun Liang, Yulun Zhang, Kai Zhang, and Luc Van Gool. Practical real video denoising with realistic degradation model. 2022.
[10] Jinshan Pan, Haoran Bai, and Jinhui Tang. Cascaded deep video deblurring using temporal sharpness prior. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3043-3051 2020.
[11] Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, and Jimmy Ren. Spatio-temporal filter adaptive network for video deblurring. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1482-2491. 2019.
[12] Senyou Deng, Wenqi Ren, Yanyang Yan, Tao Wang, Fenglong Song, and Xiaochun Cao. Multi-scale separable network for ultra-high-defiiiion video deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14030-14039, 2021.
[13] Vaishnav Potlapalli, Syed Waqus Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one blind image restoration. arXiv preprint arXiv:2306.13090 2023.
[14] Yitong Jiang, Zhaoyang Zhang, Tianfan Xue, and Jinwei Gu. Autodir: Automatic all-in-one image restoration with latent diffusion. arXiv preprint 1I11:2310010123,, 2023.
[15] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018 2023,
[16] Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 17452-17462, 2022.
[17] Yijun Yang, Angelica Aviles-Rivero Huazhu Fu, Ye Liu, Weiming Wang, and Lei Zhu. Video adverse-weatthercommmnnn suppression network via weather messenger and adversarial backpropagation. In Proceedingy of the IEEE/CVF International Conference Computer Vision, pages 13200-13210, 2023.
[18) Yuanshuo Cheng, Mingwen Shao, Yecong Wan, Lixu Zhang, Wangmeng Zuo, and Deyu Meng. Cross-consistent deep unfolding network for adaptive all-in-one video restoration. arXiv preprint arXiv:2309.01627, 2023.
[19] Yijun Yang, Hongtao Wu, Angelica I Aviles-Rivens. Yulun Zhang, Jing Qin, and Lei Zhu. Genuine knowledge from practice: Diffusion test-time adaptation for video adverse weather removal. arXiv preprint arXiv:2403.0768/ 2024.
[20] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back nack-projection network for video super-resolution. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3897-3906, 2019.
[21] Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real-time video super-resolution with spatio-temporal networks and motion compensation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1778-4787, 2017.
[22] Jianing Deng, Li Wang, Shiliang Pu, and Cheng Zhuo. Spatio-temporal deformable convolution for compressed video quality enhancement. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10696-10703, 2020.

---

[23] Takashi Isobe, Songjiang Li, Xu Jia, Shanxin Yuan, Gregory Slabaugh, Chunjing Xu, Ya-Li Li, Shengjin Wang, and Qi Tian. Video super-resolution with temporal group attention. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2020.
[24] Chao Zhu, Hang Dong, Jinshan Pan, Boyang Liang, Yuhao Huang, Lean Fu, and Fei Wang. Deep recurrent neural network with multi-scale bi-directional propagation for video deblurring. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 3598-3607,
2022.
[25] Meisong Zheng, Qunliang Xing, Minglang Qiao, Mai Xu, Lai Jiang, Huaida Liu, and Ying Chen. Progressive training of a two-stage framework for video restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1024-1031, 2022.
[26] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Basicvsr: The search for essential components in video super-resolution and beyond. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4947-4956, 2021.
[27] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[28] Ruoteng Li, Robby T Tan, and Loong-Fah Cheong. All one bad weather removal using architectural search. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 3175-3185, 2020.
[29] Wei-Ting Chen, Zhi-Kai Huang, Cheng- Cheng-Che Tsai, Hao-Hsiang Yang, Jian-Jiun Ding, and Sy-Yen Kuo. Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward unified model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17653-17662, 2022.
[30] Yurui Zhu, Tianyu Wang, Xueyang Fu, Xuanyu Yang, Xin Guo, Jifeng Dai, Yu Qiao, and Xiaowei Hu. Learning weather-general and weather-specific features for image restoration under multiple adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21747-21758, 2023.
[31] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang. Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[32] Jiaqi Ma, Tianheng Cheng, Guoli Wang, Qian Zhang, Xinggang Wang, and Lefei Zhang. Prores: Exploring degradation-aware visual prompt for universal image restoration. arXiv preprint 2306v 13653, 2023.
[33] Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, and Ran He. Multimodal prompt perceiver: Empower adaptiveness generalizability and fidelity for all-in-one image restoration. In Proceedings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pages 25432 25444, 2024.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[35] Yuanbiao Gou, Haiyu Zhao, Boyun Li, Xinyan Xiao, and Xi Peng. Test-time degradation adaption for open-set image restoration. arXiv preprint arXiv:2312.021100 2024.
[36] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
12

---

[37] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773,2011
[38] Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2288 2766, 2015.
[39] Jiezhang Cao, Qin Wang. Jingyun Liang Yulun Zhang, Kai Zhang, Radu Timofte, and Luc Van Gool. Learning task-oriented flows mutually guide feature alignment in synthesized and real video denoising. arXiv preprint arXiv:2208. 1803, 2022.
[40] Kai Zhang, Yawei Li, Jingyun Liang, Jiezhang Cao, Yulun Zhang. Hao Tang, Deng-Ping Fan, Radu Timofte, and Luc Van Gool. Practical blind image denoising via swin-conv-une unet and data synthesis. Machine Intelligence Research, 20(6):822-836, 2023.
[41] F. Perazzi, J. Pont- Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016.
[42] Matias Tassano, Julie Delon, and Thomas Veit. Dvdnet: A fast network for deep video denoising. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1805-1809 IEEE, 2019.
[43] Anurag Ranjan and Michael J Black. Optical flow estimation using spatial pyramid network. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 4161-4170, 2017.
[44] Simon Niklaus. A reimplementation of SPyNet using Py PoTorh. https://github... com/ sniklaus hwwlluusppytorch-eoo spynet 2018.
[45] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Cimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[46] Pierre Charbonnier, Laure Blanc-Feraut Gilles Aubert, and Michel Barlaud. Two determin- istic half-quadratic regularization algorithms for computed imaging. In Proceedings of 1st international conference on image processing, volume 2, pages 168-172. IEEE, 1994.
[47] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6900, 2014.
[48] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[49] Ozan Özdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[50] Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel. Transweather: Transformer- based restoration of images degraded by adverse weather conditions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2353-2363, 2022.
A Appendix
In this section, we first present more details about degradations in the data synthesis approach. Then, we present more qualitative results to demonstrate the effectiveness of our AverNet. Finally, we discuss the broader impacts and limitations of this work.
13

---

A.1 Degradations in Video Synthesis
To address data scarcity issue for studying time-vrrying unknown degradations (TUD), we propose new approach based on the degradation model in [40 39] to synthesize corrupted-clent video pairs that contain TUD. In the synthesis pipeline, the video clips are degraded by three major categories of degradations, i.e. noise, blur, and compression. In the following, we will detail the types and parameters of each degradation within these categories.
Noise. In the pipeline, there are three kinds of common noise, iee., Gaussian noise, Poisson noise, and speckle noise. For Gaussian noise and speckle noise, the noise levels both uniformly sampled from [10, 15]. The Poisson noise is mathematically modeled as
n ~ P(10" x x)/100 2,
(12)
where the a is uniformly sampled from [2,4].
Blur. There are two types of blur in pipeline, i.e., Gaussian blur and resizing blur, which usually appear in action videos and Internet videos. For Gaussian blur, the kernel size is uniformly sampled from (3,5,7), and the kernel is randomly chosen from ('iso', 'aniso', "generalized iso', 'generalized amiso', 'plateau_iso" 'plateau_aniss For resizing blur, the resize scale and the interpolation mode is uniformly sampled from [0.5, 2] and ("bilinear', 'area", 'bicubic') respectively.
with the probabilities of (0.405, 0.225, 0. 0.108, 0.027, 0.108, 0.027).
Compression. This degradation includes JPEG and video compression. For JPEG compression, the quality factor is randomly chosen from (20,30,40). For video compression, the codecs and bitrate are randomly selected from ("libx264', 'h264', "mpeg4') and [1e4, 1e5), respectively.
A.2 Qualitative Results on Videos with TUD
In addition to the qualitative results presented in the main body of the paper, we show more results on the datasets with TUD. To be specific, Fig. [ and present the qualitative results on DAVIS-tes 441
and Set8 [42] datasets with variation intensity t = 6. Moreover, Fig. 8 shows the qualitative results in the &oise&blur combination on DAVIS-IeSt As shown in Fig. 6 and BasicVSR++ [2] and Shift-Net [5] leave noise and artifacts in the frames. Furthermore, RVRT produces frames with significant color distortion. contrast, our method yields results with finer details and less artifacts. From Fig. g one could observe that the results of all-in-one image restoration methods are blurry, and video restoration methods Basic VSR+ and RVRT leave artifacts in the results. In contrast, the results of our method have clearer outlines and are closer to the GT.
A.3 Broader Impact
In this section, we discuss the impact of our AverNet in a broader vision. Generally, AverNet the first all-in-one solution to recover videos that contain time-varying unknown degradations, which are prevalent in real-world scenarios. Therefore, may have multiple applications such as film restoration, surveillance video enhancement, and medical image restoration. However, the videos restored by AverNet may not have the permission the original copyright holder, thereby infringing the rights of others. Moreover, the training and testing of the model consume lot of electricity, which causes carbon emissions.
A.4 Limitation
In addition to the three types of degradations studied in this paper, there are more degradations such as rain, haze and snow that are worth exploring under the TUD setting. Besides, the training data for AverNet is based the aforementioned video synthesis approach, which generates videos with time-varying unknown degradations closing to real-world scenarios. However, the corruption in the real-world videos more complex and hard to be simulated. Therefore, in real-world applications, our AverNet needs further validation and improvement.
14

---

Input
WDiffussin
Trans Weather
AirNet
PromptR
Basic VSR++
Shift-Net
RVRT
AverNet (Ouns)
GT
Figure 6: Qualitative results on the "orchid" video from DAVIS (t = 6), from which one could see that existing methods leave noise or artifacts in the results. contrast, the results of method have less artifacts and finer details.
Input
WDiffussio
Trans Weather
AirNet
PromptIR
Basic VSR-
Shift-Net
RVRT
AverNet (Ous)
GT
Figure 7: Qualitative results on the "park_joy video from Set8 (t = 6). from which one could observe that existing methods yield blurry or distorted results. In contrast, the results of methods are clearer and closer to the GT.
5767
5767
St
5767
St
5767/
St
5767
St
Input
WDifiusiin
Trans Weather
AirNet
PromptR
5767 St 5767 St BasieVSR
5767
5767
St RVRT
5767 St AverNet (Ours)
5767 St
Shift-Net
Figure& 8: Qualitative results on the "subway" video from DAVIS test in the noise&blur degradation combination, from which one could observe that the results of existing methods are blurry. In contrast, the results of our method have clearer outlines and tones that are more similar to the GT.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation and training details clearly described for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[Yes]
Justification:
The code and data are publicly available. The link is included in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeeee 7/nips.com htc/://bbb///gdddd///eeeeeueisss ssioiolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The experiment settings are clearly presented the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are reported the experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts are discussed in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: It is challenging to provide effective safeguards for video restoration.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Proper citations are provided throughout the document and the licenses will be included with the code when it is released.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The document will accompany the code upon its release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "I2z8KBVhs1", "reviewer": "Reviewer_hhCd", "summary": "This paper presents a video restoration method capable of addressing time-varying unknown degradations (TUD) with a single model. The proposed method employs two modules, i.e., the prompt-guided alignment (PGA) module and the prompt-conditioned enhancement (PCE) module in the propagation to leverage the temporal information for restoration. Experiment results on various types of degradations demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses section.", "limitations": "1. The authors should provide the discussion of limitations of this paper. \n2. Related works should be addressed in a more sufficient way.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper considers a more practical and valuable problem named TUD in video restoration and presents a feasible solution to handle TUD with a single model. \n2. The proposed modules take advantage of prompt learning to handle TUD and explicitly consider the degradations during propagation, which is interesting and innovative.", "weaknesses": "1. While the paper first studies time-varying unknown degradations, these degradations are synthesized through a degradation model, which may not accurately reflect real-world degradation distributions.\n2. The intervals t of degradation variations in the test sets are all multiples of 6, which is the interval during training. It is uncertain that the proposed could generalize well on other intervals such as 9.\n3. Previous works [1,2] could adaptively selected keyframes based on video changes. In contrast, the PCE module selects keyframes according to a fixed interval T, Why do not take the adaptive methods?\n4. Some related works [3,4] are not included as they guided this area a lot. The authors should discuss them too.\n[1] Yule Li, Jianping Shi, Dahua Lin: Low-Latency Video Semantic Segmentation. CVPR 2018: 5997-6005.\n[2] Yu-Syuan Xu, Tsu-Jui Fu, Hsuan-Kung Yang, Chun-Yi Lee: Dynamic Video Segmentation Network. CVPR 2018: 6556-6565.\n[3] Jiaqi Ma, Tianheng Cheng, Guoli Wang, Qian Zhang, Xinggang Wang, Lefei Zhang: ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration. CoRR abs/2306.13653 (2023).\n[4] Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, Ran He: Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration. CoRR abs/2312.02918 (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  Evaluations on the realistic video dataset.**\n\nWe further evaluate the effectiveness of our pipeline and network on realistic video dataset VideoLQ [1]. The results show that the models trained on our pipeline generalizes well on the realistic degradations. As shown in Table 1, our network and RVRT are not re-trained on the realistic video datasets such as RealVSR [2] but still shows great performance on VideoLQ. \n\nTable 1. Quantitative results on the realistic video dataset VideoLQ.\n\n| Method     | RVRT   | AverNet    |\n| ---------- | ------ | ---------- |\n| NIQE ↓     | 4.6602 | **4.6234** |\n| PI ↓       | 3.6491 | **3.6464** |\n| CNNIQA ↑   | 0.5470 | **0.5487** |\n| HyperIQA ↑ | 0.4547 | **0.4560** |\n| CLIPIQA ↑  | 0.3800 | **0.3899** |\n\n[1] Investigating Tradeoffs in Real-World Video Super-Resolution.\n\n[2] Real-world video super-resolution: A benchmark dataset and a decomposition based learning scheme.\n\n**Q2：Other degradation intervals.**\n\nAs suggested, we conduct experiments on different intervals to show the generalization of AverNet. From Table 2, one could observe that AverNet shows better performance on two additional intervals. \n\nTable 2. Quantitative results on different intervals t=9 and t=15.\n\n| Method  | t=9       |            | t=15      |            |\n| ------- | --------- | ---------- | --------- | ---------- |\n| Metric  | PSNR      | SSIM       | PSNR      | SSIM       |\n| RVRT    | 33.92     | 0.9320     | 34.07     | 0.9347     |\n| AverNet | **34.01** | **0.9356** | **34.17** | **0.9373** |\n\n**Q3：Adaptive keyframe selection.**\n\nFollowing [1,2], we adopt an adaptive strategy to select the keyframes, which chooses the frames with largest changes as keyframes. The results are presented in Table 3, which show that the adaptive strategy only brings slight improvement in SSIM. However, the computational burden of the adaptive strategy is not negligible in practice.\n\nTable 3. Quantitative comparisons between fixed and adaptive keyframe strategy.\n\n| Keyframe Strategy | Fixed  | Adaptive |\n| ----------------- | ------ | -------- |\n| PSNR              | 34.09  | 34.09    |\n| SSIM              | 0.9339 | 0.9341   |\n\n[1] Low-Latency Video Semantic Segmentation.\n\n[2] Dynamic Video Segmentation Network. \n\n**Q4：More related works [1,2] should be discussed.**\n\nAs suggested, we will discuss the all-in-one image restoration methods [1,2] in the following and include them in the related works. MPerveiver [1] proposes a multimodal prompt learning approach to exploit the generative priors of Stable Diffusion to achieve high-fidelity all-in-one image restoration. ProRes [2] introduces additional visual prompts to incorporate task-specific information and utilize the prompts to guide the network for all-in-one restoration.\n\n[1] ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration.\n\n[2] Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration. \n\n**Q5：Discussion on limitations.**\n\nThe training data for AverNet is based on the video synthesis approach, which generates videos with time-varying unknown degradations close to real-world scenarios. However, the corruptions in the real-world videos are complex and difficult to be simulated. Therefore, in real-world applications, our AverNet may need further validation and improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8oBl2pJBl", "reviewer": "Reviewer_wJZg", "summary": "The paper considers the problem of all-in-one restoration in videos, which is fundamentally different from images due to time-varying notion of degradations affecting the videos. The paper proposes prompt based modules to condition the restoration of frames on.", "questions": "**Q1.** It is unclear up until Table 5 ablation what $t$ refers to i.e, the interval in key frames. If $t$ refers to the key frame interval, what does variation intensity mean in \"Evaluation on Different Variation Intensity\" paragraph, and how is interval in key frames used to synthesize the datasets?\n\n**Q3.** Have the authors considered a non-prompt based setting for ablation experiments? In Table 4, all scenarios have prompts.\n\n**Q2.** Have the authors considered a controlled setting wherein the degradations necessarily worsen with time i.e., severe noise, blur, etc. are introduced as time increases? This setting would highlight how well the prompts can adapt to the changing degradations.", "limitations": "The limitations, and societal impact are discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** The paper extends the problem of all-in-one restoration from the image domain to the video setting.\n\n**S2.** A recurrent prompt-based architecture is proposed for the said purpose.\n\n**S3.** Two datasets are synthesized, due to lack of such datasets, based on seven degradations with varying intensity of degradation over time.", "weaknesses": "**W1.** In longer videos (Set8), the performance of RVRT is very comparable to the proposed AverNet. Although RVRT does not include any prompts (implicit or explicit) to condition the restoration on.\n\n**W2.** The paper lacks thorough exploration of the problem. Considering weather-induced degradations as base (instead of just DAVIS/Set8), and synthesizing the video datasets then would have been a more challenging problem to evaluate the effectiveness of the prompts in longer videos.\n\n**W3.** It would benefit to include a baseline that considers the problem of all-in-one video restoration (or multiple degradations with one model) since those architectures are designed to condition the restoration procedure on the degradation information [1], [2]. I agree with the reasoning about [2] in line 84 onwards, however results on [1], and/or [2] would indicate the importance of the proposed formulation (i.e., the conditioning should take into account/model time-varying degradations).\n\n[1] Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation\n\n[2] Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1：Effectiveness of prompts in longer video Set8 with complex degradation changes.**\n\nActually, our prompt-based AverNet is more effective in dealing with complex degradation changes in longer video Set8. To highlight how well the prompts can adapt to changing degradations, we synthesize new test sets with increasing degradation severity based on Set8. The results are shown in Table 1, from which one could observe that the prompts endow AverNet with a greater capacity to handle degradation changes in long videos. In detail, we synthesize four test sets where the degradations progressively worsen over time. In the first test set, different types of degradations are gradually introduced, with their intensities increasing over time. In the other three test sets, only single type of degradation is added, with the intensity increasing over time.\n\nTable 1. Quantitative results on Set8 with increasing degradation severity over time. Multiple degradations denote noise, blur, and compression are gradually added and their severity worsen with time. Noise, Blur, and Compression denote single type of degradation which worsen with time.\n|Set8|Multiple Degradations|| Noise||Blur||Compression||\n|-|-|-|-|-|-|-|-|-|\n|Metric|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|\n|RVRT|29.63|0.8623|33.64|0.9457|26.99|0.7864|28.69|0.8794|\n|AverNet|**29.77**|**0.8656**|**33.88**|**0.9482**|**27.19**|**0.7924**|**28.89**|**0.8876**|\n\n**Q2：Effectiveness on weather-induced degradations.**\n\nAs suggested, we synthesize a new dataset with time-varying weather degradations based on the video dataset REVIDE [1]. The results in Table 2 demonstrate that our prompt-based AverNet effectively handles weather-induced degradations. Specifically, we introduce three types of weather degradations (i.e., haze, snow and rain) through our data pipeline and the synthesis approaches similar to [2,3]. Due to time limitation, we train models on the dataset with 200k iterations, and use a fast baseline BasicVSR++ for comparison. RVRT is not compared since its training is too time-comsuming to finish in the rebuttal phase.\n\nTable 2. Quantitative results on weather-induced degradations.\n|Method|PSNR|SSIM|\n|-|-|-|\n|BasicVSR++|37.36|0.9704|\n|AverNet|**39.82**|**0.9740**|\n\n[1] Learning to Restore Hazy Video: A New Real-World Dataset and A New Method.\n\n[2] Blind Image Decomposition.\n\n[3] Relationship Quantification of Image Degradations.\n\n**Q3：Comparisons with all-in-one video restoration methods.**\n\nAs suggested, we compare our method with ViWSNet [1] and present the results in Table 3. The results show that ViWSNet struggles with handling time-varying degradations and produces unsatisfying results. We speculate that this is because ViWSNet imposes a strong assumption through loss function, i.e., only single type of degradation exists in single video. \n\nNote that Diff-TTA [2] is not compared since its code is not available, and we were unable to reproduce it during the rebuttal phase.\n\nTable 3. Quantitative results of ViWSNet on DAVIS-test and Set8.\n|Test Sets|DAVIS(t=6)||DAVIS(t=12)||DAVIS(t=24)||Set8(t=6)||Set8(t=12)||Set8(t=24)||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Metric|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|\n|ViWSNet|16.38|0.5278|16.38|0.5273|16.38|0.5289|13.73|0.3579|13.72|0.3605|13.70|0.3574|\n|AverNet|34.07|0.9333|34.09|0.9339|34.28|0.9356|31.73|0.9219|31.47|0.9145|32.45|0.9189|\n\n[1] Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation.\n\n[2] Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal.\n\n**Q4：Clarifications on variation intensity t and key frame interval T.**\n\nThe confusions may arise from the misunderstanding betweeen the variation intensity 't' in the data pipeline and the keyframe interval 'T' in the PCE module. Specifically, the lowercase ‘t’ controls the interval of degradation changes in the data pipeline. A smaller t corresponds to a higher variation intensity of degradations. For instance, t=6 indicates degradation changes every six frames. Additionally, the uppercase 'T' is the hyperparameter in the PCE module that controls the interval of keyframes. For example, T=12 indicates PCE module select one keyframe every twelve frames.\n\n**Q5：Non-prompt ablation studies.**\n\nWe carry out non-prompt ablation study as suggested. As shown in Table 4, non-prompt baseline shows a significant drop in both the PSNR and SSIM metrics, highlighting the effectiveness of the two prompt-based modules.\n\nTable 4. Ablation studies on the proposed prompt-based modules.\n||PGA|PCE|DAVIS-test||Set8||\n|-|-|-|-|-|-|-|\n||||PSNR|SSIM|PSNR|SSIM|\n|(A)|||32.43|0.8910|27.99|0.8404|\n|(B)||✓|32.59|0.9157|30.14|0.8958|\n|(C)|✓||32.99|0.9156|29.80|0.8755|\n|(D)|✓|✓|34.09|0.9339|31.47|0.9145|\n\n**Q6：Experiments on degradations worsen with time.**\n\nAs suggested, we carry out experiments under the above setting. The results are presented in Table 5, from which one could observe that our prompt-based AverNet significantly outperforms RVRT, which demonstrates the prompts effectively adapt to the changing degradations.\n\nTable 5. Quantitative results on test sets with increasing degradation severity over time. Multiple degradations denote noise, blur, and compression are gradually added and their severity worsen with time. Noise, Blur, and Compression denote single type of degradation which worsen with time.\n|DAVIS-test|Multiple Degradations||Noise||Blur||Compression||\n|-|-|-|-|-|-|-|-|-|\n|Metric|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|\n|RVRT|33.26|0.9201|35.81|0.9541|28.78|0.8264|28.99|0.8613|\n|AverNet|**33.41**|**0.9238**|**36.38**|**0.9577**|**29.24**|**0.8389**|**29.31**|**0.8804**|\n|**Set8**|**Multiple Degradations**||**Noise**||**Blur**||**Compression**||\n|Metric|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|\n|RVRT|29.63|0.8623|33.64|0.9457|26.99|0.7864|28.69|0.8794|\n|AverNet|**29.77**|**0.8656**|**33.88**|**0.9482**|**27.19**|**0.7924**|**28.89**|**0.8876**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bIS6wauxz", "reviewer": "Reviewer_Zzy9", "summary": "The authors propose a prompt learning based framework for all-in-one video restoration with time-varying degradations. Their work employs a prompt-guided alignment module to overcome pixel shifts caused by time-varying degradations. Multiple unknown degradations are learned through a prompt-conditional module.", "questions": "- How does the model performance change under more complex degradation pipelines used in works such as BSRGAN or Real-ESRGAN? It is also not clear why these degradation pipelines were not at least considered as a starting point.\n- How does the model perform when adding multiple degradation to the same frame snippets or having a collection of different degradations per frame snippet instead of sequentially adding different single degradations to the video snippets", "limitations": "N/a", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors have a solid motivation to address time varying degradations in videos in an unified restoration setting\n- Their proposed method consistently outperforms prior work in their considered settings", "weaknesses": "- The data pipeline used in this study appears overly simplistic and disrupts the content dependencies of corruptions like noise, which varies with overexposed or underexposed frames. Instead of merely adding random degradation types to random video frames, it would be more realistic to simulate the severity of these degradations over time (e.g., increasing JPEG compression, blur, or noise), thereby preserving temporal dependencies.\n- Merely increasing the variation intensity by reducing the number of frames per degradation is too simplistic. As mentioned earlier, increasing the severity of degradation is more beneficial.\n- Following the practicality aspect of proposed method, the authors do not provide evaluation on realistic degraded videos, such as VideoLQ or NoisyCity4, whether their time-varying degradation model can compete with prior work in this more complex setting. \n- The efficiency comparison in Table 1 appears inaccurate, as the number of parameters for both PromptIR and AIRNet is incorrect. There is a significant discrepancy between the officially reported numbers and those listed in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1&Q2：Increase the severity of degradations over time.**\n\nAs suggested, we conduct new experiments by synthesizing four test sets with progressively worsening degradations and the results prove the effectiveness of our method. Specifically, in the first test set, different types of degradations are gradually introduced, with their intensities increasing over time. In the other three test sets, only one type of degradation is added, with its intensity increasing over time. Since the models have seen various degradations and their variations during training, we directly apply them to these test sets without retraining. The results are presented in the table below. From the tables, one could observe that our AverNet is effective in handling various degradation changes and outperforms RVRT in the settings where degradations worsen over time.\n\nTable 1. Quantitative results on increasing degradation severity over time. Multiple degradations denote noise, blur, and compression are gradually added and their severity worsen with time. Noise, Blur, and Compression denote single type of degradation that worsen with time.\n\n|DAVIS-test|Multiple Degradations||Noise||Blur||Compression||\n|-|-|-|-|-|-|-|-|-|\n|Metric|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|\n|RVRT|33.26|0.9201|35.81|0.9541|28.78|0.8264|28.99|0.8613|\n|AverNet|**33.41**|**0.9238**|**36.38**|**0.9577**|**29.24**|**0.8389**|**29.31**|**0.8804**|\n|**Set8**|**Multiple Degradations**||**Noise**||**Blur**||**Compression**||\n|Metric|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|PSNR|SSIM|\n|RVRT|29.63|0.8623|33.64|0.9457|26.99|0.7864|28.69|0.8794|\n|AverNet|**29.77**|**0.8656**|**33.88**|**0.9482**|**27.19**|**0.7924**|**28.89**|**0.8876**|\n\n**Q3：Evaluations on realistic degraded videos.**\n\nAs we do not find the NoisyCity4 dataset, we can only conduct evaluations on the realistic video dataset VideoLQ [1]. The results in Table 2 show that our AverNet is more effective in dealing with realistic and complex degradations.\n\nTable 2. Quantitative results on realistic video dataset VideoLQ.\n\n| Method         | RVRT   | AverNet    |\n| -------------- | ------ | ---------- |\n| NIQE ↓         | 4.6602 | **4.6234** |\n| PI ↓           | 3.6491 | **3.6464** |\n| CNNIQA **↑**   | 0.5470 | **0.5487** |\n| HyperIQA **↑** | 0.4547 | **0.4560** |\n| CLIPIQA **↑**  | 0.3800 | **0.3899** |\n\n[1] Investigating Tradeoffs in Real-World Video Super-Resolution. \n\n**Q4：Parameters for PromptIR and AirNet.**\n\nTo comprehensively compare the models, we use the PyTorch model profiling API THOP to calculate the parameters. THOP calculates parameter counts using hooks on modules, which may result in lower values than those officially reported. We recalculate and update the results, i.e, PromptIR has 35.60M parameters, and AirNet has 8.93M parameters.\n\n**Q5：Clarification on our degradation pipeline and that of BSRGAN and Real-ESRGAN. The performance under pipelines of BSRGAN and Real-ESRGAN.**\n\nWe argue that the degradation pipelines used in BSRGAN [1] or Real-ESRGAN [2] are not necessarily more complex than ours. In fact, our method has a comparable level of complexity with them. To provide a clearer comparison, we have summarized the key components of our pipeline alongside those of BSRGAN and Real-ESRGAN in Table 3 below. Additionally, our pipeline is designed for all-in-one video restoration that addresses time-varying unknown degradations in videos, whereas these pipelines are developed for blind image super-resolution.\n\nTable 3. Comparison of degradation pipelines.\n\n| Degradation Types | Blur                              | Downsampling | Noise                                                        | Compression                             |\n| ----------------- | --------------------------------- | ------------ | ------------------------------------------------------------ | --------------------------------------- |\n| BSRGAN | Gaussian Blur| Resize| Gaussian  Noise, Processed camera sensor Noise| JPEG Compression                        |\n| Real-ESRGAN | Gaussian Blur, 2D sinc filter |Resize|Gaussian Noise, Poisson Noise, Color Noise, Gray Noise|JPEG Compression|\n| Ours | Gaussian Blur, Resizing Blur |-|Gaussian Noise, Poisson Noise, Speckle Noise |JPEG Compression, Video Compression |\n\nAs suggested, we synthesize new test sets based on the pipelines of BSRGAN and Real-ESRGAN to evaluate the performance of our models. The results, presented in Table 4, show that our AverNet achieves comparable or even superior performance on both test sets synthesized through BSRGAN and Real-ESRGAN. Note that downsampling operation was removed to maintain the frame size. \n\nTable 4. Quantitative comparisons on the test sets synthesized through BSRGAN and Real-ESRGAN.\n\n| Pipeline | BSRGAN    |            | Real-ESRGAN |            |\n| -------- | --------- | ---------- | ----------- | ---------- |\n| Metric   | PSNR      | SSIM       | PSNR        | SSIM       |\n| RVRT     | 26.31     | **0.7004** | 25.36       | **0.6036** |\n| AverNet  | **26.34** | 0.6977     | **25.37**   | 0.5971     |\n\n[1] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution.\n\n[2] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data.\n\n**Q6：Clarification on the degradation pipeline.**\n\nActually, our pipeline adds multiple degradations simultaneously to the same frame snippets. Specifically, in our pipeline, each type of degradation has a 0.55 probability of being sampled and applied to per snippet. In other words, each frame snippet in our test sets usually involves multiple degradations. Consequently, our experiments indeed evaluate the models on test sets containing multiple degradations per snippet, rather than just a single degradation per snippet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAm6xXsO0e", "reviewer": "Reviewer_2PV4", "summary": "This paper studies the time-varying unknown degradations in videos and proposes an all-in-one video restoration network to recover corrupted videos. Specifically, the network consists of two modules named PGA and PCE, which are designed to address the pixel shifts issue caused by time-varying degradations and to tackle multiple unknown degradations, respectively. Through the collaboration of them, the network could effectively handle the time-varying unknown degradations.", "questions": "Please see the weaknesses.", "limitations": "Potential impacts and limitations have been discussed in the supplementary material.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of time-varying unknown degradations studied in this work is practical and challenging. In real-world scenarios, the degradations in videos dynamically change over time, and their types and levels are always unknown.\n\n2. Compared with classic video restoration methods that deal with one specific degradation, the proposed method could handle time-varying and multiple unknown degradations with one model.\n\n3. The paper comprehensively discussed existing video restoration methods and all-in-one image restoration methods, as well as their differences from the proposed method.", "weaknesses": "1. As shown in Table 2, although the proposed method could effectively handle time-varying degradations with different variation intervals, the variation intervals of the test sets are fixed. Could the proposed method handle degradations with variable intervals?\n\n2. The experiments are only conducted on the test sets with combined degradations. How about the performance on single type of degradation with variable levels? \n\n3. There is a recent method [1] that deals with multiple degradations. What are the differences between this method and the proposed one? Additionally, the authors should include it in the related works. \n\n[1] Yang, et al. Video adverse-weather-component suppression network via weather messenger and adversarial backpropagation. ICCV, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1：Evaluation on degradations with variable intervals.**\n\nAs suggested, we conduct new experiments by synthesizing new test sets with variable degradation intervals. The results show that our method effectively handles degradations with variable intervals. Specifically, the test sets are synthesized based on DAVIS-test and the intervals are randomly sampled from [t-v, t+v].\n\nTable 1. Quantitative results on the test sets with variable intervals.\n\n| Test Sets  | t=6, v=3    |            | t=12, v=6   |            |\n| ------- | ----------- | ---------- | ----------- | ---------- |\n|         | PSNR        | SSIM       | PSNR        | SSIM       |\n| RVRT    | 33.8849     | 0.9306     | 34.3231     | 0.9330     |\n| AverNet | **34.0313** | **0.9338** | **34.3317** | **0.9344** |\n\n**Q2: The performance on single type of degradation.**\n\nAs suggested, we synthesize new test sets, each containing only a single type of degradation, and evaluated the models on these sets. The results, as shown in Table 2, demonstrate that AverNet consistently outperforms RVRT across all types of degradation.\n\nTable 2. Quantitative comparisons on single type of degradation.\n\n| Degradation  | Noise     |            | Blur      |            | Compression |            |\n| ------- | --------- | ---------- | --------- | ---------- | ----------- | ---------- |\n|         | PSNR      | SSIM       | PSNR      | SSIM       | PSNR        | SSIM       |\n| RVRT    | 36.35     | 0.9603     | 35.65     | 0.9545     | 34.49       | 0.9431     |\n| AverNet | **36.88** | **0.9641** | **36.65** | **0.9618** | **34.63**   | **0.9466** |\n\n**Q3: The differences between the proposed AverNet and ViWS-Net.**\n\nThe differences between AverNet and ViWS-Net [1] are discussed below and will be included in the related works. First, AverNet aims to handle time-varying unknown degradations, whereas ViWS-Net specifically focuses on a single type of degradation within one video. Additionally, AverNet employs prompt-guided modules to conditionally restore videos, while ViWS-Net explicitly optimizes a weather discriminator to classify the degradations and guide the restoration.\n\n[1] Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The final scores for this work are as follows: strong accept, weak accept, marginal accept, and marginal reject. On a broader scale, the overall evaluation leans towards a positive reception, though there is some divergence in opinions. After the rebuttal phase, the reviewer inclined towards rejection focused primarily on the limitations regarding performance improvement and chose to maintain his rating. However, two reviewers raised their scores, with one upgrading to a strong accept. These two reviewers indicated that the authors' rebuttal had addressed their previous concerns and acknowledged the importance and efficacy of the work. In summary, despite one reviewer's concerns about the contribution to performance improvement, he did not explicitly identify issues at the methodological level, and his score remains borderline. The other three reviewers are inclined to accept the work, with two being particularly decisive in their support. Therefore, I am inclined to accept this work.
Decision: Accept (poster)