Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements
Authors: Gaurav Shrivastava
Ser-Nam Lim
Abhinav Shrivastava
Venue: NeurIPS 2023 poster (2023)
Keywords: Computational Photography
Deep Internal Learning
low-level vision
video denoising
video super-resolution
video frame interpolation
video inpainting
TL;DR: Robustly performing low-level vision tasks for videos such as denoising, super-resolution, frame-interpolation, and inpainting using only the internal statistics of videos.

Abstract: In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus. Our proposed approach directly learns the weights of neural modules by optimizing over the corrupted test sequence, leveraging the spatio-temporal coherence and internal statistics of videos. Furthermore, we introduce a novel spatial pyramid loss that leverages the property of spatio-temporal patch recurrence in a video across the different scales of the video. This loss enhances robustness to unstructured noise in both the spatial and temporal domains. This further results in our framework being highly robust to degradation in input frames and yields state-of-the-art results on downstream tasks such as denoising, object removal, and frame interpolation. To validate the effectiveness of our approach, we conduct qualitative and quantitative evaluations on standard video datasets such as DAVIS, UCF-101, and VIMEO90K-T.

Video Dynamies Prior: An Internal Learning Approach for Robust Video Enhancements
Gaurav Shrivastava University of Maryland, College Park gauravsh@und.co edu
Ser-Nam Lim
University of Central Florida sernamOucf edu
Abhinav Shrivastava University of Maryland, College Park abhinav8cs umd.edu
Abstract
In this paper, we present novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus. Our proposed approach directly learns the weights neural modules by optimizing over the corrupted test sequence, leveraging the spatio-temporal coherence and internal statistics of videos. Furthermore, we introduce novel spatial pyramid loss that leverages the property of spatio-temporal patch recurrence in video across the different scales the video. This loss enhances robustness to unstructured noise both the spatial and temporal domains. This further results in our framework being highly robust to degradation in input frames and yields state-of-the- tasks such denoising, object removal, and frame interpolation. To validate the effectiveness of our approach, we conduct qualitative and quantitative evaluations standard video datasets such DAVIS, UCF-101, and VIMEO90K-T.
results on downstream
Introduction
Recently, video enhancement [54, 42, 43, 1, 12, 18, 49, 32] and editing [56, 11, 24, 50] tasks have attracted much attention in the computer vision community. Most of these tasks are ill-posed problems, i.e., they do not have a unique solution. For example, enhancement tasks such as video frame interpolation can have infinitely many plausible solutions. Therefore, the aim is to find visually realistic solution coherent in both the space and time domain for such tasks.
Priors play critical role in finding reasonable solutions to these ill ill-posed problems by learning spatio-temporal constraints.
recent years, deep learning has emerged as most promising technique [54, 13, 39, 38, 4, 2, 43, 49, 22] for leveraging data to create priors thar enforce these spatio-temporal constraints. However, the generalization of these data-driven priors mainly relies on data distribution at training time.
In the current era of short video platforms such TikTok, Instagram reels, YouTube shorts, and Snap, millions of new artistic elements (e.g., new filters, trends, etc.) are introduced daily. This trend causes substantial variations between the train and the test sets, which results diminished effectiveness of the learned prior. Moreover, frequent fine-tum fine-tuning or frequent training to improve the efficacy of the prior significantly hamper the scalability of such models.
Navigate to the webpage for video results.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Furthermore, the existing video enhancement approaches assume clean and sharp input frames, which may not always be the case due factors like bad lighting conditions during the scene capture or video compression for internet transmission. These factors introduce noise in the clean video signal, and feeding such noisy input frames to video enhancement approaches results in suboptimal enhancements.
To address these challenges, we propose novel Video Dynamics Prior (VDP) that leverages the internal statistics of query video sequence to perform video-specific enhancement tasks, By utilizing a video's 's internal statistics, our approach eliminates the need for curated training data. Thus, removing the constraints with regard to encountering test time variations. Additionally, we introduce spatial pyramid loss that enhances robustness in the temporal domain, enabling our method to handle noisy input frames encountered at test time.
VDP utilizes two fundamental properties of a video sequence: (1) Spatio-tempont patch (ST patch) recurrence and (2) Spatio-temporal consistency. ST patch recurrence is a fundamental finding [34] that reveals significant recurrence of small ST patches (e.g.. 5x5x3 or 7x7x3) throughout a natural video sequence. To leverage this strong prior, we employ an W.STT--Convouttion-- which is detailed in Sec. 3. Moreover, to leverage the ST patch recurrence property across scales in a video, we introduced a novel spatial pyramid loss. In Sec. 3.5, we demonstrate that our novel spatial pyramid loss helps the VDP model to be more robust to unstructured noise in both spatial and temporal domains.
architecture,
Further, our approach considers the next frame prediction task to initialize our recurrent architecture. The task aims to auto-regressively predict the next frame, taking the current frame and previous frames information context. We leveraged architecture from such task as it promotes the spatio-temporal consistency in the processed video frames and models the complex temporal dynamics required for various downstream tasks. In our approach, we first formulate a next-frame prediction model with unknown parameters. Then depending on the downstream task, we maximize the observation likelihood with slight modification in the objective function.
This paper presents three key contributions to the field of video processing. Firstly, we introduce a novel inference time optimization technique that eliminates the need for training data collection to learn neural module weights for performing tasks. Secondly, we propose a novel spatial pyramid loss and demonstrate its effectiveness in providing robustness spatial and temporal noise in videos. Lastly, we showcase the superior performance of our approach in multiple downstream video processing tasks, including video denoising, video frame interpolation, and video object removal. Our method outperforms current baselines and exhibits robustness in case noisy frames are provided as input for enhancement.
The rest of the paper is organized as follows. In Sec. 2, we briefly introduce the background work done in the area of video enhancement tasks. In Sec. 3, discuss methodology and key components of our architecture. Sec. 4-7, we briefly describe the video processing tasks and accompanying modifications to the objective function for dealing with different tasks at hand. Finally, we conclude our work in Sec. 9.
2 Related Works
Over the past decades, many research efforts have been made to increase the efficacy of video enhancements. Traditional methods like [33, 16,51] utilized affine-estimation based motion approach to perform video enhancement tasks. However, these methods were very limited in their capability of learning good representation of videos to generalize well on various scenes. All of this changed with the recent developments in deep learning techniques that considerably improve the representation of videos and deliver high performance gains video enhancement tasks.
Internal Learning. In the recent past, considerable number of deep internal learning approaches were put forth to perform image enhancement tasks using just a single test image [37, 46, 3, 10, 28, 36, 29,449 48]. Such approaches utilize patch recurrence property in a natural image as strong prior to deal with inverse problems such as image denoising [46, 10, 21], super-resolution [37, 46, 29], and object removal [46]. However, when such internal learning approaches are directly applied to videos for enhancement tasks, it creates flickering artifacts in the processed video [55, 19]. These artifacts arise as image-based based methods fail to maintain a video's temporal consistency, critical component of video signal. Extending the deep internal- based approach to videos remains

---

Recabual Oonnection
X
Pass
Transposedt Upacall (20
FDN
FDN
Backward Pass
Uppccale (20
TranspseeDD
LFPN
212
Fg N(O,))
Chonneie
Frame Decoder Network (FDNet)
(1) Unrolling the LFPNet time
VDP Model Pipelliee
Figure Model pipeline: We are given XI...., X, frames of person exercising (depicted in the Fig). Here, we start by sampling 2g ~ (0, 1) as an initial latent representation. Our model comprises two key modules: (i) the Frame Decoder Network (FDNet) and (ii) the Latent Frame Predictor Network (LFPNet), depicted in the figure. These modules, together form the pipeline of our VDP model, as shown (iii). We optimize the weights these neural modules on this specific input sequence by doing backpropagation over the loss pertaining to the task at hand.
challenging problem. There very few works proposed in the area that deal with videos, and all of these works are tailored towards a specific video enhancement task [57, 56]. In this paper, we extend deep internal-leaming based framework to the video space that effectively explores various video enhancement tasks such as denoising, super-resolution, frame interpolation and inpainting, utilizing single structure of our VDP model.
Flow Guided Methods. In the image processing tasks, most of the methods utilize the 2D CNN modules. However, if we use such models in standalone settings, we often find that they are inept processing the additional temporal dimension present in a video sequence. order to overcome the temporal consistency problem, some researchers have opted for optical-flow based approach to capture the motion occurring in the videos [54, 9, 42, 26, 15, 20, 12]. Essentially, they model the flow of the video sequences by doing explicit motion estimation. Then they perform warping operations (motion compensation) to generate enhanced video frames. These optical-flow based approaches have found plenty of applications in tasks such as video denoising [54, 42], video super- resolution [54, 12, 6, 5], video object removal [11, 52], video frame interpolation [54, 2, 17, 14, 23]. However, accurate motion estimation using optical flow is inherently challenging and computationally expensive. Therefore, deep neural networks such as RAFT [44] and SPyNet [27] are employed to estimate optical flow. These networks rely on lots of training data learn to predict the optical flow. This double reliance on data by both the video enhancement approach and optical flow prediction networks can introduce dataset biases. Further, we showcase that the flowbased models for video super-resolution and frame interpolation tasks are not robust to noise or artifacts arising from capturing the scene low-lighting or bad lighting conditions. In our work, instead of explicitly performing motion estimation and motion compensation, we utilize LSTM-Conv decoder networks represented in Fig. 1 to capture the temporal information implicitly. Furthermore, our method is able to infer context directly from the test video sequence itself. Thus, mitigating the problems arising from dataset biases. Additionally, we empirically demonstrate that our approach is stable for tasks like super-resolution and frame interpolation, even in the presence of noisy input frames sequence.
3
Model Overview
Given video sequence as context, we perform various video-specific processing tasks on this sequence of frames. In traditional settings, the training data used learn good representations that capture the patterns in the video sequences. However, in our case of no data, we rely on proper choice of a deep neural network that can capture the ST patch recurrence property of a video as an informative prior for the video processing task at hand. Our model for performing all video-specific tasks is a combination of two modules. (1) The latent frame predictor network and (2) The frame decoder network as depieted in Fig. (iii....
3.1 Latent Frame predictor Network (LFPNet)
The task of the latent frame predictor network is to maintain the spatio-temporal consistency of the video. It learns the dynamics of a video sequence in latent space, i.e., it models the temporal relations between the frames in latent space. We use LSTM cells as the building block for this module as they implicitly capture the temporal relation between the latent frames. This LFPNet module (is inspired
3

---

by (9)) g(-) : R + R" has a fully-connectoo layer, followed by four LSTM layers with 1024 cells each, and a final output fully-connected layer. The output fully-connected layer takes the last hidden state from LSTM (he+1) and outputs 24+1 after tanh(-) activation.
3.2 Frame Decoder Network (FDNet)
This network maps the frame embeddings from the latent space to the image space, i.., f(-) R² + Rexhxw The architecture for our VDP's frame decoder network is depicted in the Fig. 1.(i.. The residual skip connections in our architecture play crucial role in the performance of our module FDNet by addressing the vanishing gradient problem and facilitating faster convergence.
3.3 Formulation of Video Dynamics Prior
For our proposed Video Dynamies Prior VDP prior only rely on the two modules namely LFPNet and FDNet. We start by sampling an initial latent noise vector 2g ~ NO /) where ag RR Here, D is the dimension of latent space. For experiments, we keep the dimension D of latent space as 1024. Now, as illustrated in Fig. 1, we pass this 2g embedding through the 'LFPNet" and obtain the next frame embedding 21. We then pass 21 through the frame decoder network to obtain the next image frame X1 Mathematically, future frame as X.+1 (=)))) Where, f RDD + Rexhx denotes the frame decoder network ($3.2) that maps the latent space to image space i.e. X(+1 = [(=++1) Function g(-) denotes the latent frame predictor network. This network takes the latent embedding at the current step as the input and predicts the latent embedding at the next step as the output, i..., 2++1 g(x). Leveraging future frame prediction task helps in initializing the LFPNet in the latent space.
can write the expression for the forward pass obtain the
3.4 Neural Module Weights Optimization
Since we optimize the weights of neural modules (LFPNet and FDNet) using corrupted sequence, reconstruction loss alone is not enough to find a visually plausible enhancement of the input video. Hence, we introduce few other regularization losses.
Reconstruction Loss: We use combination of L1 and perceptual loss to formulate the reconstruc- tion loss. It is given by Eqn. 1,
Cree = ||++1 - f(g + -
(I)
Here, (b-) in the Eqn. denotes the pre-trained VGG network on ImageNet [31]. X., X, denotes the input and reconstructed video frames at timestep t, respectively.
Spatial Pyramid Loss: It has been well established that small ST patches of size 5 x 5 x 3 recur within and across the scales for a video. Additionally, bicubic downscaling of video results in coarsening of the edges in the downscaled video. This results in the downscaled video to have less motion speed and less motion blur. Both of these properties are extremely useful for video enhancement tasks. Hence, to utilize such lucrative properties of a video, we introduced spatial pyramid loss. For this loss, we scale down the input image to 3 levels (Ar where k is 2,4,8) using downsampler and calculate the reconstruction loss for each level. is given by Eqn. 2,
Capl
Σ 1=2,4,8
Here, X1+1 denotes the input video frames at timestep + 1. d=(-) in the Eqn. 2 denotes the downsampler network and i denotes scale of downsampler. For example, dz(-)) Rexhxw mexh/22x/2
Variation Loss: This is variant of Total variational loss introduced in the paper [30]. Variation loss smoothens the frames while preserving the edge details in the frame. It is given by Eqn. 3,
W-1 EE
Evar (Xi)
Σ3
Σ
+ 11 #/g-1)--
(3)

---

-22
Loss
Less
L1
Veriston
MI m025
-80
Veitst
Egocht
MI 0.19
(ii)
(iii)
Figure 2: (i) Video B is generated by replacing the second frame of video A. (ii) Top depicts the video obtained during the early round of optimization of the objective given by Eq. for video B. Further, the average Mutual information [41] of video(intermediate) is much closer video A. (iii) The loss plot depicts the convergence of our VDP model for in five different settings.
Here, fi denotes the pixel of the processed video frames (X, f(g(2(1))) at timestep t and location [.]
Final Loss: We combine all the losses together into final loss follows,
T-1 Xrehhee Cuph + Avathrvar
Linul
Σ
(4)
3.5 Robustness of Video Dynamics Prior
In this section, explore the internal learning properties of our model. For thorough exploration, we perform various experiments to explore the spatio-temporal patch recurrence [34] in natural videos using our model. In the past, authors of the papers [10, 46] have explored the utilization of the patch recurrence property in images by the deep learning models to perform various enhancement tasks. Following these prior works, we extend such understanding to the non-trivial spatio-temporal patch (or ST ST-patch) recurrence phenomenon exhibited by our model.
We design simple experiment where we have a natural video sequence A denoted as V, = {X1, 333 and depicted in Fig. 2.).). We create another sequence V2 by replacing the frame X₂ with random noisy frame X, resulting in V. = {X1, Xut, X33) As shown in Fig. 2. (ii), the noisy frame is fairly independent
frames X1 or X3 hence, we get mutual information between these frames close to zero. This replacement of frame X, with X,y results in higher entropy [45] of the video sequence V,, (larger diversity in temporal dimension + lower ST patch recurrence - lower self-simillrity-> higher entropy of sequence) in comparison original sequence (lower diversity temporal dimension + higher patch recurrence > higher self-similarity - lower entropy of sequence). Hence when the VDP model is optimized over both the sequences V. and V.. it requires a longer duration for loss convergence as depicted by Fig. (ii).. During the optimization over the video sequence V., model obtains the sequence resembling Va (given by Vintemnddane in Fig. 2.)i in early rounds of optimization, near the epochs where we observe the first dip in the convergence curve and before ending up overfitting sequence V/.
We conducted such experiments on our model under five different settings. 1) Using only L1 reconstruction loss (given by Eqn. 1) video V,. 2) Using only L1 reconstruction loss video Vn, 3) Using only L1 reconstruction loss and spatial pyramid loss (given by Eqn. 2) on video Vn, 4) Using only L1 reconstruction loss and variation loss (given by Eqn. 3) on video V.. 5) Using L.1 reconstruction loss, spatial pyramid loss, and variation loss on video Vo.
The energy plot in Fig, 2(ii) represents the trend in MSE loss for the aforementioned five settings. It can be observed that the absence any noise in the video signal (yellow), the model fits the input quickly. The quick convergence can be attributed to higher ST patch recurrence in video sequence A. The VDP model optimized over the video sequence V/ using only the reconstruction loss (green) provides some impedance to the temporal noise but quickly overfits the noise after couple of iterations. Further, the VDP model optimized under setting 3 (red) or (blue) provides higher impedance to noise than setting 2 but still converges after a few hundred epochs. However, when the VDP model is optimized using setting (cyan), it provides considerably higher impedance to noise in the temporal domain.
5

---

(a) Poisson Noise (A 25)
(b) V Dnet (PSNR 18.81488
(c) UDVD (PSNR 21.1294) )
M2F2 (PSNR 18.8224)
Ours (PSNR 31.26994
Ground Truth
Frame from a video in the DAVIS dataset denoised using different approaches. (a) Frame corruрted with additive poisson noise of intensity 25 (relative to intensity range [0-255]). Figure (b) depicts the denoised frames generated by FastDVDnet [43] method. Figure (c) depicts the denoised frame generated by UDVD [35] method. Figure (d) depicts the denoised frame generated by M2F2 [8] method. Figure (e) depicts the denoised frames generated by our method. Figure (f) shows the clean ground truth frames.
Figure 3: Video Denoising:
Table 1: Quantitative video denoising results: To evaluate our method, we calculate the Avg PSNR score between the generated noise-free video and ground truth video. The a denotes the standard deviation for the added Gaussian noise to the frames. While the À denotes the intensity of additive Poisson noise. For all baselines and our method, the score is calculated on the DAVIS dataset [25]. Also, blue (bold) denotes the best score. Please note use pretrained models for baselines provided by the authors. The baselines marked with denote a data-driven baseline.
Metrie
Additive Noise Type
Noise Level
M2F2 [8]"
FastDVDnet [43]"
UDVD [35]*
Ours
PSNR
Gaussian
(a 20)
34.89
35.77
35.12
35.74
PSNR
Gaussian
30)
33.21
34.04
33.92
34.07
PSNR
Poisson
(A 25)
18. 19
19.02
22.08
31.96
PSNR
Poisson
30)
17.12
18.03
20.98
30.07
We observe a similar phenomenon when optimizing our model with a larger void filled with noisy frames. Additionally, when we optimize our VDP model over noisy video (here, every frame in the video sequence consists of Gaussian noise), we do not see convergence at all. We attribute this behavior of our VDP model to promoting self-similarity of the st-patches over the output video sequence. Moreover, incorporating the spatial pyramid loss introduces frame matching at multiple levels of coarse scale. This encourages more structured signal in the output and provides higher impedance to the unstructured noise. We performed the same experiment over 250 different video sequences (randomly sampled from different datasets like Vimeo-90K, DAVIS, and UCF101) and came up with similar findings as given by Fig. (.ii)..
4 Video Denoising
In this section, present VDP model utilized for the video denoising task. A noisy video be defined as the combination clean video signal and additive noise. As shown the previous section, VDP model exhibits higher impedance the noise signal and lower impedance the clean video signal. Exploiting this property, we propose a denoiser for video sequences. Unlike traditional approaches, our denoiser does require data collection,
demonstrate that the final loss function (Eqn. 4) alone is sufficient for effective video denoising.
We optimize our model parameters the noisy test-time video sequence using the loss function given by Eqn. We then use these optimized LFPNet FDNet modules for generating high- quality clean video frames. The new enhanced video frames for each timestamp be obtained by X ((((=)))
The presence of noise in a video can be attributed to various factors, such as non-ideal video capture settings lossy video streaming over the internet. While curating training datasets for basclines,

---

Frame 1
Noisy Frame 2
Frame 3
GT
Interpolated Frame
Interpolated Frame
DAIN
Soft- Splat
RIFE
Ours
Figure 4: Video Frame Interpolation: In order compare our method existing baseline methods, we conducted visual comparisons using three frames for each method. We introduced Gaussian noise (a 15) in the second frame and tasked all methods with interpolating frames between them. can be observed that our method not only denoises the second noisy input frame but also generates denoised intermediate frames. This is in contrast to other baseline methods that interpolate noise along with the clean signal.
Table 2: Quantitative video frame interpolation results: For the evaluation of method calculate the Avg PSNR and SSIM score between the generated interpolated video frames and ground truth video. For all baselines and our method, the score is calculated on the UCF101 dataset [40]. The color blue (bold) denotes the best performance. Please note we use pretrained models for baselines provided by the authors. The baselines marked with denote data-driven baseline.
Metries
# Noisy Frames
Noise lovel
ToFlow [54]"
DAIN
SoftSplat [23]"
RIFE
Ours
PSNR
None
None
34.68
35.00
35.39
35.41
35.41
SSIM
None
None
0,9677
0.968
0,970
0.970
0.970
PSNR
2d Frame
=
20.30
20.87
18.25
20.22
34.92
2" Frame
0.514
0.4012
0.512
common to assume that the additive noise follows Gaussian distribution. However, at test time, when this additive noise is drawn from Poisson distribution, the performance of baseline models deteriorates significantly. In contrast, the loss given in Eqn. 4, achieves highly effective denoising results. Fig. provides a qualitative comparison between method and the baselines, clearly demonstrating the superior performance of our approach.
VDP model, optimized on the test sequence by minimizing
To further evaluate the effectiveness of our method, we conduct experiments on the DAVIS dataset using different noise settings. We compare our model against state-of-the-art baselines and present the quantitative evaluation in Table. 1. The results highlight the superiority of our method in video denoising tasks, showcasing its robustness handling diverse noise distributions. We encourage readers to refer to supplementary material for additional video results.
5 Video Frame Interpolation
Frame interpolation is challenging task that involves generating intermediate frames between the given frames in video sequence. The primary objective is synthesize high-quality and realistic frames that accurately capture the apatio-tempora coherence of the original video. However, existing baseline methods for frame interpolation assume that the input video consists of high-quality frames without any noise. In real-world settings, video sequences often contain noise due to non-ideal capturing conditions, which poses challenges and diminishes the performance these baseline methods. Such limitations are addressed by VDP model.
7

---

(8) Noise-Frec High Resolution Frame
Bicubic
(c) BSstcVSR++
EDVR
(c) Ours
(1) GT
Comparison results of the 'Cameraman" sequence from VIMEO-90K-T dataset. Gaussian noise of std a 5 is added to the lower-resolution input frames. (a) High-resolution noise-free ground truth frame. (b) Patch from higher resolution frame generated by BiCubic method. (c) Patch from higher resolution generated by BasicVSR++ [5]. (d) Patch from higher resolution frame generated by EDVR [49] method. (c) Patch from the higher resolution by Ours VDP method. (f) Patch from higher resolution ground truth frame. Note the noise suppression and details the camera in the patch generated by our method compared to baselines.
Figure Video Super Resolution:
Table Quantitative video super-resolution results:
For the evaluation of our method, caleu- late the Avg PSNR and SSIM scores between the generated super-resolution video and ground truth video for all baselines along with our method. The score is calculated on the standard VIME0-90K-1 dataset. Best results in the table are shown in blue (bold). Please note we use pretrained models for baselines provided by the authors. The baselines marked with denote a data-driven baseline.
Metries
Noisy Frames
Noisy Inteusity
BiCubic
ToFlow [54]"
EDVR [49]"
BasicVSR++ [5]"
Ours
PSNR
X
31.12
35.79
35.95
SSIM
X
0.870
0,920
0.937
0.940
0.936
PSNR
/
5
29.90
30.20
32,02
31.58
33.87
0.720
For this task, we utilize the latent embeddings to perform the frame interpolation. First, we optimize our VDP model parameters using the loss function given by Eqn. 4. Now, to generate intermediate frames, we linearly interpolate the latent embeddings of the video using the equation.
intermediate f(a2++1
s + (1 - a) Ya E (0,
(5)
Here, Xnneemediate is the interpolated frame between the frames X, and X For example, to perform 4x frame interpolation, we select j0.25, 0.5, 0.75) in the Eqn. 5. To evaluate the performance of our method, we conduct experiments on the UCF101 (triplet) dataset [2, 40] under two conditions: 1) ideal triplet input sequence, 2) triplet input sequence containing one noisy frame. We make qualitative comparison between our method and the baselines depicted Fig. It can observed from Fig. 4 that method is more robust to noisy input and outputs the crisp intermediate frames without noise, unlike baseline methods DAIN [2], SoftSplat [23] and RIFE [14].
Table. 6 represents the quantitative evaluation of our approach on the UCF-101 dataset. It can be observed from the table that our method outperforms the existing baselines on PSNR and SSIM metrics in the video frame interpolation task, further validating the effectiveness and suitability of our approach for this task.
Video Super Resolution
Given our ever-increasing screen resolutions, there great demand for higher-resolution video content. In this task, we are given a lower-resolution video sequence as input. We are tasked to change this input higher-resolution video sequence.
For this task, we modify the loss objective given by Eqn. 4 as follows,
T-1
arg min f.g
Σ de +
+ Xf
(6)

---

Imput
Mask
InterVI
FGVC
LongVI
Ours
Figure 6: Video Object Removal. In the above figure, we describe a video object removal task. The input in the figure represents the original video sequence frame without any alterations. The InterVI [56], FGVC [11], and LongVI [24] in the figure represent the frames generated by removing the object "car" from the video sequence. The rightmost frame depicts the removal of the object from the video frame by our method. It can be observed that our method do not create hallucinations.
Table 4: Quantitative Object Removal results: We compare the average PSNR and SSIM scores of our method with the existing baselines on the DAVIS dataset. The best performance denoted by the score in blue (bold). Please note we use pretrained models for baselines provided by the authors. The baselines marked with denote a data-driven bascline.
Metries
DIP [46]
DFGVI[53]]
FGVC
InterVI [56]
Long [24]
Ours
PSNR
26.03
27.57
31.92
30.86
31.80
32.06
SSIM
0.921
0.9289
0.9499
0.9350
0.9457
0,9512
The modified objective for the video super-resolution task is given by Eqn. 6. Here, function (.) denotes the downsampling function. For example, to perform 4x super-resolution, we would define downsampling function as duc _ + (M/4)))(()4))
We evaluate our method for the video super-resolution task on the VIMEO90-K K-19 dataset and contrast model against the tfate-of-the-ete art baseline methods. We evaluate method on two different settings: 1) When sharp lower-resolution frames are provided, 2) When noisy reworuienolution frames provided. We perform 4x super resolution of frames in the video sequences. We make qualitative comparison between our method and the baselines for the setting when noisy frames are given input frames, depicted in Fig. 7. It can be observed from Fig. 7 that our method outputs better and noise-free higher resolution frames with more details in comparison to the baseline methods. Table. 5 represents the quantitative evaluation our approach. It can be observed from the table that there is a significant drop in the performance of baselines to our method when a slight amount of Gaussian noise is added to the input frames. We demonstrate empirically that our method is much more robust to noise as compared to other baseline methods.
7 Video Object Removal
Unwanted items in video can grab the viewers' attention from the main content. In this task, we remove such objects from the original video stream. For this task, we assume a mask M = V'm,
{m²)1 Vm, {0, 1] xx is provided for each frame remove an object from the video sequence.
To perform the removal of an object, we modify the objective function given by Eqn. 4 as follows,
T Aec mi Ca (a. (.ffff((()()))))
arg min f.g
We evaluate our method for the video object removal task on the DAVIS dataset. A qualitative comparison between our method and the baselines can be seen in Fig. 6. It can be observed from Fig. that our method performs better than the existing methods in removing the object from the video without creating any artifacts in the processed videos.
Table. 4 represents the quantitative evaluation of our approach. We perform the evaluation based on the stationary mask setting. In this setting, stationary mask is applied to all frames of the video sequence. This setting helps us evaluate the model's 's ability to generate temporally coherent frames even if some information in the frames is consistently missing.

---

8 Limitations
One severe limitation our approach is that is an offline approach, i.e., it can not be used in real-time to perform video enhancements. One way to rectify this limitation is to work reducing the parameters for the frame decoder network. One suggestion is modifying the residual skip connection from concatenation- baased connections additive connections. This would drastically reduce the number of parameters to be trained with minimal effects on the efficacy of the model.
The second limitation of our approach is that it is a recurrent approach and can not be scaled using multiple GPUs. This issue can be alleviated using a temporal attention-base approach that can have long-term memory and is scalable across multiple GPUs.
Our approach, in its current stage, requires no data and only able perform low-level vision tasks. However, to perform high-level vision tasks such as performing video segmentation or video manipulation, we conjecture that external knowledge would be required, like a collection of external datasets or pretrained model on such external dataset to aid our VDP method.
9 Conclusion
We conclude in this work that a video-specific approach is way to go for a considerable number of video processing tasks. Our VDP method not only performed better than the task-specific approaches but also it is not hamstrung by the limited availability of either the computing power or the training data. We demonstrate that the introduction of spatial pyramid loss enhances the robustness of our proposed VDP method. Leveraging this loss, we were able to achieve state-of-the-ar results in all enhancement tasks, even the presence noise in the input frames. One limitation of our method is the long processing time compared to other methods at the test time inference. Nevertheless, it is important note that we do not require training as the other baseline methods (requires multiple GPU hrs if not days). In the future, we will focus on improving efficiency to shorten the processing time increased practical utility. One great advantage our approach removing the data collection requirement for such processing tasks. We hope that this step towards data-free approach will reduce the massive data mining across the internet and provide better data privacy measures.
10 Broader Impact
Our approach offers significant advantage by eliminating the need for data collection in processing tasks. This shift towards a data-free approach has the potential to mitigate the extensive data mining activities conducted across the internet, thereby addressing concerns related to data privacy. By reducing the reliance on data collection, our approach promotes more ethical and responsible use of technology.
However, it is crucial to acknowledge that our approach also has some limitations. We believe that addressing these challenges through ongoing research and development will contribute the continuous improvement and refinement of data-free approaches, thereby enhancing their broader impact.
11 Acknowledgement
This project was partially funded by the DARPA SAIL-ON (W911NF2020009) program and an independent grant from Meta AI.
References
[1] Arias and J.-M. Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60(1):70-93, 2018.
[2] W. Bao, W.-S. Lai, C. Ma, X. Zhang, Z. Gao, and M.-H. Yang. Depth-aware video frame interpolation. Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 1, 3, 7, 8, 15, 18
10

---

[3] S. Bell-Kligler, A. Shocher, and M. Irani. Blind super-resolution kernel estimation using an interna-gaan. arXiv preprint arXiv:1909. 06581, 2019.
[4] N. Bodla, G. Shrivastava, R. Chellappa, and A. Shrivastava. Hierarchical video prediction using relational layouts for human-objes interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12 12146-12155 2021.
[5] K. C. Chan, S. Zhou, X. Xu, and C. C. Loy. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5972-5981, 2022. 8, 15
[6] M. Chu, Y. Xie, L.. Leal-Taixé, and N. Thuerey. Temporally coherent gans for video super- resolution (tecogan). arXiv preprint arXiv: /8/ 1.0003333 (2):3, 2018. 3
[7] E. Denton and R. Fergus. Stochastic video generation with a learned prior. In International Conference Machine Learning, pages 1174-1183, PMLR, 2018. 4
[8] V. Dewil, J. Anger, A. Davy, T. Ehret, G. Facciolo, and P. Arias. Self-supervised training for blind multi-frame video denoising. In Proceedings of the IEEE/CVF winter conference on applications computer vision, pages 2724 2734, 2021. 6
[9] A. Dosovitskiy, P. Fischer, E. 1lg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 2758-2766, 2015. 2015.
3
[10] Y. Gandelsman, A. Shocher, and M. Irani. 'double-dip": Unsupervised image decomposition via coupled deep-image-priont In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11026-11035, 2019. 2,5
[1]] C. Gao, A. Saraf, J.-B. Huang, and J. Kopf. Flow-edge guided video completion, 2020. 1, 1,3,
[12] M. Haris, G. Shakhnarovich, and N. Ukita. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3897-3906, 2019. 1,3
[13] M. Haris, G. Shakhnarovich, and N. Ukita. Space-time-awart ment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 1
video enhance-
[14] Z. Huang, T. Zhang, W. Heng, Shi, and S. Zhou. Real-time intermediate flow estimation for video frame interpolation. In Proceedings of the European Conference on Computer Vision
(ECCV), 2022. 3, 7. 15
[15] E. Ilg. N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017.
[16] M. Irani and S. Peleg. Image sequence enhancement using multiple motions analysis. Hebrew University of Jerusalem. Leibniz. Center for Research in Computer.... 1991.
[17] H. Jiang, D. Sun, V. Jampani, M.-H. Yang, E. Learned-Miile and J. Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 9000-9008, 2018 8..
[18] Y. Jo, S. W. Oh, J. Kang, and S. Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 3224-3232, 2018.
[19] D. Kim, S. Woo, J.-Y. Lee, and I. S. Kweon. Deep video inpainting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5792-5801, 2019. 2
[20] T. H. Kim, M. S. M. Sajjadi, M. Hirsch, and B. Scholkopf. Spatio-temporal transformer network for video restoration. Proceedings of the European Conference on Computer Vision (ECCV), September 2018. 3
11

---

[21] Laine, T. Karras, J. Lehtinen, and T. Aila. High-quality self-supervised deep image denoising. Advances Neural Information Processing Systems, 32:6970-698 69800 2019.
[22] C. Lei, Y. Xing. and Q. Chen. Blind video temporal consistency via deep video prior. In Advances in Neural Information Processing Systems, 2020.
[23] S. Niklaus and F. Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437-5446, 2020. 3, 3,7,8.1 15
[24] H. Ouyang, T. Wang, and Q. Chen. Internal video inpainting by implicit long range propagation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 14579-
14588, 2021. 1,9
[25] F. Perazzi, J. Tuste, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine Hookinn- A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Puttern Recognition, 2016. 6, 18
[26] A. Ranjan and M. J. Black. Optical flow estimation using spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161- 4170, 2017.
[27] A. Ranjan and M. J. Black. Optical flow estimation using spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 3
[28] D. Ren, Zhang, Q. Wang, Q. Hu, and W. Zuo. Neural blind deconvolution using deep priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3341-3350 2020. 2
[29] Rott Shaham, T. Dekel, and T. Michaeli. Singan: Learning a generative model from a single natural image. In Computer Vision (ICCV), IEEE International Conference on, 2019. 2
[30] L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-268, 1992. 4
[31] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L.. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2211-252, 2015.
[32] M. Sajjadi, Vemulapalli, and M. Brown. Frame-recurrent video super-resolution. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 2018. 1
[33] R. R. Schultz and R. L. Stevenson. Extraction high-resolution frames from video sequences. IEEE transactions image processing, 5(6)99961111,, 1996. 2
[34] O. Shahar, A. Faktor, and M. Irani. Space-time super-resolution from a single video. IEEE, 2011. 2,5
[35] D. Sheth, S. Mohan, L.. Vincent, Manzorro, A. Crozier, M. M. Khapra, Simoncelli, and C. Fernandez-GGrands Unsupervised deep video denoising. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 1759-1768, 2021.
[36] A. Shocher, S. Bagon, Isola, and M. Irani. Ingan: Capturing and retargeting the "dna" of natural image. In The IEEE International Conference on Computer Vision (ICCV), 2019, 2
[37] A. Shocher, N. Cohen, and M. Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 3118-3126, 2018. 2
[38] G. Shrivastava. Diverse Video Generation, PhD thesis, University of Maryland, College Park, 2021.
[39] G. Shrivastava and A. Shrivastava. Diverse video generation using gaussian process trigger. arXiv preprint arXiv:2107.04619, 2021. 1.4
12

---

[40] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 8, 18
[41] C. Studholme, D. J. Hawkes, and D. L. Hill. Normalized entropy measure for multimodality image alignment. In Medical imaging 1998: image processing, volume 3338, pages 132-143. SPIE, 1998. 5
[42] M. Tassano, J. Delon, and T. Veit. Dvdnet: A fast network for deep video denoising. In 2019 IEEE International Conference Image Processing (ICIP), pages 1809. IEEE, 2019. 1, 3
[43] M. Tassano, J. Delon, and T. Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 1, 6, 14, 15
[44] 7. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow, 2020. 3
[45] M. Thomas and A. T. Joy. Elements of information theory. Wiley- Intersceters,e 2006. 5
[46] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446 9446-9454, 2018. 2,5,9
[47] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:1812.01777, 2018. 19
[48] Y. Vinker, E. Horwitz, N. Zabari, and Y. Hoshen. Deep single image manipulation. arXiv preprint arXiv:2007.01289, 2020. 2
[49] X. Wang, C. K. Chan, K. Yu, C. Dong, and C. C. Loy. Edvr: Video restoration with enhanced deformable convolutional networks, 2019. 1, 8, 15
[50] Y. Wexler, E. Shechtman, and M. Irani. Space-time completion of video. IEEE Transactions paltern analysis and machine intelligence, 29(3) 2007.
[51] T. Wiegand, E. Steinbach, and B. Girod. Affine multipicture motion- prediction. IEEE Transactions on Circuits and Systems for Video Technology, 15(2):197-209, 2005. 2
[52] R. Xu, X. Li, B. Zhou, and C. C. Loy. Deep flow-guided video inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[53] R. Xu, X. Li, B. Zhou, and C. C. Loy. Deep flow-guided video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2019.
[54] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (IJCV), 127(8): 2019. 1, 8, 15, 18
[55] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3505-5514, 2018. 2
[56] H. Zhang, L. Mai, N. Xu, Z. Wang, J. Collomosse, and H. Jin. An internal learning approach to video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2720-2729, 2019. 3,9
[57] L. P. Zuckerman, E. Naor, G. Pisha, S. Bagon, and M. Irani. Across scales and across dimensions: Temporal super-resolution using deep internal learning. In European Conference on Computer Vision, pages 62-68. Springer, 2020.
13

---

Contents
Introduction
Related Works
Model Overview
Video Denoising
Video Frame Interpolation
Video Super Resolution
Video Object Removal
Limitations
Conclusion
Broader Impact
Acknowledgement
A Comparison with cascaded models
14
Ablation study
15
C Processing time..
16
Optimization Details
17
E
Additional Details FDNet
18
Dataset Descriptions
18
A
Comparison with cascaded models
Existing video enhancement approaches rely clean and sharp input frames perform the task at hand. However, this is not always the case depending on the lighting condition scene capture or compressing a video stream send it over the internet. All of these introduce noise the clean video signal. Applying video enhancement approaches in such conditions results in sub-optimal enhancements, demonstrated in the main paper. One other way of dealing with such conditions is the use of a two-stage cascaded approach. For example, we have to provide super-resolute frames for a noisy video sequence. We can get the solution by first denoising the frames and then processing these with super-resolution approaches. However, this also results in error propagation from one stage to another and results artifacts.
To evaluate the aforementioned reasoning, we perform a two-stage cascaded approach for video super-resolution and frame interpolation tasks. In this cascaded approach, we apply the frame denoiser first and then apply the VSR or VFI model. For video denoiser we use the FastDVDnee [43]. Table. and describe the quantitative evaluation of baselines(cascadee super-resolution and frame interpolation tasks. It can be observed that there is deterioration in results for the super-resolution task for the cascaded approach than directly applying the baseline. This is primarily due to the fact that when the denoiser is applied in the first stage, it washes away important details from the lower-resolution frames. When these frames are passed through the video cnper-resolution baselines, the error from the first stage is propagated further. A qualitative visualization of this is depicted Fig. 7. However, for the frame interpolation task, the case reversed, and cascaded baseline approaches yield better results than only baseline approaches. It can be seen from Table. that the cascaded baselines perform better than only baseline approaches but still perform worse than our VDP approach.
approach) with our VDP model for
14

---

(a) Bicubbi
(b) Bicubic
(c) BascVSR++
EDVR
(e) Ours
(1) GT
(Nbisy)
(Denxised)
Comparison results of the 'Cameraman" sequence from VIMEO-90K-1 dataset. Gaussian noise of std o = 5 is added to the lower-resolution input frames. (a) Bicubic extrapolation of noisy low-resolution frame (b) Bicubic extrapolation of denoised low- resolution frame. (c) Patch from higher resolution generated by BasicVSR++ [5] over the denoised low-resolution frame. (d) Patch from higher resolution frame generated by EDVR [49] over the denoised low-resolution frame. (e) Patch from the higher resolution by Ours VDP method. (f) Patch from higher resolution ground truth frame. We use standard video denoiser VastDVDNet [43] to denoise the input resolutttti video frames. Based the figure, it appears that the denoiser used in the first stage of the model removes vital information from the lower-resolutio frames, which leads to errors in the second stage. Our VDP model, the other hand, preserves more details compared to the baselines.
Figure Video Super Resolution:
Table 5: Quantitative cascaded video super-resolution(4x results: For the evaluation of our method, calculate the Avg PSNR and SSIM scores between the generated super-resolution video and ground truth video for all baselines along with our method. The score is calculated on the standard VIMEO-90S
dataset. Best results the table are shown blue (bold). Please note use VSR baselines by cascading it with the video denoiser model FastDVDnet [43]. The baselines marked with denote a data-driven baseline.
Metries
Noisy Frames
Noisy Intensity
BiCubic
EDVR [49]*
BasicVSR++ [5]'
Ours
PSNR
5
27.78
31.56 0.857
31.03
33.87
SSIM
a
5
0.806
0.853
0.878
Also, we have shown in the main paper that denoisers like FastDVDnet [43] are also prone to variations noise distribution. This would hurt the efficacy of the cascaded models if the noise is coming from a distribution other than the Gaussian distribution.
Table 6: Quantitative cascaded video frame interpolation results: For the evaluation our method, calculate the Avg PSNR and SSIM between the generated interpolated video frames and ground truth video. For all baselines and our method, the score is calculated on the UCF101 triplet dataset. Please note we use VFI baselines by cascading it with the video denoiser model FastDVDnet [43]. The color blue (bold) denotes the best performance. The baselines marked with denote
baseline.
Metrics
Noisy Frames
Noise level
ToHlow [54]
DAIN 12|"
Sof1Splat [23]"
RIFE [14]
Ours
PSNR
2d Frame
15
33.31
33.51
34.20
33.95
34.92
SSIM
Frame
a 15
0.882
0.891
0.898
0.892
B Ablation study
We performed ablation experiments to study the effect of losses for our video dynamics prior. We report the quantitative results for all the enhancement tasks in Table. 7. can be observed from the denoising section of the table that spatial pyramid loss Lxpi (given by Eqn. 2) is very important to provide robustness to both the spatial and temporal domains. Additionally, it can be observed that variation loss Lvar is very important for tasks like super-resolution, frame interpolation, and object removal. Fig. 8-11, we show qualitative comparison between VDP model optimized with different subsets of losses.
15

---

(a) Poisson Noise (A = 25)
(b) Eyee
(c) Lyee thoppt
(d) lore Lyar
(0) Eree eppp
Ground Truth
Figure 8: Video Denoising (Ablation): Frame from a video in the DAVIS dataset denoised using loss functions. (a) Frame cornupted with additive poisson noise of intensity 25 (relative to intensity range [0-255]]. Figure (b) depicts the denoised frames generated by only Lee-. Figure (c) depicts the denoised frame generated by free + Lypt. Figure (d) depicts the denoised frame generated by free + L Figure (e) depicts the denoised frames generated by free + Lypt + L Figure (f) shows the clean ground truth frames.
Video denoising: It can be observed from Fig. that only using the reconstruction loss Cee results in very suboptimal denoising. Optimizing for a high number of epochs using only Lise results in the total reconstruction of the noisy signal. Adding the variation loss L free results in smoothening of the video frame as can be observed from the Fig. 8. However, adding the spatial pyramid loss Lup free results in very strong denoiser. This can also be observed the quantitative evaluation given by Table. 7.
Video Super Resolution: It can be observed from Fig. that only using the reconstruction loss Leee results in suboptimal quality of super resolute frames with edginess in the texture. The addition of the spatial pyramid loss Spp to Lie results better texture in the high-resolution frames, yet some edgyness in the texture remains. However, the addition of the variation loss E to L + Lree results in smoothening of the texture while retaining the edges. This can also be observed in the quantitative evaluation given by Table. 7.
Video Frame Interpolation: It can be observed from Fig. 10 that only using the reconstruction loss free results in suboptimal quality interpolated frames with rough textures and missing details. The addition of the spatial pyramid loss Lep Liee results in better texture and increased details in the interpolated frames however, some edginess in the texture remains. However, the addition of the variation loss Lvar to Lup + Lree results in smoothening of the texture while retaining the details. This can also be observed in the quantitative evaluation given by Table. 7.
Video Object Removal: It can be observed from Fig. 11 that only using the reconstruction loss Croe results in suboptimal quality of interpolated region the frames with rough textures and color jitter artifacts. The addition of the spatial pyramid loss Cupl to Liee results in better texture and reduced jittering artifacts in the interpolated region in the frames. However, the addition the variation loss Lvar to Lxp Lree results in smoothening of the texture and merging of the inpainted region with the background. This can also be observed in the quantitative evaluation given by Table. 7.
C Processing time
We would like to point out here that if we take cumulative training and inference time, then our baseline would come out to be a better bargain many scenarios, as depicted by Table. 8. To put in perspective, data-heavy baseline requires ample amount of time to train, even when provided with perfect hyperparameters, making exploring different architectures quite expensive. In addition, the training dataset is limited in terms of representing real-world scenarios; this would further increase
16

---

(a) Bicihic
fiee
Cree Maapt
(d) Muur
(1t) Hiupt +hear
Comparison of results of the 'Cameraman' sequence from VIMEO-90K-T dataset. (a) Bicubic extrapolation low-resolution frame. (b) Patch from higher resolution generated by utilizing only the reconstruction loss given by Eqn. 1. (c) Patch from
Figure 9: Video Super Resolution (Ablation):
higher resolution frame generated by Lie + Set (d) Patch from the higher resolution frame generated by free var (e) Patch from the higher resolution frame generated by free Lapt La (f) Patch from higher resolution ground truth frame.
Ground Truth frame
(6) Lyec
(b) Eeect
(c) Eyec Cvar (6) Eyec thegt Lear
Figure 10: Video Frame Interpolation (Ablation): In order to compare method with different
loss functions, we conducted a visual comparison. (a) Patch generated by utilizing only the recon- struction loss given by Eqn. 1. (b) Patch generated by Lree + Lapl (c) Patch generated by Leee + Lvar. (d) Patch generated by Lree + Lepl + Lvar-
the spending time and effort in fine-tuning the network every time for setting. Please note that the inference time for our method has been calibrated on one Nvidia A6000 GPU.
D Optimization Details
To optimize our model, we use single Nvidia A6000 GPU with 48G memory to process a single video at time with 15-frame sequences of size 448x256. We optimize the modules LFPNet and FDNet weights using the entire test sequence with the Adam optimizer at a learning rate in the range of [0.0002, 0.002].
Video Denoising: For getting a good performance on the denoising task, we utilize the following weights for the different losses; Aree Aspl 0.0001 and Avar 0.0001. We observed that the performance of our VDP denoiser plateaued after 3600 epochs.
Video Frame Interpolation: For getting good performance the frame interpolation task, utilize the following weights for the different losses; > = 1, Aupt = 0.0001 and Avar 0.0001. We observed that the performance of our VDP frame interpolation model plateaued after 1800 epochs.
Table 7: Quantitative ablations results: To evaluate our method, caleulate the Avg PSNR score between the generated noise-free video and ground truth video. blue (bold) denotes the best score.
Task
Dutaset
Metric
Additive Noise Type
Noise Level
Exe
Exe
Eus
Lm
Lm + C.w Eigt
Denoising
(A
31.96 30.07
PSNR
Poisson
(A 30)
20.11
23.82
29.81
Super-Resolation
Vimeo-90KT
PSNR
31.23
33.86
32.18
35.70
SSIM
x
0.890
0.924
0.915
0.936
Prame-iolerpolation
UCF-101
PSNR
x 33.29
34.13
33.88
35.41
SSIM
x
0.943
0.945
0.963
0.970
Object Removal
DAVIS
PSNR
X 29.31
31.78
30.12
32.06
SSIM
X
0.9125
0.9451
0.9352
0.9512
17

---

Input
Mask
(a) Frec
(b) Lree thept
(c) +hear (d) Free the +L
Figure 11: Video Object Removal (Ablation). In the above figure, describe video object removal task. The GT in the figure represents the original video sequence frame without any alterations. (a) Patch generated by utilizing only the reconstruction loss given by Eqn. 1. (b) Patch generated utiliizig the loss given by Cree + Set (c) Patch generated by utilizing the loss given by Liee Lvrr. (d) Patch generated by utilizing the loss given by L + Capl + Lvar-
Table 8: Training and inference time comparison with the best baselines. Note that the inference time in the table is given as the time required to process per frame video.
Task
Dataset size
Resolution
Ours
Best Baseline
(Train + Infer)
(Train + Infer)
Super-Resolution
38,990 (7 fr/v) 50 seq (3450 fr)
112 x 64 + 448 x 256
0 15s
300 hrs + Is
Denoise
448 x 256
0 12s
200 hrs 0.1s
Object Removal
50 seq (3450 fr)
448
0 8.5s
160 hrs 8.3s
Frame Interpolation
3,782 triplets
448 256
0 10s
120hrs 0.08s
Video Super-Resolution For getting a good performance on the VSR task, we utilize the following weights for the different losses; Are = 1, Axpl = 0.01 and Avar 0.0001. We observed that the performance of our VDP SR model plateaued after 4200 epochs.
Video Object removal: For getting good performance the object removal task, utilize the following weights for the different losses; Arec = 1, Aupt 0.01 and Ava 0.0001. We observed that the performance of our VDP object removal model plateaued after 1800 epochs.
Additionally, we conducted our experiments using multiple random initializations FDNet and LFPNet modules and observed consistent quantitative results across all runs.
E
Additional Details FDNet
Each convolution layer is followed by batch normalization and Icaky rectified linear unit (LcakyReLU; negative slope = 0.2), except for the last layer of the decoder. In the last layer of the decoder, we use the sigmoid activation function. We utilize Pytorch for our implementation.
F Dataset Descriptions
Vimeo-90K-T [54] This dataset contains 7824 short clips downloaded from 'vimeo.com com'. Each clip only contains frames per clip. We used this dataset for a video super-resolution task. The lower resolution input frames have size of x 112 x 64 while the higher resolution frames have size of 3 x 448 x 256.
DAVIS [25] This dataset contains 50 curated video clips, and each clip contains unique object in the clip. The dataset has masking annotation present for this object for each clip. Each frame each clip contains mask annotation. There is 3450 total number of frames in this dataset. We used this dataset for video denoising and object removal task. We resized the resolution of each clip for the aforementioned tasks. The resolution of input frames for the task of the size x 448 x 256.
UCF [40] The UCF101 dataset contains videos with large variety human actions. We utilized this dataset for the video frame interpolation task. We used this dataset under standard settings put forth by paper [2]. There are 379 triplets with resolution of 256 x 256 pixels.
18

---

Table 9: We include LPIPS and FVD [47] metrics for completeness of quantitative evaluation of enhancement tasks (video denoising, object removal, and video interpolation). FVD a deep neural network based metric that evaluates both the spatial and temporal quality of the processed video with the ground truth video. FVD requires minimum of frames in a sequence for evaluation. Hence, we were unable to evaluate FVD for task like VFI and VSR on UCF101 triplet and Vime090KT datasets.
Task
Method
Dataset
Noisy Frames Noise Intensity FVD. LPIPS
PSNR1 SSIM1
Denoise (Poisson)
FastDVDnet
DAVIS
À 25
1743
0.65
19.02
Ours
54
0.0056
31.96
Object
FGVC
DAVIS
X
580
0.0556
31.92
0.9499
Removal
Ours
445
0.0408
32.06
0,9512
Inter-
RIFE
UCF101
nd Frame
= 15
0.412
20.22
0.512
-polation
SoftSplat-CF
0.624
18.25
0.401
Ours
0.102
34.92
0.918
VSR
EDVR
Vimeo90KT
5
0.5051
32.02
0.758
Basic
0.5263
31.58
0.720
Ours
0.181
33.87
0.878
19


Reviews:
[{"review_id": "3MR9D5l1hy", "reviewer": "Reviewer_6LSS", "summary": "This paper introduces a new inference time optimization approach for low-level image enhancement tasks. The proposed approach does not need any training data and leverages the spatio-temporal coherence and internal statistics of test videos. The authors also propose a spatial pyramid loss which makes the proposed approach robust to spatial and temporal noise. The proposed approach is evaluated on multiple video enhancement tasks (denoting, frame interpolation, super-resolution, and object removal) and is shown to perform competitively or better than existing works that use a training dataset.", "questions": "See my response in the weaknesses section.", "limitations": "The main limitation of the proposed approach is that it is an offline approach. Authors have clearly acknowledged the limitations of their approach and discussed potential ways to address them (in the supplementary material).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is written well and was easy to follow\nThe proposed approach is applicable to many video enhancement tasks without the need for training task-specific networks on large training datasets\nExperimental evaluation on several datasets clearly show the effectiveness of the proposed approach.", "weaknesses": "Since the proposed approach trains a network on the test video, the results depend on how long it is trained and the required training steps may vary from video to video depending on the content. If we train long enough, the model may start fitting to the corruptions in the input video to some extent (though the losses are providing impedance). It is unclear to me if the results presented in the paper correspond to using the same number of steps for all videos, or if the authors determined when to stop based on the loss curve for each video independently.\n\nThe performance of the proposed approach also depends on the network architecture used. Authors haven't conducted experiments to show how sensitive the results are to the choice of architectures of LFPNet and FDNet. Different architectures may have different inductive biases and may have an impact of the behavior of loss observed in Figure 2(iii).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking their time out of a busy schedule and evaluating our submission. We are glad reviewer 6LSS found our paper well-written and easy to follow. We greatly appreciate that reviewer 6LSS finds our approach versatile on many video tasks while requiring no external data. We are happy to note that the reviewer found our experimental section comprehensive. Below we address specific weaknesses and questions of the reviewer.\n\n**Weaknesses addressed:**\n\nTo ensure a fair and consistent comparison across different videos, we maintained a uniform number of training steps for all the videos in our test dataset. We mentioned the number of epochs required per task in our supplementary material Sec. D. Specifically, the optimization was executed for the mentioned number of epochs for each video, irrespective of the video content. Our presented results align with this uniform number of training steps across all videos, guaranteeing consistency in our approach. The choice of these epochs were made such because the performance of VDP plateaued after these many epochs. Thus, the potential concern raised by the reviewer about varying the training steps depending on the loss curve for each video independently was not an issue in our experimentation due to our consistent approach.\n\n**Denoising regime could be different for different architectures:** We thank the reviewer for raising this important point as exploring better architecture for FDNet and LFPNet in itself can be a good research problem. We explored two settings of FDNet (one with residual connections and another without them just the vanilla conv decoder) and LFPNet ( one with lstm cells and the other with vanilla RNN cells). The denoised regime of these can be seen from the plot in the rebuttal pdf. It can be observed from the plot that vanilla conv decoder based FDNet and rnn based LFPNet have a suboptimal denoised regime than our current design for FDNet and LFPNet."}, {"review_id": "cagMpJmXZt", "reviewer": "Reviewer_Pc2m", "summary": "This work proposed an internal learning method for multiple video-processing tasks. Without collecting large-scale training data, the proposed method can achieve comparable or even better results than baselines. But the slow computation is one obvious limitation.", "questions": "see weakness", "limitations": "see weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method is optimized individually for each video and thus does not need to collect training data.\n- Extensive experiments on different tasks such as super-resolution, denoising, and inpainting show effectiveness.", "weaknesses": "- As the proposed method is optimized for each video, I believe that it will work on many examples and it will also fail on many examples. How about the generalization ability of the proposed method? Are results in the paper cherry-picked or randomly sampled? How about failure cases?\n- The writing for the technical part can be further improved. The current version just  lists some equations and the procedures, and more motivation, explanation, insights would be better for readers.\n- One common problem of internal learning methods is that it would fail if there is insufficient information within a video. For example, if some objects keep blurry across the videos, it is difficult to recover it with internal learning. How could you solve this problem? Could the proposed method be combined with some pretrained models?\n- The optimization process is slow", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking their time out of a busy schedule and evaluating our submission. We greatly appreciate that reviewer Pc2m finds the effectiveness of our approach while requiring no external data as one of the strengths of our approach. We are happy to note that the reviewer found our experimental section comprehensive. Below we address specific weaknesses and questions of the reviewer.\n\n**Weakness addressed:**\n\n**Failure cases** primarily emerge when crucial information about the object of interest in a video is not present or discernable. For instance, we draw your attention to the provided figure in the attached rebuttal PDF. This figure demonstrates a scenario where we first downscaled the original sequence by 4x, thereby eliminating a significant amount of discernable facial information across the frames. Consequently, in the subsequent super-resolution process, our model was unable to recover all lost details, as visualized in the same figure. However, this is not a unique limitation to our approach but is also observed with other baseline methods, which is evident from the figure.\nFor the demo videos for our method, we assure you that they were not cherry-picked. We also provide quantitative results in our submission to back the evaluation of our approach. These quantitative evaluations take into account both the success and failure cases in our model performance.\n**Pretrain model usage:** In cases where a pretrained image model exists, its output could potentially be incorporated with our model to produce spatiotemporally consistent results. While we acknowledge this potential avenue, we consider this integration to be beyond the scope of the current paper.\n\n**Better motivation:** We will work on a more nuanced motivation in our introduction section of the paper by explaining the downside of relying on a data-driven pipeline and how our approach tries to resolve such issues by removing dataset biases. Building on suggestions from reviewer 6LSS, we would also incorporate how changing the architecture for FDNet and LFPNet affect the denoising regime(represented by the Fig 2(iii)) of our VDP model. \n\n**Slow optimization process:** Our primary goal in this submission was to establish the efficacy and robustness of our approach without using any external data, and we believe we have achieved that. While we acknowledge slow optimization as a current limitation, we view it as an important area for future improvements. Future work could explore more efficient optimization techniques or parallel computation strategies to accelerate the process.\n"}, {"review_id": "vWicAyPpby", "reviewer": "Reviewer_U9TZ", "summary": "Authors of the paper propose a novel framework for low-level video enhancement tasks (i.e., denoising, object removal, frame interpolation, and super resolution).\n\nThe proposed framework does not require any training data by directly optimizing the neural model parameters over the corrupted test data. \n\nThe key idea is to design modules and loss terms that leverage the spatiotemporal coherence and internal statistics of the test videos - fundamental properties of a video sequence. \n\nThe proposed approach, the VDP model, consisting of a frame encoder and a frame decoder, performs future frame prediction. \n\nThe total loss to optimize this method at the time of use (test / inference time) is a combination of (1) the typical reconstruction loss, (2) a novel spatial pyramid loss, and (3) a variational loss introduced in a prior work. \n\nVDP is more robust to spatio-temporal noises that contain in the input, and superior performance of the proposed approach was shown through qualitative results, quantitative results, comparisons to the state of the art (SOTA), and ablation studies on four low-level video enhancement tasks (i.e., denoising, object removal, frame interpolation, and super resolution).\n", "questions": "Q1: Is the proposed method the first inference time optimization technique for video tasks?\n\nQ2: Is the optimization performed on the test set (rather than a single test video) ? If so, would the size of the test set be a factor greatly influencing performance of the VDP model ? What is the size of the VDP model in terms of the number of parameters ?\n\nQ3: In section A of the Supplementary Material, line 30-31: ... for the frame interpolation task, the case is reversed, and cascaded baseline approaches yield better results than only baseline approaches.\nWhy would the denoiser wash away important details for the super-resolution task, but not frame interpolation task ?\n\nQ4: What could be other possible low-level video tasks that might benefit from ideas of this paper ? How about other video tasks that focus on pixel-level spatio-temporal dynamics, e.g., tracking and particularly tracking any points (https://deepmind-tapir.github.io/) ? Are the proposed key components of the paper mostly just for video tasks that require spatio-temporal inpainting because of the future frame prediction based formulation ? \n\nQ5: When the input contains no noise, what would be the reasons for someone to choose the proposed method over prior SOTA ? \n\nQ6: In Table 4 of the Supplementary Material, since the inference time was given as the time to process one frame of a video, would it mean that the total time (Train + Infer) for the denoising task, for example, would be 3450 frame * 12 s / 3600 ~= 12 hours,  which is also the time for the VDP denoiser to be plateaued after 3600 epochs of training ? \n\nQ7: Could you explain the following sentence ? Line 245: to perform a 4x frame interpolation, we select α = [0.25, 0.5, 0.75] in the Eqn. 5.\n", "limitations": "The auhtors have discussed the limitations. However, the proposed model might be also limited in its capability to handle videos with a high resolution, complex dynamics and a long time window. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "S1 - Novelty: \nA novel inference time optimization technique was proposed to learn neural module parameters / weights for performing various low-level video tasks.\n\nS2 - Significance: \nThe proposed method eliminates the need for training data collection. It differs from the popular paradigm and is practical. \n\nS3 - Quality:\nDetailed analyses and experimental results have demonstrated the superiority of the proposed method.\n\nS4 - Clarity:\nThe paper is well-constructed and easy to read.\n", "weaknesses": "W1 - The VDP model uses LSTM as the Latent Frame predictor Network and the choice is not well justified, although the authors have acknowledged this as a limitation of their current model and encourage readers to explore more architecture choices. \n\nW2 - Oftentimes, when the input contains no noise, the proposed approach is on par with prior SOTA. \n\nW3 - The resolution involved is still quite low (e.g., 3×448×256). Not sure if the proposed method can only handle resolution of this scale, or it can actually scale to high-resolution videos. \n\nW4 - The test videos shown in the paper have relatively simple dynamics and a short time window. E.g., for frame interpolation and object removal. When the video has more complicated interactions and dynamics, will the model still perform well ? This is not clear.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking their time out of a busy schedule and evaluating our submission. We greatly appreciate that reviewer U9TZ finds our approach novel, practical, and different from the popular paradigm. We are happy to note that the reviewer found the analysis of our approach detailed and our experimental evaluation comprehensive. We are glad that the reviewer understands the impact of our approach and how significant is the leap of getting SOTA results without the need for requiring external data. Lastly, we are glad the reviewer finds our paper well-constructed and easy to follow. Below we address specific weaknesses and questions of the reviewer.\n\n**Weakness Addressed:**\n\n**Why LSTM:**  Our choice of LSTM was motivated by its inherent ability to capture temporal dynamics in sequential data, making it a suitable choice for processing video data in our context. The memory cell in LSTM provides an efficient way to encode temporal information across frames, which is critical for video processing tasks such as denoising, super-resolution, and frame interpolation.\n\n**With no input noise the approach is at par with sota:** This is true with a caveat. Most SOTA baselines are trained and evaluated on similar datasets, which might not adequately reflect the diversity and unpredictability of real-world data. We have observed that the slightest introduction of noise into the input leads to a considerable drop in the performance of these baseline models, indicating their performance is closely tied to the specific characteristics of their training data. In contrast, our approach has demonstrated greater robustness against such perturbations, as it does not rely on any specific training data. This distinct advantage highlights the potential of our method to maintain high performance across diverse and realistic scenarios, providing more reliable and robust solutions.\nIs the approach limited by frame resolution: one simple way to scale the resolution is to add a bilinear interpolation layer before the final output layer. This way you can work with any resolution videos.\n\n**Handle complicated videos:** In our study, we utilized publicly available datasets widely acknowledged and used in previous literature. These datasets comprise videos with a diverse range of dynamics and interactions, thus offering a comprehensive testing environment for our model.\n\n**Questions:**\n\n**Are we the first internal learning approach for videos?** There exist some past internal learning approaches for videos that work on particular tasks, such as video object removal (referenced in our paper[23,53]) and traditional approaches (referenced in our paper[33,47]).\n\n**Would the size of the test set affect the performance?** The performance of our model is independent of the test set size. Number of parameters for our model is approximately 40 million.\n\n**Why the result changes with VSR and VFI when using cascaded models?** We think this might happen because the number of pixels to be predicted by the model increases 16 times in VSR(4x) task as opposed to 2 times in VFI(2x) task. So if the details are washed away by denoiser, this produces eight times more error in VSR than VFI.\n\n**Can this be extended to other approaches like point-tracking?** Our model is built on the idea of keeping the spatio-temporal consistency in a video which is also an essential component of point-tracking. So we do not see a reason why it would not be possible to extend this approach for such tasks. But at this point we considered this integration to be beyond the scope of the current paper.\n\n**Q5:** In a perfect world where no compression is taking place and all the videos are captured with ideal settings then it might be advantageous to use a trained baseline and achieve quick results.  However, this constraint does not hold true and artifacts are introduced because of compression of videos or non-ideal capture settings.\n\n**Q6:** Yes that calculation seems correct.\n\n**Q7:** When we say 4x frame interpolation, it means that for every single original frame, the system generates three additional frames to fit in-between the original ones, resulting in a video that plays at four times the original frame rate. These 3 frames are inserted at [0.25,0.5,0.75] intervals."}, {"review_id": "i1s0YU7cOY", "reviewer": "Reviewer_fnUo", "summary": "The paper presents a framework for various low-level vision tasks, such as denoising, object removal, frame interpolation, and super-resolution. What sets this framework apart is that it does not require any external training data. Instead, it directly leverages the internal statistics of videos at test-time to optimize neural network modules to perform the target task. Experiments are conducted on DAVIS, UCF-101 and VIMEO90K-T.", "questions": "1) Quantitative evaluation includes only PSNR and SSIM. However, these metrics have been identified in previous works to be problematic with respect to human-aligned notions of perceptual quality [1][2]. This problem is exacerbated for longer spatiotemporal sequences in video. Ideally, the authors could include user studies like [2][3]; in the absence of this evaluation, other metrics like FID, IS, LPIPS, etc. should be reported.\n\n2) Fundamentally, I'm not convinced that the suite of tasks chosen by the authors is indeed the best for demonstrating the strength of their framework. Concretely, while the setting of explicitly removing dependency on external data is an interesting one, I'm wondering if this is a somewhat artificial constraint, particularly for tasks such as video super-resolution, interpolation, denoising, or object removal. Such tasks already have a well-established corpus of strong pretrained generative models which demonstrate convincing zero-shot performance on new images/videos. Further works similarly perform a few iterations of test-time optimization on single images/videos to improve performance on specific downstream tasks, as in [4][5]. Why should these video tasks be treated from a \"external data\"-free paradigm?\n\n3) Questions 1) and 2) notwithstanding, the baselines used for comparison in each of the tasks are incomplete. For example, two strong prior works in [6] and [7] provide similar approaches to task-agnostic, semantic editing of videos, directly at test-time. These should be included for each of the tasks.\n\n[1] [Complex Wavelet Structural Similarity: A New Image Similarity Index](https://pubmed.ncbi.nlm.nih.gov/19556195/)\n\n[2] [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://arxiv.org/abs/1801.03924)\n\n[3] [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://arxiv.org/abs/2306.09344)\n\n[4] [Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models](https://arxiv.org/abs/2303.17599v1)\n\n[5] [MyStyle: A Personalized Generative Prior](https://arxiv.org/abs/2203.17272v1)\n\n[6] [Blind Video Temporal Consistency via Deep Video Prior](https://arxiv.org/abs/2010.11838)\n\n[7] [Learning Blind Video Temporal Consistency](https://arxiv.org/abs/1808.00449)\n\n", "limitations": "Please see the above questions with respect to limitations.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) VDP presents a simple and intuitive framework for accomplishing test-time, task-agnostic video editing.\n\n2) The evaluation across multiple video tasks lends strong evidence to the generality of the proposed framework.\n\n", "weaknesses": "1) Lack of novelty: the propagation of video latent codes for editing tasks such as object removal, super-resolution, video interpolation etc. have been explored in previous works, such as [1][2]. The perceptual similarity and total variation losses have been borrowed from previous works without modification, and the spatial pyramid loss is simply a multi-level reconstruction loss, which has also been leveraged in previous works, such as [3][4][5][6].\n\n2) Experimental setup and evaluation results are rather weak. For specific concerns, see the following questions.\n\n[1] [Video-Specific Autoencoders for Exploring, Editing and Transmitting Videos](https://arxiv.org/abs/2103.17261v1)\n\n[2] [Temporally Consistent Semantic Video Editing](https://arxiv.org/abs/2206.10590)\n\n[3] [Loss Functions for Neural Networks for Image Processing](https://arxiv.org/abs/1511.08861)\n\n[4] [Training a Task-Specific Image Reconstruction Loss](https://arxiv.org/abs/2103.14616)\n\n[5] [Learning Multi-Scale Photo Exposure Correction](https://arxiv.org/abs/2003.11596)\n\n[6] [Masked Image Modeling with Local Multi-Scale Reconstruction](https://arxiv.org/abs/2303.05251)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking their time out from a busy schedule and evaluating our submission. We greatly appreciate that reviewer fnUo finds our framework simple and intuitive for performing video enhancement tasks. We are glad that the reviewer finds our framework generalizable to multiple video tasks. Below we address specific weakness and questions of the reviewer.\n\n**Weakness addressed:**\n\nWe agree that previous works such as [3][4][5][6] have implemented a multiscale approach, however, their scope was mainly restricted to the spatial domain. In contrast, our contribution is the proposal of a novel spatial pyramid loss that extends this exploration of it’s properties in both the spatial and temporal domains. This combined utilization of spatial pyramid loss, perceptual loss, and TV loss is unique to our work and has allowed us to achieve results on par with, or superior to, state-of-the-art baselines, even those that are trained on external data. We believe this constitutes a significant leap forward, as it demonstrates that careful formulation of the loss function and model architecture can lead to solutions that match or exceed the performance of current data-driven methods, but without their inherent biases.\n\nFurther highlighting our novelty, our approach demonstrates robustness against dataset biases. We showed this by introducing mild noise to the dataset and then conducting enhancement tasks, noting a significant drop in the performance of baselines trained using a dataset. This test underscores the strength of our method - the capability to maintain performance even when test data distribution varies slightly from the training data.\n\nRegarding work [2], we appreciate the mention, but we believe there is no overlap with our research.\n\nAs for work [1], we weren't aware of this arXiv submission and will include it in our related work section. It indeed explores a video-specific approach, but there are key differences from our work. Firstly, for their reprojection property to function, they require a non-corrupted portion of the same video sequence, which may be less practical for enhancement tasks such as denoising and super-resolution. Secondly, their approach lacks a component to model temporal coherency beyond the proposed reprojection property. Thirdly, they rely solely on the L1 reconstruction loss to find a video-specific manifold, placing their approach in the L1 loss regime depicted in Fig. 2(iii) of our submission. This implies their solution's performance may significantly deteriorate in the presence of video signal noise. We appreciate your careful review and hope that these clarifications demonstrate the novelty and contributions of our work, warranting reconsideration of the reject rating.\n\n**Questions addressed:**\n\n**Metrics reporting:** We observed a general trend of reporting the PSNR and SSIM metrics in the previous papers; hence, we included only these metrics in our main paper. However, we also included other metrics like LPIPS and FVD metrics. These metrics are reported in the supplementary material in Table 5. It can be seen that our method performs better than current SOTA baselines.\n\n**Why an external data-free paradigm?** All the external corpus of the dataset is collected using some or other artificial constraints. For example, the Vimeo 90K dataset, which is commonly used to train video super-resolution baselines. The low-resolution frames in this dataset are devoid of noise or other artifacts arising after compression, a condition that seldom holds in real-world scenarios. Consequently, baselines trained on such datasets are optimally effective only under artifact-free conditions, limiting their utility in real-world settings.\n\nIn contrast, our approach is not constrained by these artificial conditions, making it more robust against artifacts arising from non-ideal capture settings. We aim to demonstrate that a well-formulated model and loss function, independent of any training data, can deliver results on par with, if not superior to, the current data-driven methods.\n\nAlthough our method could certainly benefit from a strong text-to-image model or robust image baselines, we considered such enhancements to be beyond the scope of this submission.\n\n**Q3:** Please refer to the table in rebuttal pdf."}]

Author rebuttal:
Extra results in pdf file.

Meta-review: After the review and rebuttal three reviewers argue for acceptance, one for rejection. All reviewers agree that the current experimental validation is limited, but three reviewers appreciate the technical novelty, while one reviewer found the presented work to be too close to prior work. After reading the paper and flagged related work, the AC agrees with three reviewers that the presented work as sufficiently different to warrant acceptance.
Decision: Accept (poster)