SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Quantitative video frame interpolation results: For the evaluation of our method we calculate the Avg PSNR and SSIM score between the generated interpolated video frames and ground truth video. For all baselines and our method, the score is calculated on the UCF101 dataset [40]. The color blue (bold) denotes the best performance. Please note we use pretrained models for baselines provided by the authors. The baselines marked with * denote a data-driven baseline.
Metrics	# Noisy Frames	Noise level	ToFlow [54]*	DAIN [2]*	SoftSplat [23]*	RIFE [14]*	Ours
PSNR	None	None	34.68	35.00	35.39	35.41	35.41
SSIM	None	None	0.9677	0.968	0.970	0.970	0.970
PSNR	2ndFrame	σ = 15	20.30	20.87	18.25	20.22	34.92
SSIM	2ndFrame	σ = 15	0.514	0.521	0.4012	0.512	0.918
is common to assume that the additive noise follows a Gaussian distribution. However, at test time, when this additive noise is drawn from a Poisson distribution, the performance of baseline models deteriorates significantly. In contrast, our VDP model, optimized on the test sequence by minimizing the loss given in Eqn. 4, achieves highly effective denoising results. Fig. 3 provides a qualitative comparison between our method and the baselines, clearly demonstrating the superior performance of our approach.
To further evaluate the effectiveness of our method, we conduct experiments on the DAVIS dataset using different noise settings. We compare our model against state-of-the-art baselines and present the quantitative evaluation in Table. 1. The results highlight the superiority of our method in video denoising tasks, showcasing its robustness in handling diverse noise distributions. We encourage readers to refer to our supplementary material for additional video results.
5 Video Frame Interpolation
Frame interpolation is a challenging task that involves generating intermediate frames between the given frames in a video sequence. The primary objective is to synthesize high-quality and realistic frames that accurately capture the spatio-temporal coherence of the original video. However, existing baseline methods for frame interpolation assume that the input video consists of high-quality frames without any noise. In real-world settings, video sequences often contain noise due to non-ideal capturing conditions, which poses challenges and diminishes the performance of these baseline methods. Such limitations are addressed by our VDP model.
(a) Noise-Free High Resolution Frame
Figure 5: Video Super Resolution: Comparison of results of the 'Cameraman' sequence from VIMEO-90K-T dataset. Gaussian noise of std  $\sigma = 5$  is added to the lower-resolution input frames. (a) High-resolution noise-free ground truth frame. (b) Patch from higher resolution frame generated by BiCubic method. (c) Patch from higher resolution generated by BasicVSR++ [5]. (d) Patch from higher resolution frame generated by EDVR [49] method. (e) Patch from the higher resolution by Ours VDP method. (f) Patch from higher resolution ground truth frame. Note the noise suppression and details of the camera in the patch generated by our method compared to baselines.
Table 3: Quantitative video super-resolution results: For the evaluation of our method, we calculate the Avg PSNR and SSIM scores between the generated super-resolution video and ground truth video for all baselines along with our method. The score is calculated on the standard VIMEO-90K-T dataset. Best results in the table are shown in blue (bold). Please note we use pretrained models for baselines provided by the authors. The baselines marked with * denote a data-driven baseline.
(b) Bicubic
(c) BasicVSR++
(d) EDVR
(f) GT
Metrics	Noisy Frames	Noisy Intensity	BiCubic	ToFlow [54]*	EDVR [49]*	BasicVSR++ [5]*	Ours
PSNR	X	-	31.12	33.20	35.79	35.95	35.70
SSIM	X	-	0.870	0.920	0.937	0.940	0.936
PSNR	✓	σ = 5	29.90	30.20	32.02	31.58	33.87
SSIM	✓	σ = 5	0.796	0.712	0.758	0.720	0.878
For this task, we utilize the latent embeddings to perform the frame interpolation. First, we optimize our VDP model parameters using the loss function given by Eqn. 4. Now, to generate intermediate frames, we linearly interpolate the latent embeddings of the video using the equation.
$$
X _ {\text {i n t e r m e d i a t e}} = f \left(\alpha z _ {t + 1} + (1 - \alpha) z _ {t}\right) \quad \forall \alpha \in (0, 1). \tag {5}
$$
Here,  $X_{\text{intermediate}}$  is the interpolated frame between the frames  $X_{t}$  and  $X_{t+1}$ . For example, to perform a 4x frame interpolation, we select  $\alpha = [0.25, 0.5, 0.75]$  in the Eqn. 5. To evaluate the performance of our method, we conduct experiments on the UCF101 (triplet) dataset [2, 40] under two conditions: 1) ideal triplet input sequence, 2) triplet input sequence containing one noisy frame. We make a qualitative comparison between our method and the baselines depicted in Fig. 4. It can be observed from Fig. 4 that our method is more robust to noisy input and outputs the crisp intermediate frames without noise, unlike baseline methods DAIN [2], SoftSplat [23] and RIFE [14].
Table. 6 represents the quantitative evaluation of our approach on the UCF-101 dataset. It can be observed from the table that our method outperforms the existing baselines on PSNR and SSIM metrics in the video frame interpolation task, further validating the effectiveness and suitability of our approach for this task.
6 Video Super Resolution
Given our ever-increasing screen resolutions, there is a great demand for higher-resolution video content. In this task, we are given a lower-resolution video sequence as input. We are tasked to change this input to a higher-resolution video sequence.
For this task, we modify the loss objective given by Eqn. 4 as follows,
$$
\arg \min  _ {f, g} \sum_ {t = 0} ^ {T - 1} \lambda_ {\mathrm {r e c}} \mathcal {L} _ {\mathrm {r e c}} \left(X _ {t + 1}, d _ {\mathrm {s c}} (f (g (z _ {t})))\right) + \lambda_ {\mathrm {s p l}} \mathcal {L} _ {\mathrm {s p l}} \left(X _ {t + 1}, d _ {\mathrm {s c}} (f (g (z _ {t})))\right) + \lambda_ {\mathrm {v a r}} \mathcal {L} _ {\mathrm {v a r}} \left(f (g (z _ {t}))\right). \tag {6}
$$
Figure 6: Video Object Removal. In the above figure, we describe a video object removal task. The input in the figure represents the original video sequence frame without any alterations. The InterVI [56], FGVC [11], and LongVI [24] in the figure represent the frames generated by removing the object "car" from the video sequence. The rightmost frame depicts the removal of the object from the video frame by our method. It can be observed that our method do not create hallucinations.
Table 4: Quantitative Object Removal results: We compare the average PSNR and SSIM scores of our method with the existing baselines on the DAVIS dataset. The best performance is denoted by the score in blue (bold). Please note we use pretrained models for baselines provided by the authors. The baselines marked with * denote a data-driven baseline.
Metrics	DIP [46]	DFGVI [53]*	FGVC [11]*	InterVI [56]	LongVI [24]	Ours
PSNR	26.03	27.57	31.92	30.86	31.80	32.06
SSIM	0.921	0.9289	0.9499	0.9350	0.9457	0.9512
The modified objective for the video super-resolution task is given by Eqn. 6. Here, function  $d_{\mathrm{sc}}(\cdot)$  denotes the downsampling function. For example, to perform  $4\mathrm{x}$  super-resolution, we would define downsampling function as  $d_{\mathrm{sc}}: \mathbb{R}^{c \times h \times w} \to \mathbb{R}^{c \times (h/4) \times (w/4)}$ .
We evaluate our method for the video super-resolution task on the VIMEO90-K-T dataset and contrast our model against the state-of-the-art baseline methods. We evaluate our method on two different settings: 1) When sharp lower-resolution frames are provided, 2) When noisy lower-resolution frames are provided. We perform a  $4 \times$  super resolution of frames in the video sequences. We make a qualitative comparison between our method and the baselines for the setting when noisy frames are given as input frames, depicted in Fig. 7. It can be observed from Fig. 7 that our method outputs better and noise-free higher resolution frames with more details in comparison to the baseline methods. Table. 5 represents the quantitative evaluation of our approach. It can be observed from the table that there is a significant drop in the performance of baselines to our method when a slight amount of Gaussian noise is added to the input frames. We demonstrate empirically that our method is much more robust to noise as compared to other baseline methods.
7 Video Object Removal
Unwanted items in a video can grab the viewers' attention from the main content. In this task, we remove such objects from the original video stream. For this task, we assume a mask  $M = \{m_t\}_{1}^{T}\forall m_t\in \{0,1\}^{c\times w\times h}$  is provided for each frame to remove an object from the video sequence. To perform the removal of an object, we modify the objective function given by Eqn. 4 as follows,
$$
\underset {f, g} {\arg \min } \sum_ {t = 1} ^ {T} \lambda_ {\operatorname {r e c}} \mathcal {L} _ {\operatorname {r e c}} \left(m _ {t} X _ {t}, m _ {t} f \left(g \left(z _ {t - 1}\right)\right)\right) + \lambda_ {\operatorname {s p l}} \mathcal {L} _ {\operatorname {s p l}} \left(m _ {t} X _ {t}, m _ {t} f \left(g \left(z _ {t - 1}\right)\right)\right) + \lambda_ {\operatorname {v a r}} \mathcal {L} _ {\operatorname {v a r}} \left(f \left(g \left(z _ {t - 1}\right)\right)\right). \tag {7}
$$
We evaluate our method for the video object removal task on the DAVIS dataset. A qualitative comparison between our method and the baselines can be seen in Fig. 6. It can be observed from Fig. 6 that our method performs better than the existing methods in removing the object from the video without creating any artifacts in the processed videos.
Table. 4 represents the quantitative evaluation of our approach. We perform the evaluation based on the stationary mask setting. In this setting, a stationary mask is applied to all frames of the video sequence. This setting helps us evaluate the model's ability to generate temporally coherent frames even if some information in the frames is consistently missing.
One severe limitation of our approach is that it is an offline approach, i.e., it can not be used in real-time to perform video enhancements. One way to rectify this limitation is to work on reducing the parameters for the frame decoder network. One suggestion is modifying the residual skip connection from concatenation-based connections to additive connections. This would drastically reduce the number of parameters to be trained with minimal effects on the efficacy of the model.
The second limitation of our approach is that it is a recurrent approach and can not be scaled using multiple GPUs. This issue can be alleviated using a temporal attention-based approach that can have long-term memory and is scalable across multiple GPUs.
Our approach, in its current stage, requires no data and is only able to perform low-level vision tasks. However, to perform high-level vision tasks such as performing video segmentation or video manipulation, we conjecture that external knowledge would be required, like a collection of external datasets or a pretrained model on a such external dataset to aid our VDP method.
We conclude in this work that a video-specific approach is a way to go for a considerable number of video processing tasks. Our VDP method not only performed better than the task-specific approaches but also it is not hamstrung by the limited availability of either the computing power or the training data. We demonstrate that the introduction of a spatial pyramid loss enhances the robustness of our proposed VDP method. Leveraging this loss, we were able to achieve state-of-the-art results in all enhancement tasks, even in the presence of noise in the input frames. One limitation of our method is the long processing time compared to other methods at the test time inference. Nevertheless, it is important to note that we do not require training as the other baseline methods (requires multiple GPU hrs if not days). In the future, we will focus on improving efficiency to shorten the processing time for increased practical utility. One great advantage of our approach is removing the data collection requirement for such processing tasks. We hope that this step towards a data-free approach will reduce the massive data mining across the internet and provide better data privacy measures.
Our approach offers a significant advantage by eliminating the need for data collection in processing tasks. This shift towards a data-free approach has the potential to mitigate the extensive data mining activities conducted across the internet, thereby addressing concerns related to data privacy. By reducing the reliance on data collection, our approach promotes more ethical and responsible use of technology.
However, it is crucial to acknowledge that our approach also has some limitations. We believe that addressing these challenges through ongoing research and development will contribute to the continuous improvement and refinement of data-free approaches, thereby enhancing their broader impact.
11 Acknowledgement
This project was partially funded by the DARPA SAIL-ON (W911NF2020009) program and an independent grant from Meta AI.
[1] P. Arias and J.-M. Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60(1):70-93, 2018. 1
[2] W. Bao, W.-S. Lai, C. Ma, X. Zhang, Z. Gao, and M.-H. Yang. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 1, 3, 7, 8, 15, 18
Model Overview
Video Denoising
Video Frame Interpolation
Video Super Resolution
Video Object Removal
A Comparison with cascaded models 14
B Ablation study 15
C Processing time 16
D Optimization Details 17
E Additional Details - FDNet 18
F Dataset Descriptions 18
A Comparison with cascaded models
Existing video enhancement approaches rely on clean and sharp input frames to perform the task at hand. However, this is not always the case depending on the lighting condition of scene capture or compressing a video stream to send it over the internet. All of these introduce noise in the clean video signal. Applying video enhancement approaches in such conditions results in sub-optimal enhancements, as demonstrated in the main paper. One other way of dealing with such conditions is the use of a two-stage cascaded approach. For example, we have to provide super-resolute frames for a noisy video sequence. We can get the solution by first denoising the frames and then processing these with super-resolution approaches. However, this also results in error propagation from one stage to another and results in artifacts.
To evaluate the aforementioned reasoning, we perform a two-stage cascaded approach for video super-resolution and frame interpolation tasks. In this cascaded approach, we apply the frame denoiser first and then apply the VSR or VFI model. For video denoiser we use the FastDVDnet [43]. Table. 5 and 6 describe the quantitative evaluation of baselines(cascaded approach) with our VDP model for super-resolution and frame interpolation tasks. It can be observed that there is a deterioration in results for the super-resolution task for the cascaded approach than directly applying the baseline. This is primarily due to the fact that when the denoiser is applied in the first stage, it washes away important details from the lower-resolution frames. When these frames are passed through the video super-resolution baselines, the error from the first stage is propagated further. A qualitative visualization of this is depicted in Fig. 7. However, for the frame interpolation task, the case is reversed, and cascaded baseline approaches yield better results than only baseline approaches. It can be seen from Table. 6 that the cascaded baselines perform better than only baseline approaches but still perform worse than our VDP approach.
(a) Bicubic (Noisy)
(b) Bicubic (Denoised)
(c) BasicVSR++
Figure 7: Video Super Resolution: Comparison of results of the 'Cameraman' sequence from VIMEO-90K-T dataset. Gaussian noise of std  $\sigma = 5$  is added to the lower-resolution input frames. (a) Bicubic extrapolation of noisy low-resolution frame (b) Bicubic extrapolation of denoised low-resolution frame. (c) Patch from higher resolution generated by BasicVSR++ [5] over the denoised low-resolution frame. (d) Patch from higher resolution frame generated by EDVR [49] over the denoised low-resolution frame. (e) Patch from the higher resolution by Ours VDP method. (f) Patch from higher resolution ground truth frame. We use standard video denoiser FastDVDNet [43] to denoise the input low-resolution video frames. Based on the figure, it appears that the denoiser used in the first stage of the model removes vital information from the lower-resolution frames, which leads to errors in the second stage. Our VDP model, on the other hand, preserves more details compared to the baselines.
Table 5: Quantitative cascaded video super-resolution  $(4\times)$  results: For the evaluation of our method, we calculate the Avg PSNR and SSIM scores between the generated super-resolution video and ground truth video for all baselines along with our method. The score is calculated on the standard VIMEO-90K-T dataset. Best results in the table are shown in blue (bold). Please note we use VSR baselines by cascading it with the video denoiser model FastDVDnet [43]. The baselines marked with  $*$  denote a data-driven baseline.
Metrics	Noisy Frames	Noisy Intensity	BiCubic	EDVR [49]*	BasicVSR++ [5]*	Ours
PSNR	✓	σ = 5	27.78	31.56	31.03	33.87
SSIM	✓	σ = 5	0.806	0.857	0.853	0.878
Also, we have shown in the main paper that denoisers like FastDVDnet [43] are also prone to variations in noise distribution. This would hurt the efficacy of the cascaded models if the noise is coming from a distribution other than the Gaussian distribution.
Table 6: Quantitative cascaded video frame interpolation results: For the evaluation of our method, we calculate the Avg PSNR and SSIM score between the generated interpolated video frames and ground truth video. For all baselines and our method, the score is calculated on the UCF101 triplet dataset. Please note we use VFI baselines by cascading it with the video denoiser model FastDVDnet [43]. The color blue (bold) denotes the best performance. The baselines marked with * denote a data-driven baseline.
Metrics	# Noisy Frames	Noise level	ToFlow [54]*	DAIN [2]*	SoftSplat [23]*	RIFE [14]*	Ours
PSNR	2ndFrame	σ = 15	33.31	33.51	34.20	33.95	34.92
SSIM	2ndFrame	σ = 15	0.882	0.891	0.898	0.892	0.918
B Ablation study
We performed ablation experiments to study the effect of losses for our video dynamics prior. We report the quantitative results for all the enhancement tasks in Table 7. It can be observed from the denoising section of the table that spatial pyramid loss  $\mathcal{L}_{\mathrm{spl}}$  (given by Eqn. 2) is very important to provide robustness to both the spatial and temporal domains. Additionally, it can be observed that variation loss  $\mathcal{L}_{\mathrm{var}}$  is very important for tasks like super-resolution, frame interpolation, and object removal. In Fig. 8-11, we show a qualitative comparison between VDP model optimized with different subsets of losses.
(b)  $\mathcal{L}_{rec}$
(c)  $\mathcal{L}_{rec} + \mathcal{L}_{spl}$
(d)  $\mathcal{L}_{rec} + \mathcal{L}_{var}$
(e)  $\mathcal{L}_{rec} + \mathcal{L}_{spl} + \mathcal{L}_{var}$
(f) Ground Truth
Figure 8: Video Denoising (Ablation): Frame from a video in the DAVIS dataset denoised using loss functions. (a) Frame corrupted with additive poisson noise of intensity 25 (relative to intensity range [0-255]). Figure (b) depicts the denoised frames generated by only  $\mathcal{L}_{\mathrm{rec}}$ . Figure (c) depicts the denoised frame generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}}$ . Figure (d) depicts the denoised frame generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{var}}$ . Figure (e) depicts the denoised frames generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{var}}$ . Figure (f) shows the clean ground truth frames.
Video denoising: It can be observed from Fig. 8 that only using the reconstruction loss  $\mathcal{L}_{\mathrm{rec}}$  results in very suboptimal denoising. Optimizing for a high number of epochs using only  $\mathcal{L}_{\mathrm{rec}}$  results in the total reconstruction of the noisy signal. Adding the variation loss  $\mathcal{L}_{\mathrm{var}}$  to  $\mathcal{L}_{\mathrm{rec}}$  results in smoothening of the video frame as can be observed from the Fig. 8. However, adding the spatial pyramid loss  $\mathcal{L}_{\mathrm{spl}}$  to  $\mathcal{L}_{\mathrm{rec}}$  results in a very strong denoiser. This can also be observed in the quantitative evaluation given by Table. 7.
Video Super Resolution: It can be observed from Fig. 9 that only using the reconstruction loss  $\mathcal{L}_{\mathrm{rec}}$  results in suboptimal quality of super resolute frames with edginess in the texture. The addition of the spatial pyramid loss  $\mathcal{L}_{\mathrm{spl}}$  to  $\mathcal{L}_{\mathrm{rec}}$  results in better texture in the high-resolution frames, yet some edgyness in the texture remains. However, the addition of the variation loss  $\mathcal{L}_{\mathrm{var}}$  to  $\mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{rec}}$  results in smoothening of the texture while retaining the edges. This can also be observed in the quantitative evaluation given by Table. 7.
Video Frame Interpolation: It can be observed from Fig. 10 that only using the reconstruction loss  $\mathcal{L}_{\mathrm{rec}}$  results in suboptimal quality of interpolated frames with rough textures and missing details. The addition of the spatial pyramid loss  $\mathcal{L}_{\mathrm{spl}}$  to  $\mathcal{L}_{\mathrm{rec}}$  results in better texture and increased details in the interpolated frames however, some edginess in the texture remains. However, the addition of the variation loss  $\mathcal{L}_{\mathrm{var}}$  to  $\mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{rec}}$  results in smoothening of the texture while retaining the details. This can also be observed in the quantitative evaluation given by Table. 7.
Video Object Removal: It can be observed from Fig. 11 that only using the reconstruction loss  $\mathcal{L}_{\mathrm{rec}}$  results in suboptimal quality of interpolated region in the frames with rough textures and color jitter artifacts. The addition of the spatial pyramid loss  $\mathcal{L}_{\mathrm{spl}}$  to  $\mathcal{L}_{\mathrm{rec}}$  results in better texture and reduced jittering artifacts in the interpolated region in the frames. However, the addition of the variation loss  $\mathcal{L}_{\mathrm{var}}$  to  $\mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{rec}}$  results in smoothening of the texture and merging of the inpainted region with the background. This can also be observed in the quantitative evaluation given by Table. 7.
C Processing time
We would like to point out here that if we take cumulative training and inference time, then our baseline would come out to be a better bargain in many scenarios, as depicted by Table. 8. To put it in perspective, a data-heavy baseline requires an ample amount of time to train, even when provided with perfect hyperparameters, making exploring different architectures quite expensive. In addition, the training dataset is limited in terms of representing real-world scenarios; this would further increase
(a) Bicubic