SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 9: Video Super Resolution (Ablation): Comparison of results of the 'Cameraman' sequence from VIMEO-90K-T dataset. (a) Bicubic extrapolation of low-resolution frame. (b) Patch from higher resolution generated by utilizing only the reconstruction loss given by Eqn. 1. (c) Patch from higher resolution frame generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}}$  (d) Patch from the higher resolution frame generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{var}}$ . (e) Patch from the higher resolution frame generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{var}}$  (f) Patch from higher resolution ground truth frame.
Ground Truth frame
Figure 10: Video Frame Interpolation (Ablation): In order to compare our method with different loss functions, we conducted a visual comparison. (a) Patch generated by utilizing only the reconstruction loss given by Eqn. 1. (b) Patch generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}}$  (c) Patch generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{var}}$ . (d) Patch generated by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{var}}$ .
(a)  $\mathcal{L}_{rec}$
(b)  $\mathcal{L}_{rec} + \mathcal{L}_{spl}$
(c)  $\mathcal{L}_{rec} + \mathcal{L}_{var}$
(d)  $\mathcal{L}_{rec} + \mathcal{L}_{spl} + \mathcal{L}_{var}$
the spending time and effort in fine-tuning the network every time for a new setting. Please note that the inference time for our method has been calibrated on one Nvidia A6000 GPU.
D Optimization Details
To optimize our model, we use a single Nvidia A6000 GPU with 48G memory to process a single video at a time with 15-frame sequences of size  $448 \times 256$ . We optimize the modules LFPNet and FDNet weights using the entire test sequence with the Adam optimizer at a learning rate in the range of [0.0002, 0.002].
Video Denoising: For getting a good performance on the denoising task, we utilize the following weights for the different losses;  $\lambda_{\mathrm{rec}} = 1$ ,  $\lambda_{\mathrm{spl}} = 0.0001$  and  $\lambda_{\mathrm{var}} = 0.0001$ . We observed that the performance of our VDP denoiser plateaued after 3600 epochs.
Video Frame Interpolation: For getting a good performance on the frame interpolation task, we utilize the following weights for the different losses;  $\lambda_{\mathrm{rec}} = 1$ ,  $\lambda_{\mathrm{spl}} = 0.0001$  and  $\lambda_{\mathrm{var}} = 0.0001$ . We observed that the performance of our VDP frame interpolation model plateaued after 1800 epochs.
Table 7: Quantitative ablations results: To evaluate our method, we calculate the Avg PSNR score between the generated noise-free video and ground truth video. blue (bold) denotes the best score.
Task	Dataset	Metric	Additive Noise Type	Noise Level	\( {\mathcal{L}}_{\text{rec }} \)	\( {\mathcal{L}}_{\text{rec }} + {\mathcal{L}}_{\text{var }} \)	\( {\mathcal{L}}_{\text{rec }} + {\mathcal{L}}_{\text{spl }} \)	\( {\mathcal{L}}_{\text{rec }} + {\mathcal{L}}_{\text{var }} + {\mathcal{L}}_{\text{spl }} \)
Denoising	DAVIS	PSNR	Poisson	\( (\lambda = 25) \)	20.58	24.63	30.14	31.96
PSNR	Poisson	\( (\lambda = 30) \)	20.11	23.82	29.81	30.07
Super-Resolution	Vimeo-90KT	PSNR	✘	✘	31.23	33.86	32.18	35.70
SSIM	✘	✘	0.890	0.924	0.915	0.936
Frame-interpolation	UCF-101	PSNR	✘	✘	33.29	34.13	33.88	35.41
SSIM	✘	✘	0.943	0.945	0.963	0.970
Object Removal	DAVIS	PSNR	✘	✘	29.31	31.78	30.12	32.06
SSIM	✘	✘	0.9125	0.9451	0.9352	0.9512
(a)  $\mathcal{L}_{rec}$
Figure 11: Video Object Removal (Ablation). In the above figure, we describe a video object removal task. The GT in the figure represents the original video sequence frame without any alterations. (a) Patch generated by utilizing only the reconstruction loss given by Eqn. 1. (b) Patch generated utilizing the loss given by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}}$  (c) Patch generated by utilizing the loss given by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{var}}$ . (d) Patch generated by utilizing the loss given by  $\mathcal{L}_{\mathrm{rec}} + \mathcal{L}_{\mathrm{spl}} + \mathcal{L}_{\mathrm{var}}$ .
Table 8: Training and inference time comparison with the best baselines. Note that the inference time in the table is given as the time required to process per frame of a video.
Task	Dataset size	Resolution	Ours (Train + Infer)	Best Baseline (Train + Infer)
Super-Resolution	38,990 (7 fr/v)	112 × 64 → 448 × 256	0 + 15s	300 hrs + 1s
Denoise	50 seq (3450 fr)	448 × 256	0 + 12s	200 hrs + 0.1s
Object Removal	50 seq (3450 fr)	448 × 256	0 + 8.5s	160 hrs + 8.3s
Frame Interpolation	3,782 triplets	448 × 256	0 + 10s	120hrs + 0.08s
Video Super-Resolution : For getting a good performance on the VSR task, we utilize the following weights for the different losses;  $\lambda_{\mathrm{rec}} = 1$ ,  $\lambda_{\mathrm{spl}} = 0.01$  and  $\lambda_{\mathrm{var}} = 0.0001$ . We observed that the performance of our VDP SR model plateaued after 4200 epochs.
Video Object removal: For getting a good performance on the object removal task, we utilize the following weights for the different losses;  $\lambda_{\mathrm{rec}} = 1$ ,  $\lambda_{\mathrm{spl}} = 0.01$  and  $\lambda_{\mathrm{var}} = 0.0001$ . We observed that the performance of our VDP object removal model plateaued after 1800 epochs.
Additionally, we conducted our experiments using multiple random initializations of FDNet and LFPNet modules and observed consistent quantitative results across all runs.
E Additional Details - FDNet
Each convolution layer is followed by batch normalization and a leaky rectified linear unit (LeakyReLU; negative slope  $= 0.2$ ), except for the last layer of the decoder. In the last layer of the decoder, we use the sigmoid activation function. We utilize Pytorch for our implementation.
F Dataset Descriptions
Vimeo-90K-T [54] This dataset contains 7824 short clips downloaded from 'vimeo.com'. Each clip only contains 7 frames per clip. We used this dataset for a video super-resolution task. The lower resolution input frames have a size of  $3 \times 112 \times 64$  while the higher resolution frames have a size of  $3 \times 448 \times 256$ .
DAVIS [25] This dataset contains 50 curated video clips, and each clip contains a unique object in the clip. The dataset has masking annotation present for this object for each clip. Each frame in each clip contains 1 mask annotation. There is 3450 total number of frames in this dataset. We used this dataset for video denoising and object removal task. We resized the resolution of each clip for the aforementioned tasks. The resolution of input frames for the task was of the size  $3 \times 448 \times 256$ .
UCF [40] The UCF101 dataset contains videos with a large variety of human actions. We utilized this dataset for the video frame interpolation task. We used this dataset under standard settings put forth by paper [2]. There are 379 triplets with a resolution of  $256 \times 256$  pixels.
Table 9: We include LPIPS and FVD [47] metrics for completeness of quantitative evaluation of enhancement tasks (video denoising, object removal, and video interpolation). FVD is a deep neural network based metric that evaluates both the spatial and temporal quality of the processed video with the ground truth video. FVD requires minimum of 16 frames in a sequence for evaluation. Hence, we were unable to evaluate FVD for task like VFI and VSR on UCF101 triplet and Vimeo90KT datasets.
Task	Method	Dataset	Noisy Frames	Noise Intensity	FVD↓	LPIPS↓	PSNR↑	SSIM↑
Denoise (Poisson)	FastDVDnet	DAVIS	✓	λ = 25	1743	0.65	19.02	-
Ours	54	0.0056	31.96	-
Object Removal	FGVC	DAVIS	X	-	580	0.0556	31.92	0.9499
Ours	445	0.0408	32.06	0.9512
Inter-polation	RIFE	UCF101	2ndFrame	σ = 15	-	0.412	20.22	0.512
SoftSplat-LF	-	0.624	18.25	0.401
Ours	-	0.102	34.92	0.918
VSR	EDVR	Vimeo90KT	✓	σ = 5	-	0.5051	32.02	0.758
BasicVSR++	-	0.5263	31.58	0.720
Ours	0.181	33.87	0.878