SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gaurav Shrivastava
gauravsh@umd.edu
Ser-Nam Lim
Abhinav Shrivastava
abhinav@cs.umd.edu
In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus. Our proposed approach directly learns the weights of neural modules by optimizing over the corrupted test sequence, leveraging the spatio-temporal coherence and internal statistics of videos. Furthermore, we introduce a novel spatial pyramid loss that leverages the property of spatio-temporal patch recurrence in a video across the different scales of the video. This loss enhances robustness to unstructured noise in both the spatial and temporal domains. This further results in our framework being highly robust to degradation in input frames and yields state-of-the-art results on downstream tasks such as denoising, object removal, and frame interpolation. To validate the effectiveness of our approach, we conduct qualitative and quantitative evaluations on standard video datasets such as DAVIS, UCF-101, and VIMEO90K-T. $^{1}$
Recently, video enhancement [54, 42, 43, 1, 12, 18, 49, 32] and editing [56, 11, 24, 50] tasks have attracted much attention in the computer vision community. Most of these tasks are ill-posed problems, i.e., they do not have a unique solution. For example, enhancement tasks such as video frame interpolation can have infinitely many plausible solutions. Therefore, the aim is to find a visually realistic solution coherent in both the space and time domain for such tasks.
Priors play a critical role in finding reasonable solutions to these ill-posed problems by learning spatio-temporal constraints. In recent years, deep learning has emerged as the most promising technique [54, 13, 39, 38, 4, 2, 43, 49, 22] for leveraging data to create priors that enforce these spatio-temporal constraints. However, the generalization of these data-driven priors mainly relies on data distribution at training time.
In the current era of short video platforms such as TikTok, Instagram reels, YouTube shorts, and Snap, millions of new artistic elements (e.g., new filters, trends, etc.) are introduced daily. This trend causes substantial variations between the train and the test sets, which results in diminished effectiveness of the learned prior. Moreover, frequent fine-tuning or frequent training to improve the efficacy of the prior significantly hamper the scalability of such models.
1 Navigate to the webpage for video results.
Furthermore, the existing video enhancement approaches assume clean and sharp input frames, which may not always be the case due to factors like bad lighting conditions during the scene capture or video compression for internet transmission. These factors introduce noise in the clean video signal, and feeding such noisy input frames to video enhancement approaches results in suboptimal enhancements.
To address these challenges, we propose a novel Video Dynamics Prior (VDP) that leverages the internal statistics of a query video sequence to perform video-specific enhancement tasks. By utilizing a video's internal statistics, our approach eliminates the need for curated training data. Thus, removing the constraints with regard to encountering test time variations. Additionally, we introduce a spatial pyramid loss that enhances robustness in the temporal domain, enabling our method to handle noisy input frames encountered at test time.
VDP utilizes two fundamental properties of a video sequence: (1) Spatio-temporal patch (ST patch) recurrence and (2) Spatio-temporal consistency. ST patch recurrence is a fundamental finding [34] that reveals significant recurrence of small ST patches (e.g.,  $5 \times 5 \times 3$  or  $7 \times 7 \times 3$ ) throughout a natural video sequence. To leverage this strong prior, we employ an LSTM-Convolution-based architecture, which is detailed in Sec. 3. Moreover, to leverage the ST patch recurrence property across scales in a video, we introduced a novel spatial pyramid loss. In Sec. 3.5, we demonstrate that our novel spatial pyramid loss helps the VDP model to be more robust to unstructured noise in both spatial and temporal domains.
Further, our approach considers the next frame prediction task to initialize our recurrent architecture. The task aims to auto-regressively predict the next frame, taking the current frame and previous frames information as context. We leveraged architecture from such a task as it promotes the spatio-temporal consistency in the processed video frames and models the complex temporal dynamics required for various downstream tasks. In our approach, we first formulate a next-frame prediction model with unknown parameters. Then depending on the downstream task, we maximize the observation likelihood with slight modification in the objective function.
This paper presents three key contributions to the field of video processing. Firstly, we introduce a novel inference time optimization technique that eliminates the need for training data collection to learn neural module weights for performing tasks. Secondly, we propose a novel spatial pyramid loss and demonstrate its effectiveness in providing robustness to spatial and temporal noise in videos. Lastly, we showcase the superior performance of our approach in multiple downstream video processing tasks, including video denoising, video frame interpolation, and video object removal. Our method outperforms current baselines and exhibits robustness in case noisy frames are provided as input for enhancement.
The rest of the paper is organized as follows. In Sec. 2, we briefly introduce the background work done in the area of video enhancement tasks. In Sec. 3, we discuss our methodology and key components of our architecture. Sec. 4-7, we briefly describe the video processing tasks and accompanying modifications to the objective function for dealing with different tasks at hand. Finally, we conclude our work in Sec. 9.
Over the past decades, many research efforts have been made to increase the efficacy of video enhancements. Traditional methods like [33, 16, 51] utilized affine-estimation based motion approach to perform video enhancement tasks. However, these methods were very limited in their capability of learning a good representation of videos to generalize well on various scenes. All of this changed with the recent developments in deep learning techniques that considerably improve the representation of videos and deliver high performance gains on video enhancement tasks.
Internal Learning. In the recent past, a considerable number of deep internal learning approaches were put forth to perform image enhancement tasks using just a single test image [37, 46, 3, 10, 28, 36, 29, 48]. Such approaches utilize patch recurrence property in a natural image as a strong prior to deal with inverse problems such as image denoising [46, 10, 21], super-resolution [37, 46, 29], and object removal [46]. However, when such internal learning approaches are directly applied to videos for enhancement tasks, it creates flickering artifacts in the processed video [55, 19]. These artifacts arise as image-based methods fail to maintain a video's temporal consistency, a critical component of a video signal. Extending the deep internal-learning based approach to videos remains
Figure 1: Model pipeline: We are given  $X_{1},\ldots ,X_{t}$  frames of a person exercising (depicted in the Fig). Here, we start by sampling  $z_0\sim \mathcal{N}(0,I)$  as an initial latent representation. Our model comprises two key modules: (i) the Frame Decoder Network (FDNet) and (ii) the Latent Frame Predictor Network (LFPNet), depicted in the figure. These modules, together form the pipeline of our VDP model, as shown in (iii). We optimize the weights of these neural modules on this specific input sequence by doing backpropagation over the loss pertaining to the task at hand.
a challenging problem. There are very few works proposed in the area that deal with videos, and all of these works are tailored towards a specific video enhancement task [57, 56]. In this paper, we extend a deep internal-learning based framework to the video space that effectively explores various video enhancement tasks such as denoising, super-resolution, frame interpolation and inpainting, utilizing a single structure of our VDP model.
Flow Guided Methods. In the image processing tasks, most of the methods utilize the 2D CNN modules. However, if we use such models in standalone settings, we often find that they are inept at processing the additional temporal dimension present in a video sequence. In order to overcome the temporal consistency problem, some researchers have opted for an optical-flow based approach to capture the motion occurring in the videos [54, 9, 42, 26, 15, 20, 12]. Essentially, they model the flow of the video sequences by doing explicit motion estimation. Then they perform warping operations (motion compensation) to generate enhanced video frames. These optical-flow based approaches have found plenty of applications in tasks such as video denoising [54, 42], video super-resolution [54, 12, 6, 5], video object removal [11, 52], video frame interpolation [54, 2, 17, 14, 23]. However, accurate motion estimation using optical flow is inherently challenging and computationally expensive. Therefore, deep neural networks such as RAFT [44] and SPyNet [27] are employed to estimate optical flow. These networks rely on lots of training data to learn to predict the optical flow. This double reliance on data by both the video enhancement approach and optical flow prediction networks can introduce dataset biases. Further, we showcase that the flowbased models for video super-resolution and frame interpolation tasks are not robust to noise or artifacts arising from capturing the scene in low-lighting or bad lighting conditions. In our work, instead of explicitly performing motion estimation and motion compensation, we utilize LSTM-Conv decoder networks represented in Fig. 1 to capture the temporal information implicitly. Furthermore, our method is able to infer context directly from the test video sequence itself. Thus, mitigating the problems arising from dataset biases. Additionally, we empirically demonstrate that our approach is stable for tasks like super-resolution and frame interpolation, even in the presence of noisy input frames sequence.
3 Model Overview
Given a video sequence as context, we perform various video-specific processing tasks on this sequence of frames. In traditional settings, the training data is used to learn good representations that capture the patterns in the video sequences. However, in our case of no data, we rely on a proper choice of a deep neural network that can capture the ST patch recurrence property of a video as an informative prior for the video processing task at hand. Our model for performing all video-specific tasks is a combination of two modules. (1) The latent frame predictor network and (2) The frame decoder network as depicted in Fig. 1.(iii).
3.1 Latent Frame predictor Network (LFPNet)
The task of the latent frame predictor network is to maintain the spatio-temporal consistency of the video. It learns the dynamics of a video sequence in latent space, i.e., it models the temporal relations between the frames in latent space. We use LSTM cells as the building block for this module as they implicitly capture the temporal relation between the latent frames. This LFPNet module (is inspired
by [7, 39])  $g(\cdot): \mathbb{R}^d \to \mathbb{R}^d$ , has a fully-connected layer, followed by four LSTM layers with 1024 cells each, and a final output fully-connected layer. The output fully-connected layer takes the last hidden state from LSTM  $(\mathbf{h}_{t+1})$  and outputs  $\hat{\mathbf{z}}_{t+1}$  after  $\tanh(\cdot)$  activation.
3.2 Frame Decoder Network (FDNet)
This network maps the frame embeddings from the latent space to the image space, i.e.,  $f(\cdot): \mathbb{R}^d \to \mathbb{R}^{c \times h \times w}$ . The architecture for our VDP's frame decoder network is depicted in the Fig. 1.(i). The residual skip connections in our architecture play a crucial role in the performance of our module FDNet by addressing the vanishing gradient problem and facilitating faster convergence.
3.3 Formulation of Video Dynamics Prior
For our proposed Video Dynamics Prior or VDP prior we only rely on the two modules namely LFPNet and FDNet. We start by sampling an initial latent noise vector  $z_0 \sim \mathcal{N}(0, I)$  where  $z_0 \in \mathbb{R}^D$ . Here,  $D$  is the dimension of latent space. For our experiments, we keep the dimension  $D$  of latent space as 1024. Now, as illustrated in Fig. 1, we pass this  $z_0$  embedding through the 'LFPNet' and obtain the next frame embedding  $z_1$ . We then pass  $z_1$  through the frame decoder network to obtain the next image frame  $\hat{X}_1$ . Mathematically, we can write the expression for the forward pass to obtain the future frame as  $\hat{X}_{t+1} = f(g(z_t))$ . Where,  $f: \mathbb{R}^D \to \mathbb{R}^{c \times h \times w}$  denotes the frame decoder network (§3.2) that maps the latent space to image space i.e.  $X_{t+1} = f(z_{t+1})$ . Function  $g(\cdot)$  denotes the latent frame predictor network. This network takes the latent embedding at the current step as the input and predicts the latent embedding at the next step as the output, i.e.,  $z_{t+1} = g(z_t)$ . Leveraging future frame prediction task helps in initializing the LFPNet in the latent space.
3.4 Neural Module Weights Optimization
Since we optimize the weights of neural modules (LFPNet and FDNet) using a corrupted sequence, reconstruction loss alone is not enough to find a visually plausible enhancement of the input video. Hence, we introduce a few other regularization losses.
Reconstruction Loss: We use a combination of L1 and perceptual loss to formulate the reconstruction loss. It is given by Eqn. 1,
$$
\mathcal {L} _ {\text {r e c}} = \| X _ {t + 1} - f (g \left(z _ {t}\right)) \| + \| \phi \left(X _ {t + 1}\right) - \phi \left(f \left(g \left(z _ {t}\right)\right)\right) \|. \tag {1}
$$
Here,  $\phi (\cdot)$  in the Eqn. 1 denotes the pre-trained VGG network on ImageNet [31].  $X_{t}$ ,  $\hat{X}_t$  denotes the input and reconstructed video frames at timestep t, respectively.
Spatial Pyramid Loss: It has been well established that small ST patches of size  $5 \times 5 \times 3$  recur within and across the scales for a video. Additionally, bicubic downscaling of a video results in coarsening of the edges in the downscaled video. This results in the downscaled video to have less motion speed and less motion blur. Both of these properties are extremely useful for video enhancement tasks. Hence, to utilize such lucrative properties of a video, we introduced spatial pyramid loss. For this loss, we scale down the input image to 3 levels ( $\frac{h_w}{k^2}$  where  $k$  is 2,4,8) using a downsampler and calculate the reconstruction loss for each level. It is given by Eqn. 2,
$$
\mathcal {L} _ {\mathrm {s p l}} = \sum_ {i = 2, 4, 8} \| d _ {i} \left(X _ {t + 1}\right) - d _ {i} \left(f \left(g \left(z _ {t}\right)\right)\right) \|. \tag {2}
$$
Here,  $X_{t + 1}$  denotes the input video frames at timestep  $t + 1$ .  $d_{i}(\cdot)$  in the Eqn. 2 denotes the down sampler network and  $i$  denotes scale of down sampler. For example,  $d_{2}(\cdot):\mathbb{R}^{c\times h\times w}\to \mathbb{R}^{c\times h / 2\times w / 2}$ .
Variation Loss: This is a variant of Total variational loss introduced in the paper [30]. Variation loss smoothens the frames while preserving the edge details in the frame. It is given by Eqn. 3,
$$
\mathcal {L} _ {\text {v a r}} \left(\hat {X} _ {t}\right) = \sum_ {i = 1} ^ {H - 1} \sum_ {j = 1} ^ {W - 1} \left\| \hat {x} _ {i j} ^ {t} - \hat {x} _ {(i - 1) j} ^ {t} \right\| + \left\| \hat {x} _ {i j} ^ {t} - \hat {x} _ {i (j - 1)} ^ {t} \right\|. \tag {3}
$$
(iii)
Figure 2: (i) Video B is generated by replacing the second frame of video A. (ii) Top row depicts the video obtained during the early round of optimization of the objective given by Eq. 4 for video B. Further, the average Mutual information [41] of video(intermediate) is much closer to video A. (iii) The loss plot depicts the convergence of our VDP model for in five different settings.
Here,  $\hat{x}_{ij}^{t}$  denotes the pixel of the processed video frames ( $\hat{X}_t = f(g(z_{t - 1}))$ ) at timestep  $t$  and location  $\hat{X}_t[i,j]$ .
Final Loss: We combine all the losses together into a final loss term as follows,
$$
\mathcal {L} _ {\text {f i n a l}} = \sum_ {t = 0} ^ {T - 1} \lambda_ {\text {r e c}} \mathcal {L} _ {\text {r e c}} + \lambda_ {\text {s p l}} \mathcal {L} _ {\text {s p l}} + \lambda_ {\text {v a r}} \mathcal {L} _ {\text {v a r}}. \tag {4}
$$
3.5 Robustness of Video Dynamics Prior
In this section, we explore the internal learning properties of our model. For thorough exploration, we perform various experiments to explore the spatio-temporal patch recurrence [34] in natural videos using our model. In the past, authors of the papers [10, 46] have explored the utilization of the patch recurrence property in images by the deep learning models to perform various enhancement tasks. Following these prior works, we extend such understanding to the non-trivial spatio-temporal patch (or ST-patch) recurrence phenomenon exhibited by our model.
We design a simple experiment where we have a natural video sequence A denoted as  $V_{a} = \{X_{1}, X_{2}, X_{3}\}$  and depicted in Fig. 2.(i). We create another sequence  $V_{b}$  by replacing the frame  $X_{2}$  with a random noisy frame  $X_{\mathrm{nf}}$  resulting in  $V_{b} = \{X_{1}, X_{\mathrm{nf}}, X_{3}\}$ . As shown in Fig. 2. (ii), the noisy frame is fairly independent of frames  $X_{1}$  or  $X_{3}$  hence, we get mutual information between these frames close to zero. This replacement of frame  $X_{2}$  with  $X_{nf}$  results in higher entropy [45] of the video sequence  $V_{b}$  (larger diversity in temporal dimension  $\rightarrow$  lower ST patch recurrence  $\rightarrow$  lower self-similarity  $\rightarrow$  higher entropy of sequence) in comparison to original sequence  $V_{a}$  (lower diversity in temporal dimension  $\rightarrow$  higher ST patch recurrence  $\rightarrow$  higher self-similarity  $\rightarrow$  lower entropy of sequence). Hence when the VDP model is optimized over both the sequences  $V_{a}$  and  $V_{b}$ , it requires a longer duration for loss convergence as depicted by Fig. 2.(iii). During the optimization over the video sequence  $V_{b}$ , our model obtains the sequence resembling  $V_{a}$  (given by  $V_{\mathrm{intermediate}}$  in Fig. 2.(ii)) in early rounds of optimization, near the epochs where we observe the first dip in the convergence curve and before ending up overfitting sequence  $V_{b}$ .
We conducted such experiments on our model under five different settings. 1) Using only L1 reconstruction loss (given by Eqn. 1) on video  $V_{a}$ . 2) Using only L1 reconstruction loss on video  $V_{b}$ . 3) Using only L1 reconstruction loss and spatial pyramid loss (given by Eqn. 2) on video  $V_{b}$ . 4) Using only L1 reconstruction loss and variation loss (given by Eqn. 3) on video  $V_{b}$ . 5) Using L1 reconstruction loss, spatial pyramid loss, and variation loss on video  $V_{b}$ .
The energy plot in Fig. 2(iii) represents the trend in MSE loss for the aforementioned five settings. It can be observed that in the absence of any noise in the video signal (yellow), the model fits the input quickly. The quick convergence can be attributed to higher ST patch recurrence in video sequence A. The VDP model optimized over the video sequence  $V_{b}$  using only the L1 reconstruction loss (green) provides some impedance to the temporal noise but quickly overfits the noise after a couple of iterations. Further, the VDP model optimized under setting 3 (red) or 4 (blue) provides higher impedance to noise than setting 2 but still converges after a few hundred epochs. However, when the VDP model is optimized using setting 5 (cyan), it provides considerably higher impedance to the noise in the temporal domain.
(a) Poisson Noise  $(\lambda = 25)$
(b) FastDVDnet (PSNR = 18.8148)
(c)UDVD  $(\mathrm{PSNR} = 21.1294)$
Table 1: Quantitative video denoising results: To evaluate our method, we calculate the Avg PSNR score between the generated noise-free video and ground truth video. The  $\sigma$  denotes the standard deviation for the added Gaussian noise to the frames. While the  $\lambda$  denotes the intensity of additive Poisson noise. For all baselines and our method, the score is calculated on the DAVIS dataset [25]. Also, blue (bold) denotes the best score. Please note we use pretrained models for baselines provided by the authors. The baselines marked with  $*$  denote a data-driven baseline.
(d) M2F2 (PSNR = 18.8224)
(e) Ours (PSNR = 31.2694)
(f) Ground Truth
Figure 3: Video Denoising: Frame from a video in the DAVIS dataset denoised using different approaches. (a) Frame corrupted with additive poisson noise of intensity 25 (relative to intensity range [0-255]). Figure (b) depicts the denoised frames generated by FastDVDnet [43] method. Figure (c) depicts the denoised frame generated by UDVD [35] method. Figure (d) depicts the denoised frame generated by M2F2 [8] method. Figure (e) depicts the denoised frames generated by our method. Figure (f) shows the clean ground truth frames.
Metric	Additive Noise Type	Noise Level	M2F2 [8]*	FastDVDnet [43]*	UDVD [35]*	Ours
PSNR	Gaussian	(σ = 20)	34.89	35.77	35.12	35.74
PSNR	Gaussian	(σ = 30)	33.21	34.04	33.92	34.07
PSNR	Poisson	(λ = 25)	18.19	19.02	22.08	31.96
PSNR	Poisson	(λ = 30)	17.12	18.03	20.98	30.07
We observe a similar phenomenon when optimizing our model with a larger void filled with noisy frames. Additionally, when we optimize our VDP model over a noisy video (here, every frame in the video sequence consists of Gaussian noise), we do not see convergence at all. We attribute this behavior of our VDP model to promoting self-similarity of the st-patches over the output video sequence. Moreover, incorporating the spatial pyramid loss introduces frame matching at multiple levels of coarse scale. This encourages a more structured signal in the output and provides a higher impedance to the unstructured noise. We performed the same experiment over 250 different video sequences (randomly sampled from different datasets like Vimeo-90K, DAVIS, and UCF101) and came up with similar findings as given by Fig. 2.(iii).
4 Video Denoising
In this section, we present our VDP model utilized for the video denoising task. A noisy video can be defined as the combination of a clean video signal and additive noise. As shown in the previous section, our VDP model exhibits higher impedance to the noise signal and lower impedance to the clean video signal. Exploiting this property, we propose a denoiser for video sequences. Unlike traditional approaches, our denoiser does not require data collection, as we demonstrate that the final loss function (Eqn. 4) alone is sufficient for effective video denoising.
We optimize our model parameters on the noisy test-time video sequence using the loss function given by Eqn. 4. We then use these optimized LFPNet & FDNet modules for generating high-quality clean video frames. The new enhanced video frames for each timestamp can be obtained by  $\hat{X}_{t + 1} = f(g(z_t))$ .
The presence of noise in a video can be attributed to various factors, such as non-ideal video capture settings or lossy video streaming over the internet. While curating training datasets for baselines, it
Figure 4: Video Frame Interpolation: In order to compare our method to existing baseline methods, we conducted visual comparisons using three frames for each method. We introduced Gaussian noise  $(\sigma = 15)$  in the second frame and tasked all methods with interpolating frames between them. It can be observed that our method not only denoises the second noisy input frame but also generates denoised intermediate frames. This is in contrast to other baseline methods that interpolate noise along with the clean signal.