## 1 Introduction
Recent research [19; 20; 21; 22] addresses the above challenges with more efficient methods consisting of alignment and fusion. However, these methods still suffer from poor performance when there is a lot of motion or severe blur in the video. Based on the above discussions, there are still significant challenges in the existing techniques of video inpainting and video super resolution. Moreover, there is a significant disparity in the design of algorithmic models in these two domains, leading to the lack of a universal model. As a result, we introduce a generalized Video-Restoration-Net (VRN), supported by a non-local operation, to perform both tasks successively. Firstly, due to the lack of suitable datasets, we propose an automatic algorithm to synthesize full video datasets for super-resolution reconstruction, while another incomplete-complete dataset is constructed using large and varied hole-mask sequences simulating different real-world use cases. Specifically, some frames of the incomplete-complete datasets are shown in Figure 1. From top to bottom, we show examples of the collected videos, masks, and a mask sequence that we generated from a single mask. Based on the two synthetic datasets, we design a WGAN-style framework [23] to train a non-local deep generative network for inpainting and SR tasks. During inpainting tasks, we use an encoder-decoder model in the generator to extract the feature of each frame in the video and to restore frames based on the predicted feature, respectively. Between the encoder and decoder, we stack the feature maps of the frames and introduce a non-local network module [24] that models long-range, spatio-temporal dependencies to fill in the missing part of the inpainting task, while the frames should be input in an unstacked situation to recombine the non-local information from different locations of a given frame in the SR task. The recombined feature maps are then unstacked and inputted frame-by-frame to the decoder to generate a coarse video. After the encoder-nonlocal-decoder construction, we design a network to refine each frame separately. On the other hand, we employ both global and local discriminators for frame inpainting [25]. Meanwhile, we perform the video SR task by adopting another VRN based on the full video datasets frame-wisely, assisting with a similar training course adopted in the video inpainting task. Both \(\ell 1\) reconstruction loss and adversarial loss are used in supervision during training. Overall, we first present experiments on the complete-incomplete datasets we synthesize to demonstrate the effectiveness of our proposed method in video inpainting. Moreover, we adopt the full video datasets we synthesize as training datasets for SR reconstruction to explore the performance of our proposed method and validate it on public datasets, proving the universality of our Video-Restoration-Net. Our contributions can be summarized as follows:  
* We present a deep generative non-local network for the task of video inpainting and frame-by-frame video SR reconstruction as an effective and generalized baseline. The proposed model is able to deal with various shapes of masks with divergent motion while reconstructing the generated low-resolution videos frame by frame. * We propose an automatic algorithm to synthesize incomplete videos from complete ones, motivated by real-use cases of video inpainting. * We present experiments through inpainting and SR results supported by ablation studies compared with several deep-learning-based baselines and show that our method based on a non-local module produces higher quality results in both video inpainting and frame-by-frame video SR tasks.