## 2 Related Work  
### Inpainting Methods  
DL image inpainting algorithms have shown better performance than the traditional methods, and most of them are based on GANs. In [3], an encoder-decoder network is presented to restore the missing parts and an adversarial loss is employed to improve the quality of the inpainted image. In [25], a global-local discriminator is proposed that includes two networks to evaluate the consistency of the whole inpainted image and the quality of the restored missing part, respectively. Moreover, [PERSON] et al. [25] replaced some channel-wise fully connected layers with dilated convolutions to increase the receptive fields of output neurons [26], also using a global-local discriminator and proposed a contextual attention layer that searches for matching background to reconstruct the foreground. Similarly, [PERSON] et al. [27] designed a patch-swap unit to match neural patches from the boundary to the hole. In Shift-Net [28], a special shift connection layer is added to the U-Net architecture [29] to fill in the missing regions of any shape with sharp structures and fine-detailed textures. Inspired by these image inpainting algorithms, we also adopt the  
Figure 1: The datasets we synthesize for video inpainting.  
encoder-decoder-based architecture and the global-local discriminator design in our video inpainting model.  
[PERSON] et al. [30] propose a multi-task framework that incorporates structural constraints to generate clear edges. [PERSON] et al. [31] introduce a mutual encoder-decoder network that jointly learns the CNN features that correspond to the same task. Structures and textures with various layers present a challenging task to model, particularly in creating a shared architecture that effectively complements both features. [PERSON] et al. [32] adopt bidirectional attention and auto-regressive transformers to perform image inpainting. Although they promote diversity, their completion and inference performances are limited.  
The video inpainting task was first proposed in 1998 [33], and there is still no DL solution. The author of [8] divides the video into foreground and background objects and iteratively paints the hole, frame by frame. The main disadvantages of these algorithms are their long computation times and artifacts. However, patch-based methods have demonstrated better performance. [PERSON] et al. [11] find non-local patches containing occluded pixels through an approximate nearest neighbor search and iteratively optimize the global patch-based function that searches for patches and restores the hole. However, the algorithm proposed in [11] is very slow. Recently, a modified method of [11] has been proposed [12]. It uses optical flow, which has been used in some video inpainting work [10], to preserve temporal coherence and accelerate the algorithm. As a result, it achieves better results not only in terms of speed but also in terms of inpainting quality. Recently, the authors of [34] adopted an attention-based method to pay more attention to contextual information. The authors of [35] utilize 3D convolution and attention, which often produce results with limited temporal coherence due to their restricted temporal receptive fields. However, the recent models lack universality while also having limited improvement in PSNR and SSIM index and little improvement in visual effect. Thus, our work is focused on introducing a universal model capable of restoring both images and videos. Since [11] proposes a canonical method in terms of versatility, we select this as the baseline model for the inpainting task.