## 3 Materials and Methods
### The WGAN Style Training Loss  
Since the original GAN is always faced with the problem of disappearing gradients, the WGAN is proposed as a solution, which uses the Wasserstein distance to measure the gap between the real data distribution and the fake data distribution. The WGAN can be described as:  
\[\min_{G}\max_{D}\mathbb{E}_{x\sim\mathbb{P}_{r}}[D(x)]-\mathbb{E}_{G(z)\sim \mathbb{P}_{g}}[D(G(z))] \tag{2}\]  
where \(D\) and \(G\) represent the discriminator and generator, respectively. \(z\) is random noise in the WGAN, but it is the incomplete video with its mask sequence in our work. As mentioned above, the same model is adopted in both video inpainting and SR tasks and trained in two separate courses. The training loss of the video inpainting and SR tasks can be described as follows. During the video inpainting task, the discriminator we design contains two parts: the global one evaluates the consistency of the restored missing part with the entire frame, and the local one judges the quality of the restored missing part. For the global discriminator, we put the restored missing region back into the incomplete video; it can be described as:  
\[V_{g}=V_{r}\odot M+V_{i}\odot(1-M) \tag{3}\]  
where \(M\) is the mask sequence, whose pixels are 1 in the missing region and 0 in the rest of the video. We use \(V_{g}\) as the input of the global discriminator, and the label for complete video is \(V_{c}\) in both tasks. So the loss of global discriminator \(D_{g}\) is  
\[l_{D_{g}}=\mathbb{E}_{V_{r}\sim\mathbb{P}_{g}}[D_{g}(V_{g})]-\mathbb{E}_{V_{c} \sim\mathbb{P}_{r}}[D_{g}(V_{c})] \tag{4}\]  
For the local WGAN, we only focus on the quality of the restored missing-part frame. Therefore, the loss of local discriminator, \(D_{l}\), can be written as:  
\[l_{D_{l}}=\mathbb{E}_{V_{r}\sim\mathbb{P}_{g}}[D_{l}(V_{r}\odot M)]-\mathbb{E} _{V_{c}\sim\mathbb{P}_{r}}[D_{l}(V_{c}\odot M)] \tag{5}\]  
On the other hand, we combine the \(L1\) loss with the loss of WGAN to train our generator. We tried 3 forms of L1 loss. The first one is  
\[l_{L1}=\|V_{r}-V_{c}\|\odot M \tag{6}\]where we only want to compute the gap in the mask. However, this loss is not able to guide the network in learning how to restore the video. The output is chaotic and the loss does not decrease during training. Whereafter, we realize that it is necessary to compute the loss of the full video, so we simply compute the Manhattan distance between the inpainting video and the ground truth, which can be described as:  
\[l_{L1}=\|V_{r}-V_{c}\| \tag{7}\]  
Nevertheless, we finally find that increasing the loss of the hole region 10 times to increase the attention to the area is helpful in training the network. Consequently, the \(L1\) loss between the restored video, \(V_{r}\), and the ground truth (GT), \(V_{c}\), can be written as:  
\[l_{L1}=10\|V_{r}-V_{c}\|\odot M+\|V_{r}-V_{c}\|\odot(1-M) \tag{8}\]  
and the loss of generator in the video inpainting task is:  
\[l_{G}=l_{L1}-10^{-3}\times(\mathbb{E}_{V_{r}\sim\mathbb{P}_{g}}[D_{g}(V_{g})]+ \mathbb{E}_{V_{r}\sim\mathbb{P}_{g}}[D_{l}(V_{r}\odot M)]) \tag{9}\]  
Intuitively, we use the \(L1\) loss to directly regress the missing region to the complete video, and at the same time, WGAN trains the generator with adversarial gradients to make the result much more real. During the video SR task, the discriminator we design contains only the global one, different from the inpainting task, which evaluates the consistency of the reconstructed high-resolution part with the entire frame. In detail, we directly adopt the generated high-resolution frame, \(I_{HR}\), as the input of the global discriminator and the label for the ground truth is \(I_{GT}\).