# A Review of Image Inpainting Methods Based on Deep Learning
## 4 Deep-Learning-Based Image Inpainting Algorithm
### Deep-Learning-Based Image Restoration Process  
The process of image restoration using deep learning involves encoding and decoding. The damaged image is compressed into a latent space encoding through an encoder. This compressed representation is then expanded or decoded to produce a restored image. Mathematically, this can be represented as \(t^{\prime}=\text{Decoder}\ (\text{Encoder}(z))\), where \(t\) is the damaged image, and \(t^{\prime}\) is the restored image. In terms of loss functions, the L1 or L2 loss is typically used to measure the difference between the restored image and the original image. The mathematical representation for this is \(L=|t^{\prime}-t_{\text{true}}|^{P}\), where \(t_{\text{true}}\) represents the original undamaged image. The value of \(p\) can be 1 or 2, representing \(L1\) and \(L2\) losses, respectively. Furthermore, to enhance the realism of the restored images, an adversarial loss can be incorporated. This is represented as \(L_{\text{ad}}=\max_{p}\mathbb{E}_{x\sim X}[\log(D(e))]-\mathbb{E}_{z\sim Z}[ \log(1-D(G(z)))]\), where \(D\) is the discriminator, \(G\) is the generator, \(X\) is the distribution of real images, and \(Z\) is the distribution of the latent space. The network model is optimized by evaluating the L1 or L2 loss between the missing region of the original image and the corresponding pixel values of the damaged area generated by the decoder. The reason for using L1 or L2 loss is that they provide a measure of the difference between the predicted and actual pixel values. L1 loss, corresponding to the absolute difference, is more robust to outliers, while L2 loss, which is the squared difference, penalizes larger errors more heavily. By minimizing these losses, the model aims to produce inpainted images that are as close as possible to the original undamaged images. During the inpainting process, the deep learning model needs to address the semantic understanding of the missing area in the damaged image and restore the image with fine textures after understanding the semantics. [PERSON] et al. [19] was the first to introduce a deep-learning-based Generative adversarial network [23] to the image inpainting task. As shown in the figure, the restored image generated by the generator not only calculates the L1 or L2 loss but also goes through a discriminator to calculate the adversarial loss. [PERSON] et al. [19] modified the adversarial loss to only adjust the generator. Specifically, Equations (5) and (6) describe this loss computation:  
\[\min_{G}\max_{D}\mathbb{E}_{x\in\mathcal{X}}[\log(D(x))]+\mathbb{E}_{z\in \mathcal{Z}}[\log(1-D(G(z)))] \tag{5}\]  
\[\mathcal{L}_{adv}=\max_{D}\mathbb{E}_{x\in\mathcal{X}}[\log(D(x))+\log(1-D(F(( 1-\hat{M})\odot x)))] \tag{6}\]  
In these equations, \(G\) and \(D\) represent the generator and discriminator in the adversarial network, respectively; \(E\) denotes the expectation; \(x\) is the original image; and \(z\) is the latent space representation. \(\hat{M}\) is the mask indicating the missing region, and \(\odot\) is the element-wise multiplication. The adversarial loss represented by \(L_{adv}\) measures the difference between the generated inpainted image and the original image. By optimizing  
Figure 1: Context encoder network model diagram. this loss, our model aims to produce inpainted images that are indistinguishable from real images. L1 or L2 losses tend to produce blurry results, while adversarial losses tend to produce sharp but incoherent results. [PERSON] et al. (2019) combined the two to obtain a more balanced solution. Furthermore, after introducing the idea of a generative adversarial model, the inpainting network can more imaginatively generate images in the missing areas. To better understand these concepts, the key concepts in deep learning for image restoration are listed in Table 1.  
Deep-learning-based methods combining encoder-decoder structures with generative adversarial networks have achieved better results than previous methods. [PERSON] et al.