# Image Structure-Induced Semantic Pyramid Network for Inpainting
## 2 Approach
### Loss Function
#### 2.4.2 Content-Filling Loss  
Using \(I_{gt}\) to show the original image and M to indicate the mask, the missing image \(\widetilde{I}_{gt}\) is denoted as follows  
\[\widetilde{I}_{gt}=I_{gt}\otimes(1-M) \tag{12}\]  
Using \(G_{2}(\cdot)\) to denote the content-fill generator operation and \(D_{2}(\cdot)\) to denote the image discriminator operation, the resulting image is represented as:  
\[I_{pred}=G_{2}\Big{(}\widetilde{I}_{gt},C_{pred},M\Big{)} \tag{13}\]The ultimate output of the entire inpainting network is:  
\[I=\widetilde{I}_{gt}+I_{pred}\otimes M \tag{14}\]  
The adversative loss \(L_{adv2}\) is introduced as follows to train the content-filling network:  
\[L_{adv2}=E_{(I_{gt},I_{pred})}\log\Bigl{[}D_{2}\Bigl{(}I_{gt},C_{pred}\Bigr{)} \Bigr{]}+E_{C_{pred}}\log\Bigl{[}1-D_{2}\Bigl{(}I_{pred},C_{pred}\Bigr{)} \Bigr{]} \tag{15}\]  
The perceptual loss compares the feature maps acquired after the same convolution operation between the true image and the generated image and minimizes the difference between them to improve the high-level semantic consistency of the two kinds of images. Specifically, in this paper, the true image and the inpainted image are compared through the corresponding activation feature maps of pool-\(i\) (\(i\) = 1, 2, 3) in the network layer of VGG-16 trained on ImageNet. \(N_{k}\) indicates the amount of elements in the \(k^{th}\) activation layer, and \(\sigma_{k}\) represents the activation diagram of the corresponding layer. The formula for calculating perceptual loss is as below:  
\[L_{pred}=E[\sum_{k}\frac{1}{N_{k}}\|\sigma_{k}\bigl{(}I_{gt}\bigr{)}-\sigma_{k }\Bigl{(}I_{pred}\Bigr{)}\|_{1}] \tag{16}\]  
The style loss can be defined as the correlation factor between the activation values of each channel of the activation feature, measured using the activation of layer \(k\). In this article, a network activation layer consistent with the perceived loss is chosen and its correlation is represented by calculating the eccentric covariance between activation feature maps at different scales. Specifically, the style loss is defined as follows:  
\[L_{style}=E_{k}[\|G^{\sigma}_{k}\bigl{(}I_{gt}\bigr{)}-G^{\sigma}_{k}\Bigl{(}I_ {pred}\Bigr{)}\|_{1}] \tag{17}\]  
where \(G^{\sigma}_{k}\) is a Gram matrix of \(C_{k}\times C_{k}\) size composed of \(\sigma_{k}\). The introduction of the style loss can availably counter the fuzzy impact caused by transposed convolution. To complete the transmission of semantic information and detailed features in the decoding process, L1 loss is applied to the RGB image output and transformed at all levels of decoding layers. Compared with the L1 norm, the L2 norm can partly raise the convergence speed of the model. However, in the early stage of network training, the L2 norm will increase the difference between the restoration results and the corresponding pixel points of the original image. The application of the L2 norm at all levels of network will further expand this difference, easily causing gradient explosion. In contrast, the L1 norm has a steady gradient for any input value and does not cause gradient explosion concerns. By scaling the real image size to be consistent with the RGB image size output at each decoding layer and calculating the L1 normalized distance between them, the reconstruction loss between the decoded output at each layer and the real image can be represented as follows:  
\[L_{rec}=\sum_{l=1}^{L-1}\|I_{gt}^{l}-r\Bigl{(}\gamma^{l}\Bigr{)}\|_{1} \tag{18}\]  
Among them is \(I_{gt}^{l}\) that scales to real images of the same size and the decoding figure \(\gamma^{l}\); \(r(\cdot)\) is a 1*1 convolution, and it will decode \(\gamma^{l}\) for RGB image and maintain the same size.