# Image Structure-Induced Semantic Pyramid Network for Inpainting
## 2 Approach
### Texture Generation Network  
Similar to the edge inpainting network, the texture generation network is made up of a generator and a discriminator. In combination with the structural characteristics of the ATM and U-Net, the generator applies the ATM layer by layer from deep to shallow after the completion of encoding and then transfers the ATM reconstructed feature map to the multiscale decoder through the jump connection, which is mixed with the corresponding potential feature map for layer-by-layer decoding. After decoding, the features of each decoding layer are converted into RGB pictures of the same size, and the L1 loss is designed to contrast the differences between them and the true image, forcing the generator to improve the context consistency of each decoding feature. Finally, through perception loss, style loss, and adversarial loss, the final repair results are evaluated, and the network parameters are optimized. To show fine-grained inpainting effects, it is significant to decrease the local detail loss of output features at all levels of the network layer. General convolution with a convolution kernel size of 3 \(\times\) 3 and a step size of 1 is designed in both the coding and decoding stages of the image inpainting network instead of expanded convolution with different expansion rates. This is because in the U-Net network, with an increasing network depth, the scale of the feature graphs at all levels decreases step by step. However, in feature graphs at 64 \(\times\) 64 and lower scales, using expanded convolution with the grid effect will lose the continuity between adjacent pixels and ignore local important features, which will reduce the effect of subsequent ATM reconstruction features. After encoding is performed, the generator applies the ATM layer by layer to rebuild the feature map. Given a 6-layer encoder, from depth to shallow, the output feature map of the encoding procedure is recorded as \(\psi^{6}\), \(\psi^{5}\), \(\psi^{4}\), \(\psi^{3}\), \(\psi^{2}\), \(\psi^{1}\), and \(F(\cdot)\) represents the ATM operation. Then, the reconstruction features at different levels are as follows:\[ \begin{split}\varnothing^{5}&=F\big{(}\psi^{5},\psi^{6} \big{)}\\ \varnothing^{4}&=F\big{(}\psi^{4},\psi^{5}\big{)}\\ &\vdots\\ \varnothing^{1}&=F\big{(}\psi^{1},\psi^{2}\big{)}\end{split}  \tag{4}\]  
The multiscale decoder decodes the reconstructed feature map from the ATM in a jump-joined manner along with the underlying feature from the encoder as import. The output characteristics of each layer of the multiscale decoder are represented as \(\gamma^{5}\), \(\gamma^{4}\), \(\gamma^{3}\), \(\gamma^{2}\), and \(\gamma^{1}\), \(L(\cdot)\) is the transposed convolution operation, and \(\oplus\) is the feature splicing. Then, the decoding process can be represented as:  
\[ \begin{split}\gamma^{5}&=L\big{(}\varnothing^{5} \oplus L\big{(}\psi^{6}\big{)}\big{)}\\ \gamma^{4}&=L\big{(}\varnothing^{4}\oplus\gamma^{5} \big{)}\\ \vdots\\ \gamma^{1}&=L\big{(}\varnothing^{1}\oplus\gamma^{2} \big{)}\end{split}  \tag{5}\]  
On the one hand, the feature map reconstructed by the ATM can encode more low-level information for the lacking area. At the same time, even though the relevant pixel content cannot be extracted from the missing region, the feature information acquired from the compact potential feature through convolution can synthesize new targets in the missing area. Combining these two influences, multiscale decoders can utilize new substances with complete semantics and realistic texture by using the context information of images. After the completion of multiscale decoder decoding, the feature maps output at every level of the decoding layer is concatenated with the corresponding ATM feature maps to form an RGB image. The last RGB image is restored to 256 \(\times\) 256 by upsampling to get the restored image. The tanh activation function is applied in each convolution layer of the RGB image network converted from decoding feature maps at various scales.