# Structure-Guided Image Inpainting Based on Multi-Scale Attention Pyramid Network
## 3 Methodology
### Dual U-Net Constrained Structure Prediction Network  
The structure prediction of missing image regions directly influences the quality of the final image inpainting results. The detailed architecture of the proposed dual U-Net constrained structure network is depicted in Figure 2. This network serves as  
Figure 1: Overall architecture of the proposed image inpainting network, consisting of two main components: the structural restoration network and the multi-scale attention pyramid network. the first component of the entire repair system, aiming to generate edge images that closely resemble the real image structure, thereby enhancing the accuracy of the image inpainting outcomes. The network features two identical branches of generative adversarial networks (GANs), each consisting of a U-Net-based generator [42] and a PatchGAN-based discriminator [43]. The U-Net extracts features from the input image through its encoder, capturing structural and textural information across multiple scales. This multi-scale feature extraction capability enables U-Net to fully understand the context of the image, providing robust support for the subsequent repair process. In the decoder segment, U-Net gradually restores image detail by upsampling and merging with high-resolution features from the encoder, allowing the repaired image areas to seamlessly integrate with surrounding regions, maintaining the overall integrity and consistency of the image. Additionally, the skip-connection mechanism introduced by U-Net allows for direct transmission of low-level and high-level feature information, thus avoiding information loss and aiding in the recovery of more detailed image features. In the structural repair sub-network, the real image is denoted as \(I_{GT}\), the mask as \(M\) (where 1 represents known areas, and 0 represents missing areas), and the damaged image as \(I_{M}\) (\(I_{M}=I_{GT}\odot(1-M)\)). The real structural view \(S_{GT}\) and the damaged structural view \(S_{M}\) are obtained through the Canny operator applied to the real and damaged images, respectively. The final output of this network is a complete structural view \(S_{OUT}\). \(S_{OUT}\) participates in the repair tasks of the second part, the multi-scale attention pyramid network, serving as critical guiding information. The accuracy of \(S_{OUT}\) is paramount. To enhance the accuracy of the generated results, a strong constraint mechanism is proposed, employing another U-Net-based encoder-decoder network for feature extraction from the real structural view, then imposing feature constraints on each corresponding layer of the damaged structure extraction network. The discrepancy between the structural features extracted by the two branches is computed as follows:  
\[L_{\text{mul}}=\sum_{i\in N}\|G_{s_{i}}(S_{M})-G_{s_{i}}(S_{GT})\|_{2} \tag{1}\]  
where \(N\) represents the total number of layers in \(G_{s}\). \(G_{s}\) represents the image structure generator. By reducing this discrepancy, the generated damaged structural view is ensured to closely resemble the real structure. Moreover, reconstruction loss, adversarial loss, and feature matching loss are also employed to enhance the accuracy of the structural view  
Figure 2: Specific structure of an edge-enhancement network for image inpainting under strict constraints. The network comprises two encoder–decoder networks with identical structures, differing primarily in the input images. By extracting edge features from real images, the network minimizes the discrepancy between the predicted and original structures, thereby enhancing the accuracy of the reconstruction in damaged or incomplete areas. prediction.