# Structure-Guided Image Inpainting Based on Multi-Scale Attention Pyramid Network
## 3 Methodology
### Multi-Scale Attention Pyramid Network
\begin{split} N_{smuc}&=K_{s}\times K_{s}\times Int _{c}+O_{c}\times 1\times 1\times I_{c}\\ &=(K_{s}\times K_{s}+O_{c})\times I_{c}\\ &=(K_{s}\times K_{s}\times I_{c})\times(1+\frac{O_{c}}{K_{s}\times K _{s}})\end{split}  \tag{7}\]  
Figure 4: Steps of the separable gated convolution operation, detailing the entire convolution process from the input image to the output image. Figure 3: Structure of a multi-scale attention pyramid restoration network which includes a pyramid-structured encoder, decoder, and multi-scale attention modules. The network leverages edge maps obtained from a structural generation network to assist in generating a fully restored image. \(N_{com}\) and \(N_{smuc}\) represent the number of parameters required for standard convolution and separable masked update convolution (SMUC), respectively. \(K_{s}\) denotes the kernel size, while \(O_{c}\) and \(l_{c}\) represent the number of output and input channels, respectively. When \(K_{s}=1\), SMUC requires \(l_{c}\) more parameters than standard convolution. However, as the kernel size increases, SMUC requires fewer parameters compared to standard convolution. The proposed method utilizing SMUC can achieve nearly a 30 percent reduction in parameters. To capture more precise features, once the pyramid context encoder learns latent features, it transmits them from higher to lower semantic features through the multi-scale attention fusion module. Given a pyramid context encoder composed of \(L\) layers, the feature maps from deeper to shallower layers are represented as \(\phi^{L},\phi^{L-1},\ldots,\phi^{1}\), with each layer's features constructed by the multi-scale attention fusion operation:  
\[ \begin{split}\psi^{(L-1)}&=f(\phi^{(L-1)},\phi^{L }),\\ \psi^{(L-2)}&=f(\phi^{(L-2)},\phi^{(L-1)}),\\ &\vdots\\ \psi^{1}&=f(\phi^{1},\psi^{2})=f(\phi^{1},f(\phi^{2}, \ldots,f(\phi^{(L-1)},\phi^{L}))).\end{split}  \tag{8}\]  
The function \(f\) represents the multi-scale attention fusion operation. Attention is typically obtained by the affinity between blocks inside and outside a region (usually \(3\times 3\)), allowing relevant features to be transferred (i.e., weighted replication through affinity in the context) into the internal region. The output of each fusion operation is denoted as \(\psi^{l}\), which is then used as an input for the next layer's fusion process. The multi-scale attention fusion module initially learns regional affinity from the high-level feature map \(\psi^{l}\). It extracts blocks from \(\psi^{l}\) and calculates the cosine similarity between blocks inside and outside the missing area:  
\[s^{l}_{(i,j)}=\left(\frac{p^{l}_{i}}{\|p^{l}_{i}\|_{2}},\frac{p^{l}_{j}}{\|p^{ l}_{j}\|_{2}}\right) \tag{9}\]  
where \(p^{l}_{i}\) is the \(i\)-th block extracted from outside the mask in \(\psi^{l}\), and \(p^{l}_{j}\) is the \(j\)-th block. A softmax is then applied to obtain the attention scores for each block based on the similarity:  
\[\alpha^{l}_{(j,i)}=\frac{\exp(s^{l}_{(i,j)})}{\sum_{l=1}^{N}\exp(s^{l}_{(i,j)})} \tag{10}\]  
After obtaining the attention scores for the high-level feature map, the gaps in its adjacent lower-level feature map can be filled:  
\[p^{(l-1)}_{j}=\sum_{i=1}^{N}\alpha^{l}_{(j,i)}p^{(l-1)}_{i} \tag{11}\]  
where \(p^{(l-1)}_{i}\) is the \(i\)-th block extracted from outside the mask in \(\phi^{(L-1)}\), and \(p^{(l-1)}_{j}\) is the \(j\)-th block to be filled in the missing region. After computing all blocks, we can ultimately achieve feature filling by transferring attention from \(\psi^{l}\).