# A Review of Image Inpainting Methods Based on Deep Learning
## 4 Deep-Learning-Based Image Inpainting Algorithm
### Attention Mechanism
#### 4.3.1 Attention
The specific calculation process involves computing the cosine similarity of the foreground-background blocks through convolution, as described in Equation (9):  
\[s_{x,y,x^{\prime},y^{\prime}}=\left\langle\frac{f_{x,y^{\prime}}\cdot b_{x^{ \prime},y^{\prime}}}{\left\|f_{x,y}\right\|\cdot\left\|b_{x^{\prime},y^{\prime }}\right\|}\right\rangle \tag{9}\]  
A softmax operation is then performed on the similarity map in the channel dimension according to Equation (10):  
\[s^{*}_{x,y,x^{\prime},y^{\prime}}=\text{softmax}_{x^{\prime},y^{\prime}}\Big{(} \lambda s_{x,y,x^{\prime},y^{\prime}}\Big{)} \tag{10}\]  
where the softmax function is defined as:  
\[\text{softmax}(z_{i})=\frac{e^{z_{i}}}{\sum_{j=1}^{K}e^{z_{j}}} \tag{11}\]  
Using the background to perform transpose convolution on the similarity map completes the block matching reconstruction process, as shown in Equation (12). \[f_{\text{new}}=\text{deconv}(s,b) \tag{12}\]  
Subsequent methods optimized and improved upon [69]. To fix irregular holes, [PERSON] et al. [70] used partial convolution [58] instead of standard convolution based on [69]. [PERSON] et al. [71], building on [69], introduced a learnable attention mapping module for end-to-end learning of feature normalization and mask updates, which effectively adapts to irregular holes and the propagation of convolution layers. A learnable reverse attention map was also introduced to make the U-Net decoder focus on filling unknown areas rather than reconstructing the holes and known regions, resulting in a learnable bidirectional attention map. Due to the high computational complexity of the attention mechanism, the inpainting efficiency is quite low. To address this, [PERSON] et al. [72] used the Euclidean distance in the attention module to replace the cosine similarity used in [69], which calculates the similarity match of the known and to-be-restored feature blocks, significantly reducing the inpainting time. Using an improved version of this method, [PERSON] et al. [73] replaced the general dilated convolution layer in the original network with a rate-adaptive dilated convolution layer, further reducing resource consumption. Although compared to traditional convolution models, the content-aware layer can improve performance, the inpainting results still lack fine textural details, and pixels are inconsistent with the background. Existing methods often produce content with blurred textures and distorted structures due to the discontinuity of local pixels. Semantically, local pixel discontinuity mainly occurs because these methods neglect the semantic relevance and feature continuity of the area to be restored. To solve this, [PERSON] et al. [74] proposed a refined method based on deep generative models with an innovative coherent semantic attention (CSA) layer. Not only can it retain image context structure, but it can also more effectively predict defects by modeling semantic relevance between defect area features. The task is divided into coarse and refinement stages, with the CSA layer embedded in the encoder of the refinement stage. [PERSON] et al. [75] introduced an image inpainting model based on inner-outer attention (IOA) layers to improve the content-aware layer. To fully utilize feature information at different scales, some methods have been proposed using multiscale concepts for both the convolution part and the attention module. [PERSON] et al. [69]'s content-aware layer uses fixed-size image block matching, unable to effectively utilize background information at different scales. Furthermore, the content-awarelayer often lacks a mechanism to dynamically adjust the importance weight of attention based on input. [PERSON] et al. [76] proposed a content-aware layer based on selective latent space mapping (SLSM-CA). This module introduces two branches, learning pixel-level and image block-level attention from the background area. Block-level attention focuses more on structural patterns, while pixel-level attention focuses more on fine-grained texture.