# Ancient Painting Inpainting with Regional Attention-Style Transfer and Global Context Perception
## 3 RG-GAN
### Multi-Scale Fusion Generator  
In image inpainting, the U-Net architecture typically faces challenges, particularly when processing images with missing areas [27]. This is because the skip connections have almost no input signal in the center of the missing area, making it difficult to effectively pass texture information to the decoder, which can result in texture blurring and structural distortion in the repaired image. Additionally, the U-Net model has deficiencies in both global semantic consistency and local feature continuity, leading to semantically incoherent inpainting results.  
In this paper, we propose an innovative multi-scale fusion generator called the Multi-scale Fusion Generator (MFG). By designing a regional attention-style transfer module, it  
Figure 1: An overview of our proposed RG-GAN, including the Multi-scale Fusion Generator and Global Context Perception Discriminator.  
realizes fine-grained alignment between the content in the missing area and the content in the normal area, thus achieving complex style transfer while maintaining the authenticity of the content. This enhancement improves the quality of information passed through the skip connections. The MFG further leverages multi-scale fusion technology to progressively integrate filled high-level features with low-level features. This approach ensures that basic features can access and integrate semantic insights from advanced-level features, thereby improving the conveyance and application of feature data. Additionally, the MFG decoder outputs a repaired image at each scale, which is used to calculate pyramid losses, optimizing the filling effect on the missing area at each scale. The multi-scale fusion generator, which combines regional attention-style transfer modules, can adapt to various image inpainting scenarios, thereby improving the quality of inpainting.  
As shown in Figure 1, MFG consists of six layers of multi-scale residual downsampling (MRD), which reduces the size of the input feature map from 128 \(\times\) 128 to 4 \(\times\) 4. The multi-scale residual downsampling module is shown in Figure 2. The decoder part also contains a six-layer structure and interacts with the encoder through a connection. The MRD module consists of two branches. The left branch uses the 1 \(\times\) 1 convolution module and the 3 \(\times\) 3 depth convolution module to extract fine-grained features, while the right half uses the depth convolution module to improve the inpainting function of ancient painting images while reducing the model parameters. Finally, the output features of the two branches are fused, effectively extracting and integrating features of different scales, providing rich feature representation for the network and reusing input features when establishing residuals. MRD can be expressed as:  
\[\text{X1}=\text{Conv}_{1}(\text{Conv}_{1}(\text{Conv}_{3}(\text{Conv}_{1}(\text {F}))))\oplus\text{F}, \tag{1}\]  
\[\text{X2}=\text{PWConv}_{1}(\text{DWConv}_{3}(F)), \tag{2}\]  
\[\text{X}=\text{Conv}_{1}(\text{Cat}(\text{X1},\text{X2})), \tag{3}\]  
where \(\text{DWConv}_{3}\) refers to the 3 \(\times\) 3 depth-wise convolution, \(\text{PWConv}_{1}\) denotes the 1 \(\times\) 1 pointwise convolution, \(\text{Conv}_{3}\) and \(\text{Conv}_{1}\) represent the standard 3 \(\times\) 3 and 1 \(\times\) 1 convolutions, respectively, Cat is the concatenation operation, and \(\oplus\) indicates element-wise addition.  
Figure 2: Multi-scale residual downsampling module.