# SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting
## 3 Method
### LR Inpainting Network  
The LR inpainting network aims to characterize variations across the entire image in the LR field-of-view and to recover missing information. As the fundamental quality of HR reconstruction, LR inpainting must ensure structural consistency, semantic continuity, and sufficient details of filling content in the LR field-of-view. Hence, we adopt a two-stage GAN framework [9] to realize high-quality image inpainting in LR field-of-view. The LR inpainting network consists of an edge generator and an image completion network, each stage of which follows an adversarial model consisting of a pair of generator-discriminator. Specifically, the generator follows the architecture proposed by [PERSON] et al. [41] and consists of two downsampling encoders, eight residual blocks [42], and two upsampling decoders. The discriminator uses the 70 \(\times\) 70 PatchGAN [43; 44] architecture, which discriminates whether the 70 \(\times\) 70 overlapping blocks are true or not.  
Figure 1: The overall pipeline of our method (SR-Inpaint): **(top)** the pipeline of SR-Inpaint, **(bottom)** the architectures of networks.  
Figure 2: Algorithm flowchart of SR-inpaint.  
The core task of edge generator is to predict the edge map for the masked region, as shown in Equation (1). Let \(\mathbf{I}_{gt}\) be the ground truth image. \(\mathbf{C}_{gt}\) and \(\mathbf{I}_{gray}\) denote its edge map and grayscale counterpart respectively. In this stage, we use the masked grayscale image \(\bar{\mathbf{I}}_{gray}=\mathbf{I}_{gray}\odot(1-\mathbf{M})\) as input. Its edge map and image mask are denoted as \(\bar{\mathbf{C}}_{gt}=\mathbf{C}_{gt}\odot(1-\mathbf{M})\) and \(\mathbf{M}\), respectively, and used as pre-condition. Here, \(\odot\) denotes the Hadamard product.  
\[\mathbf{C}_{\text{pred}}\,=\,G_{1}\big{(}\bar{\mathbf{I}}_{\text{gray}}\,\bar{\mathbf{C}}_{gt},\mathbf{M}\big{)} \tag{1}\]  
The network is trained with both the adversarial loss \(\mathcal{L}_{adv,1}\) and feature-matching loss \(\mathcal{L}_{FM}\), as shown in Equation (2), where \(\lambda_{adv,1}\) and \(\lambda_{FM}\) are regularization parameters. The feature-matching loss is very similar to perceptual loss, and compare the activation maps in the intermediate layers of the discriminator, which further stabilize the training process by forcing the similarities of both the results of the generator and the real images.  
\[\min_{G_{1}}\max_{D_{1}}\mathcal{L}_{G_{1}}=\min_{G_{1}}\bigg{(}\lambda_{adv, 1}\max_{D_{1}}(\mathcal{L}_{adv,1})+\lambda_{FM}\mathcal{L}_{FM}\bigg{)} \tag{2}\]  
For the second stage, i.e., the image completion network, the incomplete color image \(\bar{\mathbf{I}}_{gt}=\mathbf{I}_{gt}\odot(1-\mathbf{M})\) are used as input, conditioned using a composite edge map \(C_{comp}\), which is constructed by combining the edges inferred with the first stage and ground truth edges in the remaining part of the original image, i.e., \(\mathbf{C}_{comp}=\mathbf{C}_{gt}\odot(1-\mathbf{M})+\mathbf{C}_{pred}\odot \mathbf{M}\). The network infers a color image \(I_{pred}\), with missing regions inpainted. This procedure is denoted as Equation (3).  
\[\mathbf{I}_{\text{pred}}\,=\,G_{2}\big{(}\bar{\mathbf{I}}_{gt},\mathbf{C}_{ \text{comp}}\ \big{)} \tag{3}\]  
This network is trained over a joint loss representation, as shown in Equation (4), containing \(\ell_{1}\) loss \(\mathcal{L}_{\ell_{1}}\), adversarial loss \(\mathcal{L}_{adv,2}\), perceptual loss \(\mathcal{L}_{\text{perc}}\) and style loss \(\mathcal{L}_{\text{style}}\), \(\lambda_{\ell_{1}}\), \(\lambda_{adv,2}\), \(\lambda_{p}\) and \(\lambda_{s}\) are all regularization parameters.  
\[\min_{G_{2}}\max_{D_{2}}\mathcal{L}_{G_{2}}=\min_{\bar{G}_{2}}\bigg{(}\lambda_ {\ell_{1}}\mathcal{L}_{\ell_{1}}+\lambda_{adv,2}\max_{D_{2}}(\mathcal{L}_{adv,2})+\lambda_{p}\mathcal{L}_{\text{perc}}+\lambda_{s}\mathcal{L}_{\text{style}} \bigg{)} \tag{4}\]