# Parallel Image Completion with Edge and Color Map
## 3 Materials and Methods
### Loss Function
#### 3.3.3 Image Completion Network  
The image completion network was designed for solving the constrained optimization problem in Equation (5) to complete the corrupted images. The generator \(G_{3}\) used the masked image \(I_{gt}=I_{gt}\odot(1-M)\) as input, conditioned using the composite edge map \(E_{comp}\), the composite color map \(C_{comp}\) and the mask \(M\), and then predicted an RGB image \(I_{pred}\) with the missing region filled in:  
\[I_{pred}=G_{3}\left(I_{gt},E_{comp},C_{comp},M\right). \tag{22}\]  
In addition, the composite image was constructed by:  
\[I_{comp}=I_{gt}\odot(1-M)+I_{pred}\odot M. \tag{23}\]  
This network was trained over a joint loss that consists of a pixel-wise loss \(\mathcal{L}_{pix,3}\), a perceptual loss \(\mathcal{L}_{perc}\) and an adversarial loss \(\mathcal{L}_{adv}^{G_{3}}\), shown below:  
\[\mathcal{L}_{G_{3}}=\lambda_{pix,3}\mathcal{L}_{pix,3}+\lambda_{perc}\mathcal{ L}_{perc}+\lambda_{adv,3}\mathcal{L}_{adv^{\prime}}^{G_{3}} \tag{24}\]\[\mathcal{L}_{D_{3}}=\mathcal{L}_{adv}^{D_{3}}, \tag{25}\]  
where \(\lambda_{pix,3}\), \(\lambda_{perc}\) and \(\lambda_{adv,3}\) are the regularization parameters.  
Pixel-wise loss measures the \(\ell_{1}\) difference between the predicted RGB image \(I_{pred}\) and its corresponding ground truth \(I_{gt}\). It is adopted to accelerate optimization and boost the superior performance. The pixel-loss \(\mathcal{L}_{pix,3}\) is written as:  
\[\mathcal{L}_{pix,3}=\frac{1}{F_{mean}(M)}\left\|I_{gt}-I_{pred}\right\|_{1}. \tag{26}\]  
In the image completion network, the LSGAN was used to create a stable generator \(G_{3}\) which could match the distribution of real images with fine details while traditional methods cannot. Therefore, the adversarial losses for training generator \(G_{3}\) and the discriminator \(D_{3}\) were formulated as follows, respectively:  
\[\mathcal{L}_{adv}^{G_{3}}=\frac{1}{2}\mathbb{E}\left[\left(D_{3}(I_{comp})-1 \right)^{2}\right], \tag{27}\]  
\[\mathcal{L}_{adv}^{D_{3}}=\frac{1}{2}\mathbb{E}\left[\left(D_{3}(I_{gt})-1 \right)^{2}\right]+\frac{1}{2}\mathbb{E}\left[D_{3}\left(I_{comp}\right)^{2} \right]. \tag{28}\]  
The adversarial loss made the generated region deviate from over-smooth results obtained using the pixel-wise \(\ell_{1}\) loss as the real images were not very smooth, which typically have fine details. Although the adversarial loss promoted fine details in the synthesized results, it was still far from perfect. First, the discriminator \(D_{3}\) was not aware of the image context and did not explicitly consider the composite image consisting of both the generated region and the remaining image context. Second, it was not a very challenging task to perform binary classification for the discriminator \(D_{3}\) to learn the appearance of different objects and parts in the images. Note that the semantic image completion needs to not only synthesize textures consistent with the context but also recover missing objects and parts, which requires high-level features extracted from the image context. Thus, the perceptual loss [40] was used to penalize the results that were not perceptually similar to the real images by defining a distance measure between activation maps of a pre-trained network rather than the original images. Perceptual loss was first proposed in the real-time style transfer and super-resolution fields and combined into the objective function for generating high-quality images. It is based on pre-trained networks and defined as:  
\[\mathcal{L}_{perc}=\mathbb{E}\left[\sum_{i}\frac{1}{N_{i}}\left\|\phi_{i}(I_{ gt})-\phi_{i}(I_{comp})\right\|_{1}\right], \tag{29}\]  
where \(\phi_{i}\) corresponds to feature maps from layers _relu1\(1\), _relu2\(1\), _relu3\(1\), _relu4\(1\) and _relu5\(1\) of the pre-trained VGG network on the ImageNet dataset.