# Parallel Image Completion with Edge and Color Map
## 3 Materials and Methods  
### Image Completion Problem Formulation  
Given a corrupted image \(I\) with the missing region \(\Omega\) and the remaining image context \(\tilde{\Omega}\), i.e., \(I=\Omega\cup\tilde{\Omega}\), the goal of image completion is to fill in the missing region \(\Omega\) with realistic and appealing contents making use of the information in the known region \(\tilde{\Omega}\). The intensity of a pixel within the image \(I\) is denoted as \(I(x,y)\), where \((x,y)\) is the coordinate of the pixel. \(M\) is a binary mask with \(M(x,y)=1\) representing a missing pixel, given by:  
\[M(x,y)= \begin{cases}1,&\text{if }(x,y)\in\Omega,\\ 0,&\text{if }(x,y)\in\tilde{\Omega}.\end{cases}  \tag{1}\]  
Image completion task refers to a class of the linear inverse problems that tend to be ill-posed since there are an infinite number of feasible solutions, but only a few of them are natural images. A common approach for resolving this issue is to obtain a solution of the constrained optimization problem below:  
\[I^{*}_{comp}=\arg\min_{I_{comp}\in\mathbb{I}_{comp}}\mathbb{E}_{ \begin{subarray} {c}I-1\\ I_{gt}\end{subarray} }\left[D(F(I),I_{gt})\right],\quad s.t.\ I=(1-M)\odot I_{gt}, \tag{2}\]  
where \(\mathbb{I}\), \(\mathbb{I}_{comp}\) and \(\mathbb{I}_{gt}\) are the distributions of the corrupted images, the completed images and the ground truth images, \(F:\mathbb{I}\mapsto\mathbb{I}_{comp}\) is a mapping function from the corrupted images to the completed images, \(D(\cdot)\) is the distance between the completed image and the ground truth image, and \(\odot\) denotes the Hadamard product. Because of the complexity of the natural images, it is difficult to directly solve this linear inverse problem using the deep neural networks without any priors. The image priors serve as the regularizations that stabilize the degradation inversion, and direct the outcomes toward the more plausible results. In the image-to-image (I2I) translation field, some methods [26; 27; 39] disentangle the image in order to extract content and style respectively and can synthesize photorealistic images from the binary edge maps. Consequently, edge map is chosen as the structure prior to guide the image completion network generating semantic image contents with a more intact structure in the missing region. Color information also playsa critical role in the image-to-image translation. Compared with hand-designed priors, the edge and color maps can be learned directly from the image dataset. Therefore, they are tailored to the statistics of images in the dataset, and, in principle, provide stronger regularization to this inverse problem. Because the edge and color information have also been lost in the missing region, they should be restored firstly. The edge prior can be obtained from the constrained optimization problem below:  
\[B^{*}_{comp}=\arg\min_{B_{comp}\in\mathbb{B}_{comp}}\mathbb{E}_{ \begin{subarray}{c}B_{g\to\mathbb{B}}\\ gt^{*}\to\mathbb{B}gt\end{subarray} }\left[D_{1}(G_{1}(B),B_{gt})\right],\quad s.t.\ B=(1-M)\odot B_{gt}, \tag{3}\]  
where \(\mathbb{B}\), \(\mathbb{B}_{comp}\) and \(\mathbb{B}_{gt}\) are the distributions of the corrupted edge maps, the completed edge maps and the ground truth edge maps, \(G_{1}:\mathbb{B}\mapsto\mathbb{B}_{comp}\) is a mapping function from the corrupted edge maps to the completed edge maps, and \(D_{1}(\cdot)\) is the distance between the completed edge map and the ground truth edge map. The color prior can be obtained from the constrained optimization problem below:  
\[C^{*}_{comp}=\arg\min_{C_{comp}\in\mathbb{C}_{comp}}\mathbb{E}_{ \begin{subarray}{c}C\sim C\\ C_{gt}\sim C_{gt}\end{subarray} }\left[D_{2}(G_{2}(C),C_{gt})\right],\quad s.t.