# Multi-Task Learning for Medical Image Inpainting Based on Organ Boundary Awareness
## 3 Proposed Method
### Loss Function  
The loss function returns a non-negative real number representing the difference between two quantities: the predicted label and the correct label. The loss function is like a form to force the model to pay the penalty every time it predicts its mistake, and the number of penalties is proportional to the severity of the error. In all supervised learning problems, our goal always includes minimizing the total penalty payable. Ideally, the loss function should return the minimum value of zero. During the training process, we used many different types of loss for various purposes. In our network, the input uses the distorted image \(\widetilde{I_{gt}}=I_{gt}\ \odot(1-M)\), where \(I_{gt}\) is the ground truth image and \(M\) is the mask image with 1 value for missing region and 0 for background. The symbol \(\odot\) denotes the Hadamard product. Similarly, we have \(I_{edge\_in}=I_{edge\ gt}\odot(1-M)\), where \(I_{edge\_gt}\) is the edge map extracted from ground truth images by the Canny edge detector. Our network generates three images: completed image \(I_{image\_pred}\), organ boundary map \(I_{organs\_pred}\) and edge map \(I_{edge\_pred}\) with missing regions filled in. Those images have the same resolution as the input image. Let \(G\), \(D_{1}\), \(D_{2}\), and \(D_{3}\) be the generator and the discriminator of the image generator, edge generator and organ boundary generator, respectively. \[I_{image\_pred},\ I_{edge\_pred},I_{organs\_pred}=G\left(\widetilde{I_{gt}},I_ {edge\_in}\right) \tag{1}\]  
First, we analyze the network with a decoding part which generates completed image \(I_{image\_pred}\). We employed two losses proposed in [34; 35], commonly known as perceptual loss \(Loss_{image\_perceptual}\) and style loss \(Loss_{image\_style}\). Perceptual loss is defined as follows:  
\[Loss_{image\_perceptual}=E\left[\sum_{i}\frac{1}{N_{i}}\left\|\delta_{i}\left(I _{image\_gt}\right)-\delta_{i}\left(I_{image\_pred}\right)\right\|_{1}\right] \tag{2}\]  
where \(\delta_{i}\) is the activation map in the ith layer of a pre-trained network. These activation maps are also employed to calculate style loss, which measures the differences between covariances of activation maps. Given feature maps of size \(N_{i}=C_{j}\times H_{j}\times W_{j}\), style loss is calculated by:  
\[Loss_{image\_style}=E\left[\sum_{i}\left\|G_{i}^{\delta}\left(\widetilde{I_{ image\_pred}}\right)-G_{j}^{\delta}\left(\widetilde{I_{image\_gt}}\right) \right\|_{1}\right] \tag{3}\]  
Figure 2: The architecture of Dilated Residual Network Block (DRN Block). where \(G_{j}^{S}\) is a \(C_{j}\times C_{j}\) Gram matrix generated from activation maps \(\delta_{i}\). We also used the reconstruction loss \(Loss_{image\_reconstruction}\) in our model. We chose \(l_{1}\) loss for this reconstruction loss. We also used a discriminator for our image completion part. Typically, generators' gradients often disappear quickly in generative adversarial networks [12]. To fix this problem, we employed Hinge loss [36], which is useful for classifiers. These loss functions are defined as:  
\[Loss_{image\_gen}=-E_{edge\_in}\Big{[}D_{1}\Big{(}I_{image\_pred},I_{edge\_in} \Big{)}\Big{]} \tag{4}\]  
\[Loss_{D_{1}}=E_{image\_gt\_edge\_in}\Big{[}\max(0,\;1-D_{1}\Big{(}I_{image\_gt},I_{edge\_in}\Big{)}\Big{]} \tag{5}\]  
In the components that generate the completed map and completed organ boundary, the structures are similar. The completed edge map and completed organ boundary map are denoted as \(I_{edge\_pred}\). \(I_{organs\_pred}\), respectively. We still use the perceptual losses \(Loss_{edge\_pred}\), \(Loss_{organs\_perceptual}\), style losses \(Loss_{edge\_style}\), \(Loss_{organs\_style}\) and reconstruction losses \(Loss_{edge\_reconstruction}\), \(Loss_{organs\_reconstruction}\) in our model for training the whole model.