# Multi-Task Learning for Medical Image Inpainting Based on Organ Boundary Awareness
## 2 Related Works
### Inpainting in General Field
#### 2.1.2 Learning-Based Approach  
In learning-based approaches, different types of features can be learned from a large spectrum of sample images, leading to better predictions compared to conventional methods. The deep learning approaches have been studied extensively, and the results showed a significant improvement in the performance. The context encoder (CE) network [11] uses adversarial training [12] with a novel autoencoder. Most of the early deep learning methods use standard convolutional networks over the corrupted image, using convolutional filter responses on the pixels in the masked holes, which often lead to artifacts such as color discrepancy and blurriness. Partial convolution [13] was proposed, where the convolution is masked and renormalized to be conditioned on only valid pixels. In this model, an updated mask was automatically generated for the next layer as part of the forward pass. Later, partial convolution has been generalized to a gated convolution [14] by providing a learnable dynamic feature selection mechanism for each channel and each spatial location for free-form image inpainting. In early studies using deep learning networks, the missing parts were predicted by propagating the surrounding convolutional features into the missing region to produce semantically plausible images, but they often resulted in blurry images. Spatial attention has been applied to consider the contextual relationship between the background and the hole region. The Shift-Net model [15] introduced a special shift layer to the U-Net architecture to shift the encoder feature of the known region for an estimation of the missing parts, resulting in sharper images with detailed textures. A learnable bidirectional attention map module (LBAM) [16] learned feature re-normalization on both the encoder and decoder of the U-net [17] architecture. A recurrent feature network (RFN) [18] was proposed to exploit the correlation between adjacent pixels and strengthen the constraints for estimating deeper pixels. However, these studies have not fully utilized the structural knowledge of the image. There are several approaches to exploit the inherent structure of information in the input images by using the edge or object boundaries for inpainting [4; 5; 6; 8]. EdgeConnect [4] is a two-stage adversarial model that comprises an edge generator followed by an image completion network. The edge generator hallucinates the edges in the missing area (either square-shaped or arbitrary-shaped). The image completion network fills in the distorted regions using hallucinated edges as a priori for the inpainting. Edge structures and color-aware maps are fused in a two-stage generative adversarial network (GAN) [5]. In the first stage, edges with the missing regions are used to train an edge structure generator. Meanwhile, distorted input images with the missing part are transformed into a global color feature map by the content-aware fill algorithm. In the second stage, the edge map and the colormap are fused to generate the refined image. The authors in [6] proposed a foreground-aware image inpainting system that explicitly disentangles structure inference and content completion. The foreground contours are predicted first and then the inpainting is performed using the predicted contours as a guidance. A multi-task learning framework with auxiliary tasks of edge finding and gradient map prediction is used to incorporate the knowledge of the image structure to assist inpainting [8].