# Automatic Colorization of Anime Style Illustrations Using a Two-Stage Generator
## 3 Structure of Proposed Method
### Loss Function  
In this section, we describe loss functions used in the model. The loss function for the draft model is defined in Equation (2). It takes account of four different loss functions: GAN loss (\(\mathcal{L}_{GAN}\)), reconstruction loss (\(\mathcal{L}_{recun}\)), contents loss (\(\mathcal{L}_{cont}\)), and line loss (\(\mathcal{L}_{line}\)). The effect of each component is controlled by coefficients \(w_{a}\), \(w_{r}\), \(w_{c}\), and \(w_{l}\), respectively. Notation \(l\) denotes line-art with dimension 128 \(\times\) 128, \(h\) denotes color hint, and \(c\) denotes the original color image with dimension 128 \(\times\) 128. \(D\) and \(G\) denote discriminator and generator used in the draft model. \[\mathcal{L}_{draft}=w_{a}\underset{G}{\text{\emph{min}}}\underset{D}{\text{ \emph{max}}}\mathcal{L}_{GAN}(G,D)+w_{r}\mathcal{L}_{recun}(G)+w_{c}\mathcal{L }_{cont}(G,\mathcal{F}))+w_{l}\mathcal{L}_{line}(G,ldm) \tag{2}\]  
\[\mathcal{L}_{GAN}(G,D)= \mathbb{E}_{c}[\log(D(c))]+\mathbb{E}_{l,h}[\log(1-D(G(l,h)))] \tag{3}\]  
Loss of GAN is described in Equation (3) following the work of [1,16]. Discriminator, \(D()\), estimates the probability of real color image, \(c\). The result of \(D()\) has value between 0 and 1. \(\mathbb{E}_{c}\) and \(\mathbb{E}_{l,h}\) measures the expected value of \(\log(D(c))\) and \(\log(1-D(G(l,h)))\), respectively. \(G(l,h)\) generates a colored image exploiting extracted line-art along with user provided hint. \(G(l,h)\) tries to generate a image distribution as close as original image. \(D(G(l,h))\) estimates the probability of generated instances. In theory, \(D()\) and \(G()\) are enough to work against each other to improve the quality. However, practically, we need to employ other loss functions to provide balance in GAN structure. In this paper, we exploit Equations (4)-(6) to stabilize the learning process. \[\mathcal{L}_{recun}(G)=\mathbb{E}_{l,h,c}[\|c-G(l,h)\|_{1}] \tag{4}\]  
The reconstruction loss is defined in Equation (4). Expected value is measured over \(L_{1}\) loss of real color image \(c\) and generated draft image \(G(l,h)\). Through \(\mathcal{L}_{recun}\), \(G()\) can adjust color space of given images to match the color distribution of original color image, \(c\). As a result, \(G()\) can generate a better image that can deceive \(D()\). \[\mathcal{L}_{cont}(G,\mathcal{F})=MSE(\mathcal{F}(c)-\mathcal{F}(G(l,h))) \tag{5}\]  
Content loss (\(\mathcal{L}_{cont}\)) is defined as Equation (5). \(\mathcal{L}_{cont}\) is \(L_{2}\) measurement (Mean Square Error, MSE) of \(\mathcal{F}\) feature map obtained from draft image and the original image. \(\mathcal{F}\) represents feature map generated in fourth convolution layer in VGG16 [26] which is trained with ImageNet [27]. \(\mathcal{L}_{cont}\) measures the effect of the feature maps difference of output from the generator and the original image. \(\mathcal{L}_{cont}\) is a method of calculating the distance of the extracted feature map using pre-trained \(\mathcal{F}\). Since the perceptual similarity is the main concern in this equation, using the feature map captures the characteristics of the image that cannot represented by observing a pixel. \[\mathcal{L}_{line}(G,ldm)=\mathbb{E}_{l,h,c}[\|ldm(c)-ldm(G(l,h))\|_{1}] \tag{6}\]  
Line loss (\(\mathcal{L}_{line}\)) is defined as Equation (6). \(\mathcal{L}_{line}\) calculates the \(L_{1}\) loss of line-arts of draft (\(ldm(G(l,h))\)) and the original image (\(ldm(c)\)) generated by LDM. LDM is described in Section 3.3 and converts color images \(c\) to line-art \(ldm(c)\). The logic behind \(\mathcal{L}_{line}\) is as follows. If \(G\) is sufficiently trained and the generalization of the line-art is properly progressed, the difference between the line-art \(ldm(G(l,h))\) and \(ldm(c)\) created using \(G(l,h)\) and \(c\) should be small. \(\mathcal{L}_{line}\) takes account of the line distribution difference between two line-arts (\(ldm(G(l,h))\), \(ldm(c)\)) generated by LDM in pixel space.