# Coarse-to-Fine Structure-Aware Artistic Style Transfer
## 3 Proposed Method
### Loss Function  
Our coarse network needs to train only once, and it is fixed during the training of the fine network. Compared with WCT [6], we train only one reconstruction decoder network to reconstruct the coarse stylized feature. Our coarse network can reconstruct the stylized features at three levels or directly generate a coarse stylized image by taking advantage of the reconstruction decoder. Following WCT, we adopt pixel reconstruction and perceptual loss [3] to train our decoder for image reconstruction:  
\[l_{re}=\Big{|}\Big{|}I_{o}-I_{i}\Big{|}\Big{|}\Big{|}_{2}^{2}+\lambda\Big{|} \Big{|}\Phi(I_{o})-\Phi(I_{i})\Big{|}\Big{|}_{2}^{2} \tag{3}\]  
where \(I_{i}\) and \(I_{o}\) are the input image and output image, respectively, and \(\Phi\) is the VGG encoder that extracts features at ReLU_X_1 (X = 1, 2, 3, 4). In addition, \(\lambda\) is the weight to balance the two losses. The fine network is optimized with content and style loss during training. As shown in Figure 5, we keep a single \(x_{s}\) and a set of \(x_{c}\) from a content dataset, then \(x_{cs}\) is a stylized image generated by the fine network. For \(x_{s}\), \(x_{c}\), and \(x_{cs}\), we can use a pretrained VGG-19 encoder to extract their features \(F_{c}^{(t)}\in\mathbb{R}^{c_{t}\times h_{t}\times w_{t}}\), \(F_{s}^{(t)}\in\mathbb{R}^{c_{t}\times h_{t}\times w_{t}}\), and \(F_{cs}^{(t)}\in\mathbb{R}^{c_{t}\times h_{t}\times w_{t}}\), where \(t\) denotes the features extracted at ReLU_\(t\) (\(t\) = 1\(1\)1_2\(2\)1_2\(2\)3_1\(3\)3\(4\)1_). For content loss, we adopt the commonly used perceptual loss between \(F_{c}^{(t)}\) and \(F_{cs}^{(t)}\) proposed in [3]. The perceptual loss can measure high-level perceptual and semantic differences between images, and it is defined as follows:  
\[l_{p}=\frac{1}{c_{t}h_{tr}\upsilon_{t}}\Big{|}\Big{|}F_{c}^{(t)}-F_{cs}^{(t)} \Big{|}\Big{|}_{2}^{2} \tag{4}\]  
For style loss, we adopt three style losses. The first and most significant style loss is the relaxed earth mover's distance (rEMD) loss [2], which helps the fine network generate visual effects with minimum distortion to the layout of the content image. This loss plays a  
Figure 5: The schematic for loss network. key role in migrating the structural forms of the style image to the target image. The rEMD loss between \(F_{s}^{(t)}\) and \(F_{cs}^{(t)}\) can be calculated as follows:  
\[l_{r}=\max\left(\frac{1}{h_{t}w_{t}}\sum_{i=1}^{h_{t}w_{t}}minC_{ij},\frac{1}{h_{ t}w_{t}}\sum_{j=1}^{h_{t}w_{j}}minC_{ij}\right) \tag{5}\]  
where \(C\) is the cost matrix, which can be calculated as the cosine distance between \(F_{s}^{(t)}\) and \(F_{cs}^{(t)}\):  
\[C_{ij}=D_{cos}\left(F_{s,j}^{(t)},F_{cs,j}^{(t)}\right)=1-\frac{F_{s,i}^{(t)} \cdot F_{cs,j}^{(t)}}{\left|\left|F_{s,i}^{(t)}\right|\right|\left|F_{cs,i}^{(t )}\right|\right|} \tag{6}\]  
The second style loss is the commonly used style reconstruction loss proposed by [PERSON] et al. [1], which is the difference between the Gram matrices of \(F_{s}^{(t)}\) and \(F_{cs}^{(t)}\):  
\[l_{g}=\left|\left|G\left(F_{s}^{(t)}\right),G\left(F_{cs}^{(t)}\right)\right| \right|_{2}^{2} \tag{7}\]  
where \(G\) denotes the calculation of the Gram matrix of the feature vectors. Finally, we use the mean-variance loss as the third style loss, which is similar to the style reconstruction loss. We can use this type of loss to reduce unnecessary visual effects in the stylized image and keep the magnitude of the stylized feature the same as that of the style feature:  
\[l_{m}=\left|\left|\mu\left(F_{s}^{(t)}\right)-\mu\left(F_{cs}^{(t)}\right) \right|\left|\left|2+\right|\left|\sigma\left(F_{s}^{(t)}\right)-\sigma\left(F _{cs}^{(t)}\right)\right|\right|_{2}^{2} \tag{8}\]  
where \(\mu\) and \(\sigma\) denote the mean and covariance of the feature vectors, respectively. The overall optimization objective is defined as follows:  
\[L=\alpha l_{p}+\lambda_{1}l_{r}+\lambda_{2}l_{g}+\lambda_{3}l_{m} \tag{9}\]  
where \(\alpha\), \(\lambda_{1}\), \(\lambda_{2}\), and \(\lambda_{3}\) are weight terms.