# Transform a Simple Sketch to a Chinese Painting by a Multiscale Deep Neural Network
## 3 Method
### Network Architecture  
We designed a multiscale GAN to deal with the huge size, magnificent background and exquisite details of Chinese painting. To make the generative model and the discriminative model handle multiscale input images, they are all made up by convolutional layers.  
In the generative model, to synthesize the output image whose resolution is the same as the input sketch, all convolutions are \(3\times 3\) spatial filters applied with \(stride=1\) and \(pad=1\). Using a small-sized filter can effectively reduce the number of parameters. Without changing the receptive-fields, we can make networks deeper, which can obtain a more non-linear fitting ability. Unlike the U-Net [32] structure used by Pix2 pix [30] that uses several downsampling steps and then the same number of upsampling steps by using the deconvolutional layers [33], we used all stride one convolutional layer to avoid the using of deconvolutional layers, which will produce checkerboard artifacts [34] in the output images. Tables 1 and 2 show the details of the architecture including the generative model and discriminative model, respectively.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{6}{c}{**Generative Model**} \\ \hline
**Name** & **Kernel Size** & **Stride** & **Pad** & **Norm** & **Activation** \\ \hline Conv1 & \(3\times 3\times 3\times 64\) & 1 & 1 & - & ReLU \\ Conv2 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv3 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv4 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv5 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv6 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv7 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv8 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv9 & \(3\times 3\times 64\times 64\) & 1 & 1 & BN & ReLU \\ Conv10 & \(3\times 3\times 64\times 3\) & 1 & 1 & - & ReLU \\ \hline \hline \end{tabular}
\end{table}
Table 1: Generative model network architectures. Conv means convolutional layer; BN means batch normalization; ReLU means rectified linear unit.  
There are only convolutional layers in our networks. Let \(X\) be the input. The basic convolutional layers are expressed as:  
\[F(X)=W_{k}*X+B_{k}\,. \tag{3}\]  
In Equation (3), \(W_{k}\) represents the weights of the filters and \(B_{k}\) represents the biases, while \(*\) denotes convolution operation. The Rectified Linear Unit (ReLU) is the activation function behind the convolutional layer, which is defined as:  
\[f(x)=\max(0,x)\,. \tag{4}\]  
where \(x\) is the input to a neuron. Combining Equations (3) and (4), a convolutional layer followed by a ReLU can be expressed as:  
\[F(X)=\max(0,W_{k}*X+B_{k})\,. \tag{5}\]  
We also used Batch Normalization (BN) in our networks. BN is a normalization strategy that makes the distribution of layers' input consistent at the output of layers; it eliminates the effect of the internal covariant shift. Covariate shift would amplify permutation at the deeper layers. If it happens, the inputs of the activation function will stay in the saturated region. In that region, the gradient will be very small, and the phenomenon of the vanishing gradient would happen and stop neural network training. The BN layer can be applied to any input of layers. Our generative and discriminative model is the combination of convolutional layers, ReLU and BN.