# Design of Painting Art Style Rendering System Based on Convolutional Neural Network

Authors: Xingyu Xie; Bin Lv
Journal: Scientific Programming
 (2021)

---

## Abstract

Convolutional Neural Network- (CNN-) based GAN models mainly suffer from problems such as data set limitation and rendering efficiency in the segmentation and rendering of painting art. In order to solve these problems, this paper uses the improved cycle generative adversarial network (CycleGAN) to render the current image style. This method replaces the deep residual network (ResNet) of the original network generator with a dense connected convolutional network (DenseNet) and uses the perceptual loss function for adversarial training. The painting art style rendering system built in this paper is based on perceptual adversarial network (PAN) for the improved CycleGAN that suppresses the limitation of the network model on paired samples. The proposed method also improves the quality of the image generated by the artistic style of painting and further improves the stability and speeds up the network convergence speed. Experiments were conducted on the painting art style rendering system based on the proposed model. Experimental results have shown that the image style rendering method based on the perceptual adversarial error to improve the CycleGAN + PAN model can achieve better results. The PSNR value of the generated image is increased by 6.27% on average, and the SSIM values are all increased by about 10%. Therefore, the improved CycleGAN + PAN image painting art style rendering method produces better painting art style images, which has strong application value.


---

## Body

## 1. Introduction

In recent years, deep learning has been widely used in many fields such as medical imaging [1], remote sensing [2], and three-dimensional modeling [3] and has played an important role in promoting the application of artificial intelligence in multiple industries. In order to discover useful macro information in the data, the purpose of deep learning is to combine low-level features to form more abstract features with strong representation ability. There are many types of image styles, such as oil painting, ink painting, sketch painting, etc., and applying these styles to a given image increases the diversified presentation of the image. The use of deep learning methods for style rendering is one of the hotspots in the field of image research. Image style rendering is an image conversion method based on deep learning, which can be widely used in image processing, computer picture synthesis, and computer vision. The original image style rendering is based on the optimization method proposed in [4], which uses the backpropagation of the convolutional neural network (CNN) and uses pixel-by-pixel comparison to obtain the optimal image conversion model.Style rendering is a processing method for rendering the semantic content of an image in different styles. The style is extracted from the specified style image templatex, and the extracted style features are mapped to y through mapping T without destroying the content of the image y. y performs style conversion; that is, it realizes the image style rendering of Tx≈y. The study in [5] combined deep learning and texture generation methods, using CNN to represent style images as content feature and style feature. The high-level convolutional layer of CNN model extracts global features of image content, and the low-level convolutional layer describes the image style. This method can generate a more objective artistic style effect. However, this method has slow convergence speed, long rendering time, and sometimes poor style expression. Image stylization methods based on model iteration [6] include two types of methods based on generative models and image reconstruction decoders. The study in [7] used the perceptual loss function to train the feedforward network and used the VGGNet trained on ImageNet to simplify the loss function calculation process and directly generate stylized images. The efficiency was improved by three orders of magnitude. At the same time, the pixel-by-pixel difference loss function is improved to a perceptual loss function, and super-resolution reconstruction is efficient, but this method requires artificially constructing a complex loss function. The study in [8] proposed an image stylization method based on a conditional adversarial network, which does not require artificial construction of loss functions and mapping functions, which simplifies the image stylization process.The network structure of the generative adversarial network (GAN) does not need to construct a complex loss function and achieves the global optimum through the mini-max game process of the generatorG and the discriminator D [9]. The conditional generative adversarial network (CGAN) uses different learning models. The GAN model is a mapping G:z⟶y from the random noise vector z to the output image y, while the conditional GAN model is to learn from the random noise vector z and observe the image. The generative model of the mapping is from x to the output image y, namely, G:x,z⟶y. The GAN style rendering effect is better than the CGAN model, but there is a problem that the coloring effect is not ideal.Academia researches have proposed many improved image style conversion methods based on the GAN model. The study in [10] proposed to add feature space loss (Lfea) and image space loss (L2) when training GAN. The study in [11] took the feature map difference of the middle layer of the network as the perceptual loss function and used the style transfer and superdivision images obtained by GAN to achieve real-time stylization and four times of super-resolution. The study in [12] obtained the super-resolution GAN model from low-resolution images to super-resolution images based on GAN. The study in [13] proposed a perceptual adversarial network (PAN) model that combines perceptual loss and GAN model and realized a variety of image style conversion applications. In the research of image style rendering, the method in [14] has been proved to be able to obtain good results, but its perceptual loss network is a pretrained VGG-16 network. The loss in the network is not easy to optimize, and the network is mainly used for classification. Although the ability to recognize the subject of the image is stronger, the ability to retain the background is weak. In view of this, this paper proposes a GAN model training method based on the PAN model. This method combines adversarial loss, content loss, and style loss into a new perceptual loss function, which can alternately update the loss network and the image style conversion network, thereby replacing the fixed loss network [15].In view of the abovementioned image stylization problems, this paper proposes to use a new type of cycle GAN model (CycleGAN) to achieve image style rendering and adopts a PAN model-based confrontation training method for CycleGAN model. The proposed method combines perceptual loss, content loss, and style loss into a new perceptual loss function, and the loss network and the image style conversion network can be alternately updated, thereby replacing the fixed loss network [16] and at the same time improving the original generator network structure. The experimental results prove that the proposed method can enhance the background definition of the image, make it closer to the original image in content and style, and at the same time increase the convergence speed, and the generated style rendering effect is more realistic.The rest of the paper is organized as follows: Section2 gives the related works about painting style rendering system based on CNN models, including conventional generative adversarial network, the perceptual adversarial network, and the image style rendering process. Section 3 illustrates the proposed improved CycleGAN + PAN model, image conversion network, and discriminator network. Section 4 gives the novel proposed combination of loss functions, such as content loss function, style loss function, and perceptual adversarial loss function. Moreover, the painting art segmentation rendering system is also given. Section 5 gives the painting art style rendering experiments and results on four different painting art forms. Section 6 concludes the model and results of the paper.


## 2. Related Works

### 2.1. Generative Adversarial Network

In the process of generating painting style images in the GAN model, the generatorG constantly learns and improves the ability to produce image data, and the discriminator D gradually improves the ability to discriminate data, and G and D reach a balance in the process of confronting the game. The adversarial relationship between G and DY in GAN is shown in the following equation:(1)minGmaxDYVDY,G=Ey⇔PdataylogDYy+Ez⇔Pzlog1−DGz,where y represents the image data, z is the noise input to the G network, and Gz is the generated image. Pdatay is the distribution of the real image data, Pzy represents the noise distribution of the input network G, and ⇔ represents the data obeys the distribution relationship. E is the mathematical expectation function. The deep convolutional GAN model (DCGAN) [17] is a combination of CNN and GAN, which generates images randomly, and cannot meet the requirements of style rendering between images. Therefore, this paper selects the CycleGAN model that is based on the CNN model to construct the painting art style rendering system.


### 2.2. Perceptual Adversarial Network

Inspired by the GAN model, on the basis of the existing perceptual loss [18] research, reference [19] proposed the perceptual adversarial network (PAN) model. PAN model combines perceptual loss and generative adversarial loss and conducts adversarial training between the image style conversion network and the discriminant network. Such model can continuously and automatically find the difference between the output and the real image that has not been reduced. Therefore, when the difference measured in the high-dimensional space is small, the hidden layer of the discriminant network model will be updated to a higher level, and the difference between the new output image and the real image will continue to be searched for until converging to the optimal solution.The innovation of PAN is that there is no longer a need for a complex loss function constructed based on human experience like traditional image models. This method automatically learns the mapping from input to output pictures through the adversarial network and applies it to the image conversion problem to achieve a generalization model. The PAN model is based on GAN model, which is combined with perceptual loss for adversarial training, and enhances the naturalness and realism of the image. The PAN model can realize a variety of image conversion tasks, such as image super-resolution, denoising, semantic segmentation, automatic completion, etc. Therefore, in this paper, we used the PAN model to improve the performance and efficiency of the CycleGAN model for rendering painting art style images.


### 2.3. Image Style Rendering Process

The image style rendering process is divided into two stages: training phase and performing phase [20]. In the training phase, the system selects a style map Ys and trains a transformation network model based on each style map. The samples of the content map X are continuously obtained from the training set through iterations. In each iteration, the conversion style network converts the content image X to Y and randomly transports X and Y to the discriminant network D. The network D judges between X and Y (content) and Ys and Y (style) through the adversarial loss function, and the difference between the two is fed back to the network T. T adjusts the weights and parameters and enters the next iteration. At the same time, the network D is continuously optimized to find more differences. The final purpose is to generate an image conversion network model with Y style. In the execution stage, the system inputs any content map into the well-trained Y style conversion model and converts the content map into a Y style effect image in real time, and the original content and structure remain unchanged. Therefore, this paper improves the image painting art style rendering network, as shown in Figure 1.Figure 1
Image painting art style rendering process.


## 2.1. Generative Adversarial Network

In the process of generating painting style images in the GAN model, the generatorG constantly learns and improves the ability to produce image data, and the discriminator D gradually improves the ability to discriminate data, and G and D reach a balance in the process of confronting the game. The adversarial relationship between G and DY in GAN is shown in the following equation:(1)minGmaxDYVDY,G=Ey⇔PdataylogDYy+Ez⇔Pzlog1−DGz,where y represents the image data, z is the noise input to the G network, and Gz is the generated image. Pdatay is the distribution of the real image data, Pzy represents the noise distribution of the input network G, and ⇔ represents the data obeys the distribution relationship. E is the mathematical expectation function. The deep convolutional GAN model (DCGAN) [17] is a combination of CNN and GAN, which generates images randomly, and cannot meet the requirements of style rendering between images. Therefore, this paper selects the CycleGAN model that is based on the CNN model to construct the painting art style rendering system.


## 2.2. Perceptual Adversarial Network

Inspired by the GAN model, on the basis of the existing perceptual loss [18] research, reference [19] proposed the perceptual adversarial network (PAN) model. PAN model combines perceptual loss and generative adversarial loss and conducts adversarial training between the image style conversion network and the discriminant network. Such model can continuously and automatically find the difference between the output and the real image that has not been reduced. Therefore, when the difference measured in the high-dimensional space is small, the hidden layer of the discriminant network model will be updated to a higher level, and the difference between the new output image and the real image will continue to be searched for until converging to the optimal solution.The innovation of PAN is that there is no longer a need for a complex loss function constructed based on human experience like traditional image models. This method automatically learns the mapping from input to output pictures through the adversarial network and applies it to the image conversion problem to achieve a generalization model. The PAN model is based on GAN model, which is combined with perceptual loss for adversarial training, and enhances the naturalness and realism of the image. The PAN model can realize a variety of image conversion tasks, such as image super-resolution, denoising, semantic segmentation, automatic completion, etc. Therefore, in this paper, we used the PAN model to improve the performance and efficiency of the CycleGAN model for rendering painting art style images.


## 2.3. Image Style Rendering Process

The image style rendering process is divided into two stages: training phase and performing phase [20]. In the training phase, the system selects a style map Ys and trains a transformation network model based on each style map. The samples of the content map X are continuously obtained from the training set through iterations. In each iteration, the conversion style network converts the content image X to Y and randomly transports X and Y to the discriminant network D. The network D judges between X and Y (content) and Ys and Y (style) through the adversarial loss function, and the difference between the two is fed back to the network T. T adjusts the weights and parameters and enters the next iteration. At the same time, the network D is continuously optimized to find more differences. The final purpose is to generate an image conversion network model with Y style. In the execution stage, the system inputs any content map into the well-trained Y style conversion model and converts the content map into a Y style effect image in real time, and the original content and structure remain unchanged. Therefore, this paper improves the image painting art style rendering network, as shown in Figure 1.Figure 1
Image painting art style rendering process.


## 3. Improved Image Style Rendering Network Structure

Although the VGG-16 loss network can be well trained with the ImageNet data set, the loss in the VGG-16 network is not easy to optimize. If a supervision item can be added to the hidden layer ofD to measure the effect of generation, the loss can be changed at any time when the network T is updated. Based on this setting, it is easier to get a better T.


### 3.1. Cycle Generative Adversarial Network

The CycleGAN model contains two generators and discriminators to realize the mutual mapping between imagesx and y [21]. Suppose the mapping of X⟶Y is the generator F, and the image sample x∈X generates an image similar to the sample y∈Y through Fx. CycleGAN uses the discriminator DY to determine whether the generated image is a real image or not. The loss function L between the generator F and the discriminator DY is defined as (2)LGANFX⟶Y,DY=Ey⇔PdataylogDYy+Ex⇔Pdataxlog1−DYFX⟶Yx,where ⇔, Pdata, and E are defined with equation (1). The single adversarial loss function L cannot guarantee that the learning function maps the input x to the expected output y; that is, it is not possible to train GAN only with equation (2). CycleGAN introduces the mapping from Y to X as GY⟶X, and the discriminator DX judges whether the image similar to x generated by y∈Y through Gy is true or not. We can deduce the mapping loss function L between GY⟶X and the discriminator DX similar to equation (2), as shown in the following equation:(3)LGANFX⟶Y,DX=Ex⇔PdataxlogDXx+Ey⇔Pdataylog1−DXGY⟶Xy.CycleGAN introduces the loss function of cycle consistency loss (CCL) and learns the two mappingsFX⟶Y and GY⟶X at the same time. After x is converted to y, from y to x^, the loss between x and x^ is calculated to avoid, so that it is possible that all images in X are mapped to the same image in Y. Figure 2 shows the training process of the cycle consistency loss function.Figure 2
CycleGAN backward consistency and backward consistency.Figure2 shows the forward consistency and backward consistency of CycleGAN model. The forward consistency and backward consistency constitute the overall consistency of CycleGAN model. The overall consistency is the key of the CycleGAN model. In this paper, we use FGy≈y and GFx≈x to define the cycle consistency loss LCCL as (4)LCCLFX⟶Y,GY⟶X=Ex⇔PdataxGY⟶XFX⟶Yx−x1+Ey⇔PdatayFX⟶YGY⟶Xy−y1.From equations (2)–(4), the objective function L′ of CycleGAN shown in equation (5) is obtained:(5)L′FX⟶Y,GY⟶X=LGANFX⟶Y,DY+LGANGY⟶X,DX+λCCLLCCLFX⟶Y,GY⟶X,where λCCL is the weight of adjusting the loss LCCL in the objective function.


### 3.2. Image Conversion Network

The image style conversion networkT improves on the network proposed in [22], and the image style conversion network structure is shown in Figure 3. The overall structure follows its deep residual network (ResNet), which has three convolutional layers and five residual blocks, as shown in Figure 3(a). Except that the output layer uses a scaled Tanh function to ensure that the output pixels are in range of [0, 255], all other nonresidual convolutional layers use ReLU activation functions. Since no pooling layer is used and stride convolution or microstride convolution is used for upsampling and downsampling, this operation not only reduces the parameters, but also maintains a large field of view and avoids excessive deformation of image objects [23].Figure 3
Image conversion network structure. (a) Artistic painting style rendering system model. (b) The residual structure used in the model.
(a)(b)Considering that each nonresidual convolutional layer in the network is followed by a spatial batch normalization to achieve accelerated convergence and the study in [24] showing that instance regularization is better than batch normalization, which can significantly improve performance, this paper uses instance normalization instead of batch normalization, as shown in Figure 3(b). It can be seen that the instance normalization is actually to modify the batches to 1, and the normalization is used on a single image instead of a batch of images. Therefore, the network improved by the instance normalization performs better in the test phase of painting art style rendering.


### 3.3. Discriminator Network

The discriminant networkD is a multilayer convolutional neural network, as shown in Figure 4. Behind each hidden layer are added the batch normalization and LeakyReLU linear activation functions. The first, fourth, sixth, and eighth layers are used to measure the perceptual adversarial loss between the generated image and the style image. The judgment network outputs a probability of the images, that is, whether the image is from the real data set (True) or generated by the style conversion network (Fake) [10].Figure 4
Discriminant network structure of the painting art style rendering system.


## 3.1. Cycle Generative Adversarial Network

The CycleGAN model contains two generators and discriminators to realize the mutual mapping between imagesx and y [21]. Suppose the mapping of X⟶Y is the generator F, and the image sample x∈X generates an image similar to the sample y∈Y through Fx. CycleGAN uses the discriminator DY to determine whether the generated image is a real image or not. The loss function L between the generator F and the discriminator DY is defined as (2)LGANFX⟶Y,DY=Ey⇔PdataylogDYy+Ex⇔Pdataxlog1−DYFX⟶Yx,where ⇔, Pdata, and E are defined with equation (1). The single adversarial loss function L cannot guarantee that the learning function maps the input x to the expected output y; that is, it is not possible to train GAN only with equation (2). CycleGAN introduces the mapping from Y to X as GY⟶X, and the discriminator DX judges whether the image similar to x generated by y∈Y through Gy is true or not. We can deduce the mapping loss function L between GY⟶X and the discriminator DX similar to equation (2), as shown in the following equation:(3)LGANFX⟶Y,DX=Ex⇔PdataxlogDXx+Ey⇔Pdataylog1−DXGY⟶Xy.CycleGAN introduces the loss function of cycle consistency loss (CCL) and learns the two mappingsFX⟶Y and GY⟶X at the same time. After x is converted to y, from y to x^, the loss between x and x^ is calculated to avoid, so that it is possible that all images in X are mapped to the same image in Y. Figure 2 shows the training process of the cycle consistency loss function.Figure 2
CycleGAN backward consistency and backward consistency.Figure2 shows the forward consistency and backward consistency of CycleGAN model. The forward consistency and backward consistency constitute the overall consistency of CycleGAN model. The overall consistency is the key of the CycleGAN model. In this paper, we use FGy≈y and GFx≈x to define the cycle consistency loss LCCL as (4)LCCLFX⟶Y,GY⟶X=Ex⇔PdataxGY⟶XFX⟶Yx−x1+Ey⇔PdatayFX⟶YGY⟶Xy−y1.From equations (2)–(4), the objective function L′ of CycleGAN shown in equation (5) is obtained:(5)L′FX⟶Y,GY⟶X=LGANFX⟶Y,DY+LGANGY⟶X,DX+λCCLLCCLFX⟶Y,GY⟶X,where λCCL is the weight of adjusting the loss LCCL in the objective function.


## 3.2. Image Conversion Network

The image style conversion networkT improves on the network proposed in [22], and the image style conversion network structure is shown in Figure 3. The overall structure follows its deep residual network (ResNet), which has three convolutional layers and five residual blocks, as shown in Figure 3(a). Except that the output layer uses a scaled Tanh function to ensure that the output pixels are in range of [0, 255], all other nonresidual convolutional layers use ReLU activation functions. Since no pooling layer is used and stride convolution or microstride convolution is used for upsampling and downsampling, this operation not only reduces the parameters, but also maintains a large field of view and avoids excessive deformation of image objects [23].Figure 3
Image conversion network structure. (a) Artistic painting style rendering system model. (b) The residual structure used in the model.
(a)(b)Considering that each nonresidual convolutional layer in the network is followed by a spatial batch normalization to achieve accelerated convergence and the study in [24] showing that instance regularization is better than batch normalization, which can significantly improve performance, this paper uses instance normalization instead of batch normalization, as shown in Figure 3(b). It can be seen that the instance normalization is actually to modify the batches to 1, and the normalization is used on a single image instead of a batch of images. Therefore, the network improved by the instance normalization performs better in the test phase of painting art style rendering.


## 3.3. Discriminator Network

The discriminant networkD is a multilayer convolutional neural network, as shown in Figure 4. Behind each hidden layer are added the batch normalization and LeakyReLU linear activation functions. The first, fourth, sixth, and eighth layers are used to measure the perceptual adversarial loss between the generated image and the style image. The judgment network outputs a probability of the images, that is, whether the image is from the real data set (True) or generated by the style conversion network (Fake) [10].Figure 4
Discriminant network structure of the painting art style rendering system.


## 4. Painting Art Style Rendering Based on the CycleGAN Model and Perceptual Confrontation Loss

### 4.1. Perceptual Adversarial Loss

Although the PAN model [25] has been proven to measure the difference between images from a high-dimensional visual perception level, how to extract effective feature differences through the hidden layer is still an open question. The key to the problem is how to minimize the difference between the generated image and the real image in each high-dimensional level [26]. For this reason, this paper combines the PAN model and the perceptual loss proposed in [27] and defines the perceptual adversarial loss to consist of content feature loss, style loss, and adversarial loss. In the N-layer discriminant network, the image features are regarded as feature maps of N dimensions. If the size of each layer of feature maps is Hi×Wi, the size of the feature map is Ci×Hi×Wi, where C represents the number of feature maps. Since each grid position of the image can be regarded as an independent sample, it can capture the key features. Perceptual adversarial loss is the weighted sum of content loss and style loss. It can punish the difference between the generated image and the style image when the first, fourth, sixth, and eighth hidden layers of the network are dynamically updated, so that the generated image has the best excellent content and style synthesis effect.


#### 4.1.1. Content Loss Function

The content loss functionPi uses Manhattan distance to calculate the image space loss L2 of the hidden generated image Y and the real painting art style image X as follows:(6)Pi=1NHiY−HiX,where Hi represents the L2 value of the i-th hidden layer of the discrimination network.Based on the above definition, multiple levels of content loss can be expressed as(7)LcontentX,Y=∑i=1NλiPiX,Y,where λi represents the balance factor of discriminating the N hidden layers i of the network. By minimizing the perceptual loss function Lcontent, the generated image and the content image have a similar content structure.


#### 4.1.2. Style Loss Function

Considering that the style loss function is used to penalize the deviation of the output image in style, including color and texture, this paper uses the style reconstruction method [28], which is obtained by the distance between the output image and the style image Gram matrix. Set φix as the feature map of the i-th hidden layer, so that the shape of φix is Ci×Hi×Wi, and the style loss value of the feature map of the i-th layer of the discriminant network can be expressed as(8)GramiY=φiY∗φiYTCi×Hi×Wi,GiYs,Y=GramiY−GramiYs.In order to represent the style reconstruction from multiple levels, this paper definesGiYs,Y as a set of losses (sum of losses for each layer) as(9)LstyleYs,Y=∑i=1nGiYs,Y.


#### 4.1.3. Perceptual Adversarial Loss Function

The overall perceptual loss consists of a combination of content loss and style loss as a linear function, which can be defined as follows:(10)LstyleYs,Y=∑i=1nGiYs,Y,where λc,λs are the weight parameters set based on human experience. The style conversion network T and the discriminant network D are alternately optimized based on the overall perceptual loss value. The alternate optimization between such two network models is based on the method of PAN [29] model to realize the min-max confrontation. For the generated image Y, content image X, and style image Ys, the loss function of network T and the loss function of network D are defined as follows:(11)LT=ln1−DTx+Lperc,LD=−lnDy−ln1−DTx+m−Lperc+.We set a positive boundary valuem in equation (11). Because the positive boundary value m can make the third term of LD achieve gradient descent, we minimize LT through the parameters of the network T, which can maximize the second and third terms of LD at the same time. When LT is less than m, the loss function LD will update the discriminant network to a new high-dimensional level to calculate the remaining difference. Therefore, by perceiving against loss, the diversified differences between the generated image and the style image can be continuously perceived and explored.


### 4.2. Painting Art Segmentation Rendering System

The generator of the traditional CycleGAN model adopts ResNet [21]. The generator is a fully convolutional connection type [30] and consists of an encoder, a converter, and a decoder. On the basis of the perceptual loss error, this paper improves the traditional CycleGAN network and uses CycleGAN with the DensNet [31] generator to achieve the style rendering of the image, as shown in Figure 5.Figure 5
Generation network structure of the painting art style rendering system.


#### 4.2.1. Encoder

The CNN is used to extract features from the input image. The number of filters in the first convolutional layer (Conv_Layer_1) is set to 64. The output of the first convolutional layer is sent to the second convolutional layer (Conv_Layer_2) to continue to extract features, and the number of filters in the second convolutional layer is set to 128. The connection and data transmission method of the second convolutional layer and the third convolutional layer (Conv_Layer_3) is similar to that of the first and second convolutional layers. The number of filters in the third convolutional layer is set to 256. The image size input to the encoder is 256 × 256, and the encoder extracts 256 feature vectors with a size of 64 × 64.


#### 4.2.2. Converter

The purpose of converter is to combine the different extracted features and use these features to determine how to convert the feature vector of the image from theX domain (image style template) to the feature vector of the Y domain (generated image). The original CycleGAN network converter uses a 6-layer ResNet block to convert the feature vector, as shown in the following equation:(12)fl=Hlfl−1+fl−1,where Hl is a nonlinear transformation function. fl−1 and fl are the input features and style conversion output features of the first layer as a ResNet block, including batch normalization layer [25], convolutional layer, and ReLU layer [32]. The converter designed in this paper uses DenseNet model to replace the traditional ResNet model. DenseNet model can reduce the disappearance of the gradient, enhance the feature transfer ability, and reduce the number of parameters to a certain extent. DenseNet model connects modules to each other, which improves the coupling capability of information flow between modules. The l-th module of the DenseNet model accepts the feature mapping from the previous module, as shown in the following equation:(13)fl=Hlf0,f1,…,fl−1,where f0,f1,…,fl−1 is formed by the series of generation features of 0,1,…,l−1 layers. This paper integrates the DenseNet module in the converter to reduce model parameters while avoiding overfitting and reducing the amount of calculation.


#### 4.2.3. Decoder

The decoder process is opposite to that of the encoder, using a three-level deconvolution layer (Deconv_Layer) [33] to restore the low-level features of the image from the feature vector step by step, until the image is generated. The detailed internal composition of encoder, converter, and decoder is shown in Table 1.Table 1
Internal structure of the generation network of the painting art style rendering system.
ModulesDescriptions in detailEncoderConvolutional kernelWays of network connectingNumber of filtersStride of operations17∗ 7Conv, BN, ReLU64125∗ 5Conv, BN, ReLU128133∗ 3Conv, BN, ReLU2562TransformerModules1Dense block 1………5Dense block 5DecoderConvolutional kernelWays of network connectingNumber of filtersStride of operations13∗ 3De-conv, BN, ReLU1281/223∗ 3De-conv, BN, ReLU641/237∗ 7De-conv, BN, ReLU31


#### 4.2.4. Discriminator

The discriminator of the improved network adopts the PatchGAN [34] classifier. In this model, we input the image to the discriminator, which discriminates whether the image is the original image or the image generated by the generator. The image input to the discriminator is divided into multiple 70 × 70 image blocks. In the discriminator's discrimination process, the network convolves the input image layer by layer, discriminates each image block through the one-dimensional output convolutional layer, and takes the average of the judgment results of all image blocks as the image judgment result.


## 4.1. Perceptual Adversarial Loss

Although the PAN model [25] has been proven to measure the difference between images from a high-dimensional visual perception level, how to extract effective feature differences through the hidden layer is still an open question. The key to the problem is how to minimize the difference between the generated image and the real image in each high-dimensional level [26]. For this reason, this paper combines the PAN model and the perceptual loss proposed in [27] and defines the perceptual adversarial loss to consist of content feature loss, style loss, and adversarial loss. In the N-layer discriminant network, the image features are regarded as feature maps of N dimensions. If the size of each layer of feature maps is Hi×Wi, the size of the feature map is Ci×Hi×Wi, where C represents the number of feature maps. Since each grid position of the image can be regarded as an independent sample, it can capture the key features. Perceptual adversarial loss is the weighted sum of content loss and style loss. It can punish the difference between the generated image and the style image when the first, fourth, sixth, and eighth hidden layers of the network are dynamically updated, so that the generated image has the best excellent content and style synthesis effect.


### 4.1.1. Content Loss Function

The content loss functionPi uses Manhattan distance to calculate the image space loss L2 of the hidden generated image Y and the real painting art style image X as follows:(6)Pi=1NHiY−HiX,where Hi represents the L2 value of the i-th hidden layer of the discrimination network.Based on the above definition, multiple levels of content loss can be expressed as(7)LcontentX,Y=∑i=1NλiPiX,Y,where λi represents the balance factor of discriminating the N hidden layers i of the network. By minimizing the perceptual loss function Lcontent, the generated image and the content image have a similar content structure.


### 4.1.2. Style Loss Function

Considering that the style loss function is used to penalize the deviation of the output image in style, including color and texture, this paper uses the style reconstruction method [28], which is obtained by the distance between the output image and the style image Gram matrix. Set φix as the feature map of the i-th hidden layer, so that the shape of φix is Ci×Hi×Wi, and the style loss value of the feature map of the i-th layer of the discriminant network can be expressed as(8)GramiY=φiY∗φiYTCi×Hi×Wi,GiYs,Y=GramiY−GramiYs.In order to represent the style reconstruction from multiple levels, this paper definesGiYs,Y as a set of losses (sum of losses for each layer) as(9)LstyleYs,Y=∑i=1nGiYs,Y.


### 4.1.3. Perceptual Adversarial Loss Function

The overall perceptual loss consists of a combination of content loss and style loss as a linear function, which can be defined as follows:(10)LstyleYs,Y=∑i=1nGiYs,Y,where λc,λs are the weight parameters set based on human experience. The style conversion network T and the discriminant network D are alternately optimized based on the overall perceptual loss value. The alternate optimization between such two network models is based on the method of PAN [29] model to realize the min-max confrontation. For the generated image Y, content image X, and style image Ys, the loss function of network T and the loss function of network D are defined as follows:(11)LT=ln1−DTx+Lperc,LD=−lnDy−ln1−DTx+m−Lperc+.We set a positive boundary valuem in equation (11). Because the positive boundary value m can make the third term of LD achieve gradient descent, we minimize LT through the parameters of the network T, which can maximize the second and third terms of LD at the same time. When LT is less than m, the loss function LD will update the discriminant network to a new high-dimensional level to calculate the remaining difference. Therefore, by perceiving against loss, the diversified differences between the generated image and the style image can be continuously perceived and explored.


## 4.1.1. Content Loss Function

The content loss functionPi uses Manhattan distance to calculate the image space loss L2 of the hidden generated image Y and the real painting art style image X as follows:(6)Pi=1NHiY−HiX,where Hi represents the L2 value of the i-th hidden layer of the discrimination network.Based on the above definition, multiple levels of content loss can be expressed as(7)LcontentX,Y=∑i=1NλiPiX,Y,where λi represents the balance factor of discriminating the N hidden layers i of the network. By minimizing the perceptual loss function Lcontent, the generated image and the content image have a similar content structure.


## 4.1.2. Style Loss Function

Considering that the style loss function is used to penalize the deviation of the output image in style, including color and texture, this paper uses the style reconstruction method [28], which is obtained by the distance between the output image and the style image Gram matrix. Set φix as the feature map of the i-th hidden layer, so that the shape of φix is Ci×Hi×Wi, and the style loss value of the feature map of the i-th layer of the discriminant network can be expressed as(8)GramiY=φiY∗φiYTCi×Hi×Wi,GiYs,Y=GramiY−GramiYs.In order to represent the style reconstruction from multiple levels, this paper definesGiYs,Y as a set of losses (sum of losses for each layer) as(9)LstyleYs,Y=∑i=1nGiYs,Y.


## 4.1.3. Perceptual Adversarial Loss Function

The overall perceptual loss consists of a combination of content loss and style loss as a linear function, which can be defined as follows:(10)LstyleYs,Y=∑i=1nGiYs,Y,where λc,λs are the weight parameters set based on human experience. The style conversion network T and the discriminant network D are alternately optimized based on the overall perceptual loss value. The alternate optimization between such two network models is based on the method of PAN [29] model to realize the min-max confrontation. For the generated image Y, content image X, and style image Ys, the loss function of network T and the loss function of network D are defined as follows:(11)LT=ln1−DTx+Lperc,LD=−lnDy−ln1−DTx+m−Lperc+.We set a positive boundary valuem in equation (11). Because the positive boundary value m can make the third term of LD achieve gradient descent, we minimize LT through the parameters of the network T, which can maximize the second and third terms of LD at the same time. When LT is less than m, the loss function LD will update the discriminant network to a new high-dimensional level to calculate the remaining difference. Therefore, by perceiving against loss, the diversified differences between the generated image and the style image can be continuously perceived and explored.


## 4.2. Painting Art Segmentation Rendering System

The generator of the traditional CycleGAN model adopts ResNet [21]. The generator is a fully convolutional connection type [30] and consists of an encoder, a converter, and a decoder. On the basis of the perceptual loss error, this paper improves the traditional CycleGAN network and uses CycleGAN with the DensNet [31] generator to achieve the style rendering of the image, as shown in Figure 5.Figure 5
Generation network structure of the painting art style rendering system.


### 4.2.1. Encoder

The CNN is used to extract features from the input image. The number of filters in the first convolutional layer (Conv_Layer_1) is set to 64. The output of the first convolutional layer is sent to the second convolutional layer (Conv_Layer_2) to continue to extract features, and the number of filters in the second convolutional layer is set to 128. The connection and data transmission method of the second convolutional layer and the third convolutional layer (Conv_Layer_3) is similar to that of the first and second convolutional layers. The number of filters in the third convolutional layer is set to 256. The image size input to the encoder is 256 × 256, and the encoder extracts 256 feature vectors with a size of 64 × 64.


### 4.2.2. Converter

The purpose of converter is to combine the different extracted features and use these features to determine how to convert the feature vector of the image from theX domain (image style template) to the feature vector of the Y domain (generated image). The original CycleGAN network converter uses a 6-layer ResNet block to convert the feature vector, as shown in the following equation:(12)fl=Hlfl−1+fl−1,where Hl is a nonlinear transformation function. fl−1 and fl are the input features and style conversion output features of the first layer as a ResNet block, including batch normalization layer [25], convolutional layer, and ReLU layer [32]. The converter designed in this paper uses DenseNet model to replace the traditional ResNet model. DenseNet model can reduce the disappearance of the gradient, enhance the feature transfer ability, and reduce the number of parameters to a certain extent. DenseNet model connects modules to each other, which improves the coupling capability of information flow between modules. The l-th module of the DenseNet model accepts the feature mapping from the previous module, as shown in the following equation:(13)fl=Hlf0,f1,…,fl−1,where f0,f1,…,fl−1 is formed by the series of generation features of 0,1,…,l−1 layers. This paper integrates the DenseNet module in the converter to reduce model parameters while avoiding overfitting and reducing the amount of calculation.


### 4.2.3. Decoder

The decoder process is opposite to that of the encoder, using a three-level deconvolution layer (Deconv_Layer) [33] to restore the low-level features of the image from the feature vector step by step, until the image is generated. The detailed internal composition of encoder, converter, and decoder is shown in Table 1.Table 1
Internal structure of the generation network of the painting art style rendering system.
ModulesDescriptions in detailEncoderConvolutional kernelWays of network connectingNumber of filtersStride of operations17∗ 7Conv, BN, ReLU64125∗ 5Conv, BN, ReLU128133∗ 3Conv, BN, ReLU2562TransformerModules1Dense block 1………5Dense block 5DecoderConvolutional kernelWays of network connectingNumber of filtersStride of operations13∗ 3De-conv, BN, ReLU1281/223∗ 3De-conv, BN, ReLU641/237∗ 7De-conv, BN, ReLU31


### 4.2.4. Discriminator

The discriminator of the improved network adopts the PatchGAN [34] classifier. In this model, we input the image to the discriminator, which discriminates whether the image is the original image or the image generated by the generator. The image input to the discriminator is divided into multiple 70 × 70 image blocks. In the discriminator's discrimination process, the network convolves the input image layer by layer, discriminates each image block through the one-dimensional output convolutional layer, and takes the average of the judgment results of all image blocks as the image judgment result.


## 4.2.1. Encoder

The CNN is used to extract features from the input image. The number of filters in the first convolutional layer (Conv_Layer_1) is set to 64. The output of the first convolutional layer is sent to the second convolutional layer (Conv_Layer_2) to continue to extract features, and the number of filters in the second convolutional layer is set to 128. The connection and data transmission method of the second convolutional layer and the third convolutional layer (Conv_Layer_3) is similar to that of the first and second convolutional layers. The number of filters in the third convolutional layer is set to 256. The image size input to the encoder is 256 × 256, and the encoder extracts 256 feature vectors with a size of 64 × 64.


## 4.2.2. Converter

The purpose of converter is to combine the different extracted features and use these features to determine how to convert the feature vector of the image from theX domain (image style template) to the feature vector of the Y domain (generated image). The original CycleGAN network converter uses a 6-layer ResNet block to convert the feature vector, as shown in the following equation:(12)fl=Hlfl−1+fl−1,where Hl is a nonlinear transformation function. fl−1 and fl are the input features and style conversion output features of the first layer as a ResNet block, including batch normalization layer [25], convolutional layer, and ReLU layer [32]. The converter designed in this paper uses DenseNet model to replace the traditional ResNet model. DenseNet model can reduce the disappearance of the gradient, enhance the feature transfer ability, and reduce the number of parameters to a certain extent. DenseNet model connects modules to each other, which improves the coupling capability of information flow between modules. The l-th module of the DenseNet model accepts the feature mapping from the previous module, as shown in the following equation:(13)fl=Hlf0,f1,…,fl−1,where f0,f1,…,fl−1 is formed by the series of generation features of 0,1,…,l−1 layers. This paper integrates the DenseNet module in the converter to reduce model parameters while avoiding overfitting and reducing the amount of calculation.


## 4.2.3. Decoder

The decoder process is opposite to that of the encoder, using a three-level deconvolution layer (Deconv_Layer) [33] to restore the low-level features of the image from the feature vector step by step, until the image is generated. The detailed internal composition of encoder, converter, and decoder is shown in Table 1.Table 1
Internal structure of the generation network of the painting art style rendering system.
ModulesDescriptions in detailEncoderConvolutional kernelWays of network connectingNumber of filtersStride of operations17∗ 7Conv, BN, ReLU64125∗ 5Conv, BN, ReLU128133∗ 3Conv, BN, ReLU2562TransformerModules1Dense block 1………5Dense block 5DecoderConvolutional kernelWays of network connectingNumber of filtersStride of operations13∗ 3De-conv, BN, ReLU1281/223∗ 3De-conv, BN, ReLU641/237∗ 7De-conv, BN, ReLU31


## 4.2.4. Discriminator

The discriminator of the improved network adopts the PatchGAN [34] classifier. In this model, we input the image to the discriminator, which discriminates whether the image is the original image or the image generated by the generator. The image input to the discriminator is divided into multiple 70 × 70 image blocks. In the discriminator's discrimination process, the network convolves the input image layer by layer, discriminates each image block through the one-dimensional output convolutional layer, and takes the average of the judgment results of all image blocks as the image judgment result.


## 5. Simulation Experiment and Result Analysis

### 5.1. Experiment Initialization

The experimental environment used in this paper is as follows: CPU: Intel(R) Core(TM) i7-9700K@3.20 GHZ, 16 GB memory, operating system: 64 bit Windows 10, deep learning framework: Pytorch 1.10. This open source framework is mainly written in Python and can be used across platforms to implement various common CNN models and GAN models.In order to verify the feasibility and effectiveness, the following model training and testing data sets are constructed for the perceptual adversarial loss error and the improved CycleGAN + PAN model. In the network training process, 983 oil painting images, 933 sketch painting images, and 1000 260 × 260 RGB ink wash paintings were selected as the image painting art style rendering data set. The training data used 1142 architectural images. There is no designated corresponding pairing relationship between different types of data sets during the training process. As shown in Figure6, they are CycleGAN, CycleGAN + PAN, and the image style rendering algorithm based on the improved CycleGAN + PAN that is proposed in this paper, Ground-truth ⟶ Oil painting, Ground-truth ⟶ Sketch painting, and Ground-truth ⟶ Ink wash painting for three styles of rendering model training.Figure 6
An example display of the experimental data set of painting art style rendering. (a) Ground-truth. (b) Oil painting. (c) Sketch painting. (d) Ink wash painting.
(a)(b)(c)(d)


### 5.2. Qualitative Results Analysis

In order to verify the feasibility and effectiveness of the rendering model proposed in this paper, the objective functions of CycleGAN, CycleGAN + PAN, and the improved CycleGAN + PAN are used in the experiment to perform image painting art style rendering experiments. The experimental results are shown in Figure7. The first column in Figure 7 is the original image, the second column is the style image, and the third, fourth, and fifth columns are, respectively, experimental results of image style rendering for the CycleGAN model, CycleGAN + PAN model, and the improved CycleGAN + PAN model.Figure 7
Experimental results of painting art style rendering. (a) Oil painting-based painting art style rendering. (b) Sketch painting-based painting art style rendering. (c) Ink wash painting-based painting art style rendering.
(a)(b)(c)The first row, the second row, and the third row are experiments of three comparative models using oil painting, ink painting, and sketch painting as rendering styles. From the comparison of experimental results, it can be found that, under the same number of iterations, the proposed model can achieve painting art style rendering faster and achieve more realistic style rendering effects. Under the same experimental configuration and settings, this paper compares the generation results after 50,000 iterations. Table2 shows the image style rendering results of CycleGAN, CycleGAN + PAN, and the proposed model using SSIM and PSNR image quality evaluation indicators for three different styles’ rendering.Table 2
Quality evaluation of the painting art style rendering experiment results.
Different stylesIndexesCycleGANCycleGAN + PANProposed methodOil paintingSSIM0.54510.72160.7456PSNR18.621220.451520.4425Sketch paintingSSIM0.63150.64180.6987PSNR18.518718.925120.2147Ink wash paintingSSIM0.65420.67810.7121PSNR17.815418.121419.0241It can be seen from Table2 that, in the results of the oil painting style rendering experiment, the PSNR of CycleGAN + PAN model is slightly higher than the proposed model. However, in the experimental results of the style rendering of ink and drawing styles, the SSIM and PSNR of the improved painting art style rendering of the proposed model are higher than other models.


### 5.3. Quantitative Results Analysis

CycleGAN introduces a cycle consistency loss function on the basis of traditional GAN unidirectional mapping and avoids model collapse to a certain extent through bidirectional mapping. The improved CycleGAN + PAN model proposed in this paper introduces the same mapping loss and perceptual loss function. Compared with CycleGAN + PAN model, the image style rendering process is more stable and the model is not easy to collapse. Figure8 shows the comparative experimental results of some detailed painting art rendering between CycleGAN + PAN and the improved CycleGAN + PAN model proposed in this paper. Under the same number of iterations, the results of image style rendering are compared for two hard situations, which are marked as red box in Figure 8.Figure 8
Effect of the abnormal situation of painting art style rendering between the original CycleGAN + PAN model and the improved CycleGAN + PAN model proposed in this paper.It can be seen from Figure8 that CycleGAN + PAN uses two-way mapping to reduce the probability of model collapse, but instability sometimes occurs during training. As shown in Figure 8, there are black blocks in the image style rendering with the CycleGAN + PAN training model, and the style is lost to a certain extent. The third column shows the style rendering image generated by the improved CycleGAN + PAN. The result is that the image rendering style is well preserved, and there is no model crash. Therefore, the improved CycleGAN + PAN model proposed in the paper has good training conditions and is not easy to collapse under the same number of iterations.The purpose of considering the painting art style rendering method is to achieve style loss and content loss, that is, to minimize equation (10). Therefore, the method proposed in this paper can be compared with the original CycleGAN + PAN [21] model in terms of measuring the extent of the successful reduction of the loss function. This paper uses the three compared methods to train 50, 100, and 200 images to record the value of equation (10) for each group of training. The loss function values of the three different sizes of images are shown in Figure 9. It can be seen from Figure 9 that, no matter for low-resolution or high-resolution images, the perceptual adversarial network can more effectively minimize the loss function.Figure 9
The comparison results of the loss function of the improved model in this article, the classic CycleGAN + PAN model, and the CycleGAN model. (a) 256∗ 256. (b) 515 ∗ 512. (c) 1024 ∗ 1024.
(a)(b)(c)The method proposed in this paper improves the convergence speed more than the original CycleGAN model. As shown in Figure10, CycleGAN model, CycleGAN + PAN model, and improved CycleGAN + PAN model are trained on the same data set, and the inception score is used to measure the quality and variety of style rendering images. In the case of the same number of iterations, the higher the evaluation score is, the better the quality of the style rendering image is and the faster the convergence speed is obtained. In the experimental results of the three models, the improved CycleGAN + PAN model can converge faster and obtain higher evaluation scores. Through the convergence speed comparison experiment, we can see that the improved CycleGAN + PAN model proposed in this paper has a faster convergence speed and better stylized image quality.Figure 10
Comparison of the efficiency of the three comparison models in the rendering of artistic style of painting.


## 5.1. Experiment Initialization

The experimental environment used in this paper is as follows: CPU: Intel(R) Core(TM) i7-9700K@3.20 GHZ, 16 GB memory, operating system: 64 bit Windows 10, deep learning framework: Pytorch 1.10. This open source framework is mainly written in Python and can be used across platforms to implement various common CNN models and GAN models.In order to verify the feasibility and effectiveness, the following model training and testing data sets are constructed for the perceptual adversarial loss error and the improved CycleGAN + PAN model. In the network training process, 983 oil painting images, 933 sketch painting images, and 1000 260 × 260 RGB ink wash paintings were selected as the image painting art style rendering data set. The training data used 1142 architectural images. There is no designated corresponding pairing relationship between different types of data sets during the training process. As shown in Figure6, they are CycleGAN, CycleGAN + PAN, and the image style rendering algorithm based on the improved CycleGAN + PAN that is proposed in this paper, Ground-truth ⟶ Oil painting, Ground-truth ⟶ Sketch painting, and Ground-truth ⟶ Ink wash painting for three styles of rendering model training.Figure 6
An example display of the experimental data set of painting art style rendering. (a) Ground-truth. (b) Oil painting. (c) Sketch painting. (d) Ink wash painting.
(a)(b)(c)(d)


## 5.2. Qualitative Results Analysis

In order to verify the feasibility and effectiveness of the rendering model proposed in this paper, the objective functions of CycleGAN, CycleGAN + PAN, and the improved CycleGAN + PAN are used in the experiment to perform image painting art style rendering experiments. The experimental results are shown in Figure7. The first column in Figure 7 is the original image, the second column is the style image, and the third, fourth, and fifth columns are, respectively, experimental results of image style rendering for the CycleGAN model, CycleGAN + PAN model, and the improved CycleGAN + PAN model.Figure 7
Experimental results of painting art style rendering. (a) Oil painting-based painting art style rendering. (b) Sketch painting-based painting art style rendering. (c) Ink wash painting-based painting art style rendering.
(a)(b)(c)The first row, the second row, and the third row are experiments of three comparative models using oil painting, ink painting, and sketch painting as rendering styles. From the comparison of experimental results, it can be found that, under the same number of iterations, the proposed model can achieve painting art style rendering faster and achieve more realistic style rendering effects. Under the same experimental configuration and settings, this paper compares the generation results after 50,000 iterations. Table2 shows the image style rendering results of CycleGAN, CycleGAN + PAN, and the proposed model using SSIM and PSNR image quality evaluation indicators for three different styles’ rendering.Table 2
Quality evaluation of the painting art style rendering experiment results.
Different stylesIndexesCycleGANCycleGAN + PANProposed methodOil paintingSSIM0.54510.72160.7456PSNR18.621220.451520.4425Sketch paintingSSIM0.63150.64180.6987PSNR18.518718.925120.2147Ink wash paintingSSIM0.65420.67810.7121PSNR17.815418.121419.0241It can be seen from Table2 that, in the results of the oil painting style rendering experiment, the PSNR of CycleGAN + PAN model is slightly higher than the proposed model. However, in the experimental results of the style rendering of ink and drawing styles, the SSIM and PSNR of the improved painting art style rendering of the proposed model are higher than other models.


## 5.3. Quantitative Results Analysis

CycleGAN introduces a cycle consistency loss function on the basis of traditional GAN unidirectional mapping and avoids model collapse to a certain extent through bidirectional mapping. The improved CycleGAN + PAN model proposed in this paper introduces the same mapping loss and perceptual loss function. Compared with CycleGAN + PAN model, the image style rendering process is more stable and the model is not easy to collapse. Figure8 shows the comparative experimental results of some detailed painting art rendering between CycleGAN + PAN and the improved CycleGAN + PAN model proposed in this paper. Under the same number of iterations, the results of image style rendering are compared for two hard situations, which are marked as red box in Figure 8.Figure 8
Effect of the abnormal situation of painting art style rendering between the original CycleGAN + PAN model and the improved CycleGAN + PAN model proposed in this paper.It can be seen from Figure8 that CycleGAN + PAN uses two-way mapping to reduce the probability of model collapse, but instability sometimes occurs during training. As shown in Figure 8, there are black blocks in the image style rendering with the CycleGAN + PAN training model, and the style is lost to a certain extent. The third column shows the style rendering image generated by the improved CycleGAN + PAN. The result is that the image rendering style is well preserved, and there is no model crash. Therefore, the improved CycleGAN + PAN model proposed in the paper has good training conditions and is not easy to collapse under the same number of iterations.The purpose of considering the painting art style rendering method is to achieve style loss and content loss, that is, to minimize equation (10). Therefore, the method proposed in this paper can be compared with the original CycleGAN + PAN [21] model in terms of measuring the extent of the successful reduction of the loss function. This paper uses the three compared methods to train 50, 100, and 200 images to record the value of equation (10) for each group of training. The loss function values of the three different sizes of images are shown in Figure 9. It can be seen from Figure 9 that, no matter for low-resolution or high-resolution images, the perceptual adversarial network can more effectively minimize the loss function.Figure 9
The comparison results of the loss function of the improved model in this article, the classic CycleGAN + PAN model, and the CycleGAN model. (a) 256∗ 256. (b) 515 ∗ 512. (c) 1024 ∗ 1024.
(a)(b)(c)The method proposed in this paper improves the convergence speed more than the original CycleGAN model. As shown in Figure10, CycleGAN model, CycleGAN + PAN model, and improved CycleGAN + PAN model are trained on the same data set, and the inception score is used to measure the quality and variety of style rendering images. In the case of the same number of iterations, the higher the evaluation score is, the better the quality of the style rendering image is and the faster the convergence speed is obtained. In the experimental results of the three models, the improved CycleGAN + PAN model can converge faster and obtain higher evaluation scores. Through the convergence speed comparison experiment, we can see that the improved CycleGAN + PAN model proposed in this paper has a faster convergence speed and better stylized image quality.Figure 10
Comparison of the efficiency of the three comparison models in the rendering of artistic style of painting.


## 6. Conclusion

In this paper, we select the characteristics of the PAN model and CycleGAN model and combine the advantages of such two models to construct the image painting art style rendering system. The discriminator updates parameters to explore the difference between generated images and style image, and the image conversion network uses the discriminant network to proceed before feed training until the difference is minimized to the optimal image conversion model. Such two models are alternately updated according to the perceptual loss. Experimental results have shown that, compared with the original CycleGAN network with the ResNet module, this paper adopts the improved CycleGAN + PAN image style rendering system based on perceptual confrontation loss. The style rendering effect is better, and the learned style can be rendered to new painting art style image to obtain a clearer performance effect. In future research, further optimization of the network structure and loss function will be considered in our researches. In terms of color and texture, the structural similarity and semantic consistency of content maps and style maps will be enhanced for novel combination of network structure and loss functions.



---

*Source: 4708758-2021-10-04.xml*