# Research on the Style of Art Works based on Deep Learning

Authors: Shulin Liu
Journal: Journal of Advanced Transportation
 (2022)

---

## Abstract

In view of the unsatisfactory effect and major limitations of the style transfer of art works, this paper takes Chinese ink painting for the research subject. The obvious texture characteristics of Chinese ink painting are selected as the input of the Cycle Generative Adversarial Network (CycleGAN) model builder, and the relativistic evaluator is employed to improve the model loss function and the adversarial loss function. An improved art style transfer method of the CycleGAN model is proposed. The experiment shows that the improved CycleGAN model is efficient and feasible for style transfer. Compared with the traditional CycleGAN model, the proposed model performs better in GAN train and GAN test, with a higher average pass rate, which is an increase of nearly 10%. At the same time, with the increase of the number of iterations, the training time of the improved model is close to that of the original model, but the image of the improved model training is larger, which shows that it has more advantages.


---

## Body

## 1. Introduction

Art works are products of human culture, embodying the spirit of humanism and reflecting the cultural characteristics of the times. Research on the style transfer of art works is conducive to spreading art more widely. However, the current studies of art style mainly focus on image analogy, image preprocessing, and feature extraction. For instance, Hahn et al. extracted image features from multiple art works with the same style by preprocessing images and image analogy, and carried out art style transformation, thus realized the style transfer learning of art works [1, 2]. Wang et al., for the purpose of objectively calculating the artistic style of Xin An Art School, tried to put forward digital features such as redundancy, order degree, and intricacy to represent the art style of painting algorithm based on Shannon entropy in information theory. To some extent, it reflects the artistic styles of Xin’an School of Painting in different times and the diversity of different artists. Over the years, the progression and application of deep learning have brought new opportunities for the study of art works style [3, 4]. Yang, and Ghani and Md Azahar put forward an image art classification method on the basis of the VGGNET model for the classification of various painting artistic styles in contemporary society. Convolutional neural network (CNN) was employed to complete the classification of various painting art styles, which lays a foundation for the transfer learning of art works [5, 6]. Chen et al. applied deep learning in different artistic contexts, discussing to which extent the space induced by deep learning CNN can capture the progress of works in music and visual art. Based on the main path algorithm, the evolution analysis of the vector space induced by CNN was carried out, and the reasonable connection between visual art works and songs of different styles was found, which provides interesting insights for extracting evolutionary information in any high-dimensional spaces [7, 8]. Kang et al. proposed a texture migration method for video stream, which can stylize the perception-enhanced video [9]. Ruder et al. proposed a style transfer method of deep learning, which can stylize videos of any length [10]. Zhu et al. proposed the saliency algorithm to guide style transfer, showing good performance [11]. Cui et al. proposed visual smooth bilateral convolution block (B-block) and feature fusion strategy (f-block) to improve the image quality in style conversion [12]. According to the above research, it can be found that the neural network has certain advantages in the style classification and transfer learning of art works. The style transfer image can be obtained by reorganizing the features of rented images through the network and then using the decoder to generate synthetic images. However, the transfer effect of the model based on the above neural network has to be improved. For solving this problem, this paper takes Chinese ink painting as the research object. Based on the CycleGAN, which is currently ideal style transfer model, the obvious texture features of Chinese ink painting are used as the input of CycleGAN model generator. The loss function and adversarial loss function are improved by relativistic discriminator, and an improved CycleGAN art style transfer model is proposed.


## 2. Basic Methods

### 2.1. Introduction to CycleGAN Style Transfer Model

CycleGAN style transfer model is an unsupervised style transfer method for transforming in different image fields without requiring specific image pairs. It is often used in language translation and image conversion tasks in daily life [13]. When the CycleGAN model is applied to the image conversion task, its specific operations are as follows.Given imagesX and Y of two different domains, construct generators G and F, respectively:(1)G:X⟶Y,F:Y⟶X.In other words, the real imageIA in source domain A is transformed into the false image IB in target domain B through generator G. Then, IB is reconstructed through generator F to obtain reconstructed image IRA so that the original image information can be saved. Finally, IB and its real image IB are put into the discriminator D to judge the authenticity, and the complete one-way GAN network is obtained. Combining the two one-way GAN networks, a consistent loss supervision training network can be obtained. The related formulas of the whole process are as follows [14]:(2)LGANG,DY,X,Y=Ey∼pdataylogDYy+Ex∼pdataxlog1−DYGx,LcycG,F=EX∼pdataxFGx−x1+Ey∼pdatayGFy−y1.In the formula,LGAN is the confrontation loss, while Lcyc represents the cyclic consistency loss and ⋅1 represents the L1 loss.Two GANs each contain a discriminator and share two generators, so their loss function can be represented as [15]:(3)LGANG,F,Dx,DY=LGANG,DY,X,Y+LGANF,DX,X,Y+λ1LcycG,F+λ2LidtG,F.In the formula,Lidt stands for the identity mapping loss; λ1 and λ2 stand for the weighting coefficients.Figure1 shows the framework of the CycleGAN model.Figure 1
Schematic diagram of the CycleGAN model structure.


### 2.2. Improvement of CycleGAN Style Transfer Model

#### 2.2.1. Improvement of Discriminator

This paper argues that the missing key characteristics of the CycleGAN model are the increased probability of fake data becoming real (Dxf) and the decreased probability of real data becoming real (Dxr). In the standard CycleGAN model, the identifier can be defined as follows, according to the nontransformation layer Cx [16]:(4)Dx=SigmoidCx.Relativistic discriminator is adopted to improve model discriminator and generator loss function, that is, to sample A from real/false data, then formula (4) can be rewritten to [17]:(5)Dx=sigmoidCxf.The probability that the given true data are more real than the fake data which are randomly sampled can be estimated by the above method. Therefore, the loss function of the CycleGAN model evaluator and generator are rewritten as formulas (6) to (16).(6)LD=Exr∼p1ogsigmoidCxr−Exf∼Q1og1−sigmoidCxf,(7)LG=−Exr∼Q1ogsigmoidCxf,(8)LDRGAN=−Exr,xf∼P,Q1ogsigmoidCxr−Cxf,(9)LGRGAN=−Exr,xf∼P,Q1ogsigmoidCxf−Cxr.In the formulas,xr and P represent real data and their data sets, respectively; xf and Q, respectively, represent generated data and its set. Cx is the output of the discriminator without transformation layer.In addition, to make the identifiers of the relativistic discriminator play a role in the initial definition, this paper proposes an average relativistic discriminator, as shown in formulas (10) and (11) [18]:(10)LDRaGAN=−Exr∼P1ogD¯xr−Exf∼Q1og1−D¯xf,(11)LGRaGAN=−Exf∼Q1ogD¯xf−Exr∼p1og1−D¯xr,(12)D¯xrsigmoidCxr−Exf∼QCxf,(13)D¯xf=sigmoidCxf.


#### 2.2.2. Improvement of Generator

Generator is a significant part of the CycleGAN model, which is related to the final transfer result of the model. The CycleGAN model generator loss function includes saturated loss function and unsaturated loss function. In practical applications, the saturated loss function is very unstable, which causes the failure of the CycleGAN model to reach the optimal state, and the problem of gradient disappearance is prone to occur when the learning rate is high [19]. Therefore, to solve this problem, this paper improves the model from two aspects: generator and generating adversarial loss function.(1) Generator Improvement.. The CycleGAN model generator is usually composed of residual networks, and its basic model structure is shown in Figure 2 [20], including two downsampled convolution layers with 2 steps, two upsampled convolution layers with 1/2 step, and nine residual modules. It achieves the transfer of image style by downsampling the image, changing the image characteristics through the network conversion, and finally restoring the image through upsampling. The texture features reflecting ink painting features are added as additional conditions to the generator under the original input conditions, so that the CycleGAN model generator is improved. The improved model structure is shown in Figure 3 [21].Figure 2
CycleGAN model structure.Figure 3
Improved CycleGAN model structure.(2) Improvement of Generating Adversarial Loss Function.. L2 is employed as the adversarial loss, and the average relativistic discriminator loss is used [22]. The loss of modified discriminator and the adversarial loss are shown as follows:(14)LD=Ex∼pdataxDX−Ey∼pdatayDGY−12+Ey∼pdatayDGY−Ex∼pdataxDX+12LGANG,DY,X,Y=Ey∼pdatayDYGY−Ex∼pdataxDYX−12+Ex∼pdataxDYX−Ey∼pdatayDYGX+12.In the formula,Lcyc and Lidt keep unchanged, then the model’s total loss is(15)LGANG,F,DX,DY=LGANG,DY,X,YLGANF,DX,X,Y+λ1LcycG,F+λ2LidtG,F.


## 2.1. Introduction to CycleGAN Style Transfer Model

CycleGAN style transfer model is an unsupervised style transfer method for transforming in different image fields without requiring specific image pairs. It is often used in language translation and image conversion tasks in daily life [13]. When the CycleGAN model is applied to the image conversion task, its specific operations are as follows.Given imagesX and Y of two different domains, construct generators G and F, respectively:(1)G:X⟶Y,F:Y⟶X.In other words, the real imageIA in source domain A is transformed into the false image IB in target domain B through generator G. Then, IB is reconstructed through generator F to obtain reconstructed image IRA so that the original image information can be saved. Finally, IB and its real image IB are put into the discriminator D to judge the authenticity, and the complete one-way GAN network is obtained. Combining the two one-way GAN networks, a consistent loss supervision training network can be obtained. The related formulas of the whole process are as follows [14]:(2)LGANG,DY,X,Y=Ey∼pdataylogDYy+Ex∼pdataxlog1−DYGx,LcycG,F=EX∼pdataxFGx−x1+Ey∼pdatayGFy−y1.In the formula,LGAN is the confrontation loss, while Lcyc represents the cyclic consistency loss and ⋅1 represents the L1 loss.Two GANs each contain a discriminator and share two generators, so their loss function can be represented as [15]:(3)LGANG,F,Dx,DY=LGANG,DY,X,Y+LGANF,DX,X,Y+λ1LcycG,F+λ2LidtG,F.In the formula,Lidt stands for the identity mapping loss; λ1 and λ2 stand for the weighting coefficients.Figure1 shows the framework of the CycleGAN model.Figure 1
Schematic diagram of the CycleGAN model structure.


## 2.2. Improvement of CycleGAN Style Transfer Model

### 2.2.1. Improvement of Discriminator

This paper argues that the missing key characteristics of the CycleGAN model are the increased probability of fake data becoming real (Dxf) and the decreased probability of real data becoming real (Dxr). In the standard CycleGAN model, the identifier can be defined as follows, according to the nontransformation layer Cx [16]:(4)Dx=SigmoidCx.Relativistic discriminator is adopted to improve model discriminator and generator loss function, that is, to sample A from real/false data, then formula (4) can be rewritten to [17]:(5)Dx=sigmoidCxf.The probability that the given true data are more real than the fake data which are randomly sampled can be estimated by the above method. Therefore, the loss function of the CycleGAN model evaluator and generator are rewritten as formulas (6) to (16).(6)LD=Exr∼p1ogsigmoidCxr−Exf∼Q1og1−sigmoidCxf,(7)LG=−Exr∼Q1ogsigmoidCxf,(8)LDRGAN=−Exr,xf∼P,Q1ogsigmoidCxr−Cxf,(9)LGRGAN=−Exr,xf∼P,Q1ogsigmoidCxf−Cxr.In the formulas,xr and P represent real data and their data sets, respectively; xf and Q, respectively, represent generated data and its set. Cx is the output of the discriminator without transformation layer.In addition, to make the identifiers of the relativistic discriminator play a role in the initial definition, this paper proposes an average relativistic discriminator, as shown in formulas (10) and (11) [18]:(10)LDRaGAN=−Exr∼P1ogD¯xr−Exf∼Q1og1−D¯xf,(11)LGRaGAN=−Exf∼Q1ogD¯xf−Exr∼p1og1−D¯xr,(12)D¯xrsigmoidCxr−Exf∼QCxf,(13)D¯xf=sigmoidCxf.


### 2.2.2. Improvement of Generator

Generator is a significant part of the CycleGAN model, which is related to the final transfer result of the model. The CycleGAN model generator loss function includes saturated loss function and unsaturated loss function. In practical applications, the saturated loss function is very unstable, which causes the failure of the CycleGAN model to reach the optimal state, and the problem of gradient disappearance is prone to occur when the learning rate is high [19]. Therefore, to solve this problem, this paper improves the model from two aspects: generator and generating adversarial loss function.(1) Generator Improvement.. The CycleGAN model generator is usually composed of residual networks, and its basic model structure is shown in Figure 2 [20], including two downsampled convolution layers with 2 steps, two upsampled convolution layers with 1/2 step, and nine residual modules. It achieves the transfer of image style by downsampling the image, changing the image characteristics through the network conversion, and finally restoring the image through upsampling. The texture features reflecting ink painting features are added as additional conditions to the generator under the original input conditions, so that the CycleGAN model generator is improved. The improved model structure is shown in Figure 3 [21].Figure 2
CycleGAN model structure.Figure 3
Improved CycleGAN model structure.(2) Improvement of Generating Adversarial Loss Function.. L2 is employed as the adversarial loss, and the average relativistic discriminator loss is used [22]. The loss of modified discriminator and the adversarial loss are shown as follows:(14)LD=Ex∼pdataxDX−Ey∼pdatayDGY−12+Ey∼pdatayDGY−Ex∼pdataxDX+12LGANG,DY,X,Y=Ey∼pdatayDYGY−Ex∼pdataxDYX−12+Ex∼pdataxDYX−Ey∼pdatayDYGX+12.In the formula,Lcyc and Lidt keep unchanged, then the model’s total loss is(15)LGANG,F,DX,DY=LGANG,DY,X,YLGANF,DX,X,Y+λ1LcycG,F+λ2LidtG,F.


## 2.2.1. Improvement of Discriminator

This paper argues that the missing key characteristics of the CycleGAN model are the increased probability of fake data becoming real (Dxf) and the decreased probability of real data becoming real (Dxr). In the standard CycleGAN model, the identifier can be defined as follows, according to the nontransformation layer Cx [16]:(4)Dx=SigmoidCx.Relativistic discriminator is adopted to improve model discriminator and generator loss function, that is, to sample A from real/false data, then formula (4) can be rewritten to [17]:(5)Dx=sigmoidCxf.The probability that the given true data are more real than the fake data which are randomly sampled can be estimated by the above method. Therefore, the loss function of the CycleGAN model evaluator and generator are rewritten as formulas (6) to (16).(6)LD=Exr∼p1ogsigmoidCxr−Exf∼Q1og1−sigmoidCxf,(7)LG=−Exr∼Q1ogsigmoidCxf,(8)LDRGAN=−Exr,xf∼P,Q1ogsigmoidCxr−Cxf,(9)LGRGAN=−Exr,xf∼P,Q1ogsigmoidCxf−Cxr.In the formulas,xr and P represent real data and their data sets, respectively; xf and Q, respectively, represent generated data and its set. Cx is the output of the discriminator without transformation layer.In addition, to make the identifiers of the relativistic discriminator play a role in the initial definition, this paper proposes an average relativistic discriminator, as shown in formulas (10) and (11) [18]:(10)LDRaGAN=−Exr∼P1ogD¯xr−Exf∼Q1og1−D¯xf,(11)LGRaGAN=−Exf∼Q1ogD¯xf−Exr∼p1og1−D¯xr,(12)D¯xrsigmoidCxr−Exf∼QCxf,(13)D¯xf=sigmoidCxf.


## 2.2.2. Improvement of Generator

Generator is a significant part of the CycleGAN model, which is related to the final transfer result of the model. The CycleGAN model generator loss function includes saturated loss function and unsaturated loss function. In practical applications, the saturated loss function is very unstable, which causes the failure of the CycleGAN model to reach the optimal state, and the problem of gradient disappearance is prone to occur when the learning rate is high [19]. Therefore, to solve this problem, this paper improves the model from two aspects: generator and generating adversarial loss function.(1) Generator Improvement.. The CycleGAN model generator is usually composed of residual networks, and its basic model structure is shown in Figure 2 [20], including two downsampled convolution layers with 2 steps, two upsampled convolution layers with 1/2 step, and nine residual modules. It achieves the transfer of image style by downsampling the image, changing the image characteristics through the network conversion, and finally restoring the image through upsampling. The texture features reflecting ink painting features are added as additional conditions to the generator under the original input conditions, so that the CycleGAN model generator is improved. The improved model structure is shown in Figure 3 [21].Figure 2
CycleGAN model structure.Figure 3
Improved CycleGAN model structure.(2) Improvement of Generating Adversarial Loss Function.. L2 is employed as the adversarial loss, and the average relativistic discriminator loss is used [22]. The loss of modified discriminator and the adversarial loss are shown as follows:(14)LD=Ex∼pdataxDX−Ey∼pdatayDGY−12+Ey∼pdatayDGY−Ex∼pdataxDX+12LGANG,DY,X,Y=Ey∼pdatayDYGY−Ex∼pdataxDYX−12+Ex∼pdataxDYX−Ey∼pdatayDYGX+12.In the formula,Lcyc and Lidt keep unchanged, then the model’s total loss is(15)LGANG,F,DX,DY=LGANG,DY,X,YLGANF,DX,X,Y+λ1LcycG,F+λ2LidtG,F.


## 3. The Style Transfer Method of Art Works based on the Improved CycleGAN Model

### 3.1. Feature Extraction

Feature extraction has a great influence on the construction of style transfer of art works based on the improved CycleGAN model. Therefore, before the style transfer of art works, this paper extracted the art features that can reflect the style of art works at first. Feature extraction includes line feature extraction and texture feature extraction [18]. Among them, line feature extraction is the main manifestation of emotional features in traditional Chinese paintings, and the fluidity of lines is usually expressed by curvature, as in formula (16) [23]:(16)Fline=1+fx2fyy+1+fy2fxx−2fxfyfxy1+fx2+fy23/2.In the formula,x and y represent the coordinates of a certain pixel in the image, respectively, and fx,y represents the pixel’s grey level. fx,fy,fxy, fxx, and fyy, respectively, represent the first-order, second-order, and mixed partial derivatives of fx,y, and Fline represents the Gaussian curvature of the pixel.In this paper, the LBP method is adopted to extract texture features, as shown in formula (17) [24]:(17)LBPxc,yc=∑p−0p−12psip−ic.In the formula,xc,yc indicates the central element of the 3∗3 area, and its pixel value ip,ic indicates other pixel values in the area. Sx indicates a symbolic function, and its basic definition is shown as follows [25]:(18)Sx=1,ifx≥0,0,else.By referring to relevant literature and the characteristics of Chinese ink painting, texture features that can best reflect ink painting are selected as the main features and extracted in this paper.


### 3.2. Ink Painting Transfer Process based on Improved CycleGAN

Based on the improved CycleGAN model, this paper takes ink painting as the research object and designs the style transfer process of art works as Figure4. The specific steps are as follows:Figure 4
Art Work Style transfer process based on the improved CycleGAN model.The specific steps are as follows:(1)
Image data collection and sorting: all the image data are unified in specification and size and divided them into training and test data set in a certain proportion;(2)
Feature extraction: the LBP method is used to extract texture features from images and input them into the CycleGAN model generator as additional conditions;(3)
Model training: the initial learning rate, batch size, epoch, and other parameters of the CycleGAN model are initialized, and the model is trained. The model is saved as it met the termination conditions;(4)
Input the test data set into the saved CycleGAN model and output the image transfer results.


## 3.1. Feature Extraction

Feature extraction has a great influence on the construction of style transfer of art works based on the improved CycleGAN model. Therefore, before the style transfer of art works, this paper extracted the art features that can reflect the style of art works at first. Feature extraction includes line feature extraction and texture feature extraction [18]. Among them, line feature extraction is the main manifestation of emotional features in traditional Chinese paintings, and the fluidity of lines is usually expressed by curvature, as in formula (16) [23]:(16)Fline=1+fx2fyy+1+fy2fxx−2fxfyfxy1+fx2+fy23/2.In the formula,x and y represent the coordinates of a certain pixel in the image, respectively, and fx,y represents the pixel’s grey level. fx,fy,fxy, fxx, and fyy, respectively, represent the first-order, second-order, and mixed partial derivatives of fx,y, and Fline represents the Gaussian curvature of the pixel.In this paper, the LBP method is adopted to extract texture features, as shown in formula (17) [24]:(17)LBPxc,yc=∑p−0p−12psip−ic.In the formula,xc,yc indicates the central element of the 3∗3 area, and its pixel value ip,ic indicates other pixel values in the area. Sx indicates a symbolic function, and its basic definition is shown as follows [25]:(18)Sx=1,ifx≥0,0,else.By referring to relevant literature and the characteristics of Chinese ink painting, texture features that can best reflect ink painting are selected as the main features and extracted in this paper.


## 3.2. Ink Painting Transfer Process based on Improved CycleGAN

Based on the improved CycleGAN model, this paper takes ink painting as the research object and designs the style transfer process of art works as Figure4. The specific steps are as follows:Figure 4
Art Work Style transfer process based on the improved CycleGAN model.The specific steps are as follows:(1)
Image data collection and sorting: all the image data are unified in specification and size and divided them into training and test data set in a certain proportion;(2)
Feature extraction: the LBP method is used to extract texture features from images and input them into the CycleGAN model generator as additional conditions;(3)
Model training: the initial learning rate, batch size, epoch, and other parameters of the CycleGAN model are initialized, and the model is trained. The model is saved as it met the termination conditions;(4)
Input the test data set into the saved CycleGAN model and output the image transfer results.


## 4. Simulation Experiment

### 4.1. Experimental Environment Construction

This experiment is simulated on a 64-bit Windows 10 with Intel Core i7-7800K CPU, and the GPU is NVIDI GeForceRTX2070. The memory is 16G, the open-source depth framework is pytorch, and the programming language is python.


### 4.2. Data Sources and Preprocessing

In this paper, 300 natural landscape pictures and 360 Chinese ink paintings are taken from the Internet by crawler technology as experimental data sets, and the part of the data is shown in Figure5. Considering the small number of data sets, to expand the data set, data enhancement technology is adopted to enlarge the natural landscape pictures and Chinese ink paintings to twice the original size in this paper. In addition, due to the different sizes of each image, all images are unified to 256∗256 pixels before training to meet the model input requirements [26].Figure 5
Example of experimental data set.


### 4.3. Evaluation Indicators

At present, there is no unified evaluation indicator for the study of art works style transfer, which makes it difficult to objectively evaluate the transfer results. Through consulting relevant references, this paper finds that manual evaluation test and GAN train/test can basically evaluate the art works style transfer [23, 27–29]. Therefore, the above two indicators are selected as the evaluation criteria for the transfer effect of the proposed method. Among them, manual evaluation test reflects subjective evaluation results, while the GAN train/test reflects objective evaluation of image transfer effect to a certain extent by classifying images and evaluating the quality of generated images according to classification accuracy. The GAN train/test includes two indicators, GAN train and GAN test. When the GAN train is closer to the classification accuracy GAN base, it indicates that the diversity of generated samples is closer to the real sample, that is, the better the GAN model is; when the GAN test is close to GAN base, it indicates that the higher the quality of the image generated by the GAN model.


### 4.4. Parameter Setting

The model proposed in this experiment uses Adam as the optimizer. The original learning rate is 2e − 3, the batch size is 2, and the epoch is 200. The number of iterations of each epoch is consistent with that of the larger total pictures in the training data set. The training parameters of the compared model CycleGAN are configured with default parameters, and the epoch is also 200.


### 4.5. Experimental Results

#### 4.5.1. Feature Extraction Results

The LBP method is adopted to extract texture features of works as shown in Figure6. It shows that the proposed method can well extract texture features of ink painting works, laying a foundation for subsequent style transfer of ink painting works.Figure 6
Extraction results of texture features.


#### 4.5.2. Qualitative Experimental Results

The verification of the proposed method for its effectiveness was conducted. The proposed model and CycleGAN model were used to transfer experimental data under the same parameter settings, and the transfer results of the two models were compared. The overall and partial comparisons are shown in Figures7 and 8. In the figures, the first column is the initial image, and the second and third are, respectively, the transferred images of the compared model and the model in this paper. It can be seen from the figures that compared with the transferred images of the unimproved CycleGAN model, transferred images of the model proposed in this paper are more suitable for the characteristics of Chinese ink painting. The CycleGAN model only reduced the contrast of the original input image to transfer images, making the overall image style looks closer to the input images. The transferred images generated by the model proposed in this paper are closer to the average of all style and Chinese ink painting style by the average relativistic discriminator. Therefore, the improvements of CycleGAN model in this paper are effective, and the improved model can generate images with Chinese ink painting style.Figure 7
Comparison of transferred images of different models on the whole.Figure 8
Comparison of transferred images of different models in part.Since both the proposed model and the CycleGAN model need to realize self-reconstruction, Figure9 is the reconstruction result of the experimental data set on the proposed model. In the figure, the first column is the original input image, including a picture of natural landscape and ink painting style; the second column is the transferred image corresponding to the first column; the reconstructed image of the second column is shown in the third column. As the figure shows, the generated transfer image retains the original image content, while the landscape painting style is changed to ink painting style with Chinese characteristics, and the ink painting image is changed to a more realistic landscape image. The reconstructed image can be reconstructed well through two generators that are opposite to each other. Therefore, the model proposed in this paper can realize image style transfer, which proves the effectiveness and feasibility of it.Figure 9
Image style transfer and reconstruction results.


#### 4.5.3. Quantitative Analysis Results

To quantitatively analyze the validity of the proposed model, the GAN train/test indicators are used to evaluate the model. Considering that the data set for training is small, the classifier in this paper adopts the ResNet-50 model, and the outcomes of the proposed model and the unimproved CycleGAN model are obtained. The results are shown in Table1. As the table shows, in comparison with the CycleGAN model, the GAN base of the proposed model is 94.3%, and it performs better in the GAN train and GAN test indicators. It indicates that the model proposed in this article has a certain effectiveness in the style transfer of art works.Table 1
Comparison of GAN train/test of different models.
ModelGAN train (%)GAN test (%)The original CycleGAN61.8661.02The proposed model67.0370.37As the style of art works is an abstract concept and its style transfer has a certain degree of subjectivity, this paper sets manual evaluation test indicators to further analyze the experimental results. Firstly, image sets A and B are generated by inputting 100 natural landscape images to the model proposed in this paper and the CycleGAN model, and the images in the A and B sets are disarranged. Then, 10 art students were selected to judge each image to see whether it looked like a Chinese ink painting or not. Secondly, the images that resemble ink paintings are counted, and their proportions are calculated as the pass rate. Finally, the average pass rate is taken as the final evaluation result, as shown in Table2. From the table, compared with the CycleGAN model, the average pass rate of the proposed model is higher, with an increase of nearly 10%. This shows that the proposed model is effective in improving the CycleGAN model and can obtain images that are more suitable for Chinese ink painting style.Table 2
Comparison of manual evaluation test results of different models.
ModelAverage pass rate (%)The original CycleGAN52.76The proposed model62.37In order to verify the running time and space consumption of the proposed model, the running time and graphics card usage of model training with 1 epoch and 50 epochs were recorded, respectively. Also, compared them with that of the CycleGAN model under the same conditions, the results can be seen in Table3. The table shows that in contrast with the CycleGAN model, the model proposed in this paper has no significant change in running time, which can be basically ignored. In terms of graphics card usage, the model proposed in this article has an increase of around 6.4% compared to the CycleGAN model. The reason is that additional information has been added to the generator. In general, it is feasible and valuable to improve model performance with less computing resources. Therefore, the method proposed in this paper has certain effectiveness and practical application value.Table 3
Comparison of model running time and graphics card usage.
Model1 epoch (s)50 epochGraphics card usage (MB)The original CycleGAN1171 h 40 min6703The proposed model1291 h 48 min7137


## 4.1. Experimental Environment Construction

This experiment is simulated on a 64-bit Windows 10 with Intel Core i7-7800K CPU, and the GPU is NVIDI GeForceRTX2070. The memory is 16G, the open-source depth framework is pytorch, and the programming language is python.


## 4.2. Data Sources and Preprocessing

In this paper, 300 natural landscape pictures and 360 Chinese ink paintings are taken from the Internet by crawler technology as experimental data sets, and the part of the data is shown in Figure5. Considering the small number of data sets, to expand the data set, data enhancement technology is adopted to enlarge the natural landscape pictures and Chinese ink paintings to twice the original size in this paper. In addition, due to the different sizes of each image, all images are unified to 256∗256 pixels before training to meet the model input requirements [26].Figure 5
Example of experimental data set.


## 4.3. Evaluation Indicators

At present, there is no unified evaluation indicator for the study of art works style transfer, which makes it difficult to objectively evaluate the transfer results. Through consulting relevant references, this paper finds that manual evaluation test and GAN train/test can basically evaluate the art works style transfer [23, 27–29]. Therefore, the above two indicators are selected as the evaluation criteria for the transfer effect of the proposed method. Among them, manual evaluation test reflects subjective evaluation results, while the GAN train/test reflects objective evaluation of image transfer effect to a certain extent by classifying images and evaluating the quality of generated images according to classification accuracy. The GAN train/test includes two indicators, GAN train and GAN test. When the GAN train is closer to the classification accuracy GAN base, it indicates that the diversity of generated samples is closer to the real sample, that is, the better the GAN model is; when the GAN test is close to GAN base, it indicates that the higher the quality of the image generated by the GAN model.


## 4.4. Parameter Setting

The model proposed in this experiment uses Adam as the optimizer. The original learning rate is 2e − 3, the batch size is 2, and the epoch is 200. The number of iterations of each epoch is consistent with that of the larger total pictures in the training data set. The training parameters of the compared model CycleGAN are configured with default parameters, and the epoch is also 200.


## 4.5. Experimental Results

### 4.5.1. Feature Extraction Results

The LBP method is adopted to extract texture features of works as shown in Figure6. It shows that the proposed method can well extract texture features of ink painting works, laying a foundation for subsequent style transfer of ink painting works.Figure 6
Extraction results of texture features.


### 4.5.2. Qualitative Experimental Results

The verification of the proposed method for its effectiveness was conducted. The proposed model and CycleGAN model were used to transfer experimental data under the same parameter settings, and the transfer results of the two models were compared. The overall and partial comparisons are shown in Figures7 and 8. In the figures, the first column is the initial image, and the second and third are, respectively, the transferred images of the compared model and the model in this paper. It can be seen from the figures that compared with the transferred images of the unimproved CycleGAN model, transferred images of the model proposed in this paper are more suitable for the characteristics of Chinese ink painting. The CycleGAN model only reduced the contrast of the original input image to transfer images, making the overall image style looks closer to the input images. The transferred images generated by the model proposed in this paper are closer to the average of all style and Chinese ink painting style by the average relativistic discriminator. Therefore, the improvements of CycleGAN model in this paper are effective, and the improved model can generate images with Chinese ink painting style.Figure 7
Comparison of transferred images of different models on the whole.Figure 8
Comparison of transferred images of different models in part.Since both the proposed model and the CycleGAN model need to realize self-reconstruction, Figure9 is the reconstruction result of the experimental data set on the proposed model. In the figure, the first column is the original input image, including a picture of natural landscape and ink painting style; the second column is the transferred image corresponding to the first column; the reconstructed image of the second column is shown in the third column. As the figure shows, the generated transfer image retains the original image content, while the landscape painting style is changed to ink painting style with Chinese characteristics, and the ink painting image is changed to a more realistic landscape image. The reconstructed image can be reconstructed well through two generators that are opposite to each other. Therefore, the model proposed in this paper can realize image style transfer, which proves the effectiveness and feasibility of it.Figure 9
Image style transfer and reconstruction results.


### 4.5.3. Quantitative Analysis Results

To quantitatively analyze the validity of the proposed model, the GAN train/test indicators are used to evaluate the model. Considering that the data set for training is small, the classifier in this paper adopts the ResNet-50 model, and the outcomes of the proposed model and the unimproved CycleGAN model are obtained. The results are shown in Table1. As the table shows, in comparison with the CycleGAN model, the GAN base of the proposed model is 94.3%, and it performs better in the GAN train and GAN test indicators. It indicates that the model proposed in this article has a certain effectiveness in the style transfer of art works.Table 1
Comparison of GAN train/test of different models.
ModelGAN train (%)GAN test (%)The original CycleGAN61.8661.02The proposed model67.0370.37As the style of art works is an abstract concept and its style transfer has a certain degree of subjectivity, this paper sets manual evaluation test indicators to further analyze the experimental results. Firstly, image sets A and B are generated by inputting 100 natural landscape images to the model proposed in this paper and the CycleGAN model, and the images in the A and B sets are disarranged. Then, 10 art students were selected to judge each image to see whether it looked like a Chinese ink painting or not. Secondly, the images that resemble ink paintings are counted, and their proportions are calculated as the pass rate. Finally, the average pass rate is taken as the final evaluation result, as shown in Table2. From the table, compared with the CycleGAN model, the average pass rate of the proposed model is higher, with an increase of nearly 10%. This shows that the proposed model is effective in improving the CycleGAN model and can obtain images that are more suitable for Chinese ink painting style.Table 2
Comparison of manual evaluation test results of different models.
ModelAverage pass rate (%)The original CycleGAN52.76The proposed model62.37In order to verify the running time and space consumption of the proposed model, the running time and graphics card usage of model training with 1 epoch and 50 epochs were recorded, respectively. Also, compared them with that of the CycleGAN model under the same conditions, the results can be seen in Table3. The table shows that in contrast with the CycleGAN model, the model proposed in this paper has no significant change in running time, which can be basically ignored. In terms of graphics card usage, the model proposed in this article has an increase of around 6.4% compared to the CycleGAN model. The reason is that additional information has been added to the generator. In general, it is feasible and valuable to improve model performance with less computing resources. Therefore, the method proposed in this paper has certain effectiveness and practical application value.Table 3
Comparison of model running time and graphics card usage.
Model1 epoch (s)50 epochGraphics card usage (MB)The original CycleGAN1171 h 40 min6703The proposed model1291 h 48 min7137


## 4.5.1. Feature Extraction Results

The LBP method is adopted to extract texture features of works as shown in Figure6. It shows that the proposed method can well extract texture features of ink painting works, laying a foundation for subsequent style transfer of ink painting works.Figure 6
Extraction results of texture features.


## 4.5.2. Qualitative Experimental Results

The verification of the proposed method for its effectiveness was conducted. The proposed model and CycleGAN model were used to transfer experimental data under the same parameter settings, and the transfer results of the two models were compared. The overall and partial comparisons are shown in Figures7 and 8. In the figures, the first column is the initial image, and the second and third are, respectively, the transferred images of the compared model and the model in this paper. It can be seen from the figures that compared with the transferred images of the unimproved CycleGAN model, transferred images of the model proposed in this paper are more suitable for the characteristics of Chinese ink painting. The CycleGAN model only reduced the contrast of the original input image to transfer images, making the overall image style looks closer to the input images. The transferred images generated by the model proposed in this paper are closer to the average of all style and Chinese ink painting style by the average relativistic discriminator. Therefore, the improvements of CycleGAN model in this paper are effective, and the improved model can generate images with Chinese ink painting style.Figure 7
Comparison of transferred images of different models on the whole.Figure 8
Comparison of transferred images of different models in part.Since both the proposed model and the CycleGAN model need to realize self-reconstruction, Figure9 is the reconstruction result of the experimental data set on the proposed model. In the figure, the first column is the original input image, including a picture of natural landscape and ink painting style; the second column is the transferred image corresponding to the first column; the reconstructed image of the second column is shown in the third column. As the figure shows, the generated transfer image retains the original image content, while the landscape painting style is changed to ink painting style with Chinese characteristics, and the ink painting image is changed to a more realistic landscape image. The reconstructed image can be reconstructed well through two generators that are opposite to each other. Therefore, the model proposed in this paper can realize image style transfer, which proves the effectiveness and feasibility of it.Figure 9
Image style transfer and reconstruction results.


## 4.5.3. Quantitative Analysis Results

To quantitatively analyze the validity of the proposed model, the GAN train/test indicators are used to evaluate the model. Considering that the data set for training is small, the classifier in this paper adopts the ResNet-50 model, and the outcomes of the proposed model and the unimproved CycleGAN model are obtained. The results are shown in Table1. As the table shows, in comparison with the CycleGAN model, the GAN base of the proposed model is 94.3%, and it performs better in the GAN train and GAN test indicators. It indicates that the model proposed in this article has a certain effectiveness in the style transfer of art works.Table 1
Comparison of GAN train/test of different models.
ModelGAN train (%)GAN test (%)The original CycleGAN61.8661.02The proposed model67.0370.37As the style of art works is an abstract concept and its style transfer has a certain degree of subjectivity, this paper sets manual evaluation test indicators to further analyze the experimental results. Firstly, image sets A and B are generated by inputting 100 natural landscape images to the model proposed in this paper and the CycleGAN model, and the images in the A and B sets are disarranged. Then, 10 art students were selected to judge each image to see whether it looked like a Chinese ink painting or not. Secondly, the images that resemble ink paintings are counted, and their proportions are calculated as the pass rate. Finally, the average pass rate is taken as the final evaluation result, as shown in Table2. From the table, compared with the CycleGAN model, the average pass rate of the proposed model is higher, with an increase of nearly 10%. This shows that the proposed model is effective in improving the CycleGAN model and can obtain images that are more suitable for Chinese ink painting style.Table 2
Comparison of manual evaluation test results of different models.
ModelAverage pass rate (%)The original CycleGAN52.76The proposed model62.37In order to verify the running time and space consumption of the proposed model, the running time and graphics card usage of model training with 1 epoch and 50 epochs were recorded, respectively. Also, compared them with that of the CycleGAN model under the same conditions, the results can be seen in Table3. The table shows that in contrast with the CycleGAN model, the model proposed in this paper has no significant change in running time, which can be basically ignored. In terms of graphics card usage, the model proposed in this article has an increase of around 6.4% compared to the CycleGAN model. The reason is that additional information has been added to the generator. In general, it is feasible and valuable to improve model performance with less computing resources. Therefore, the method proposed in this paper has certain effectiveness and practical application value.Table 3
Comparison of model running time and graphics card usage.
Model1 epoch (s)50 epochGraphics card usage (MB)The original CycleGAN1171 h 40 min6703The proposed model1291 h 48 min7137


## 5. Conclusion

To sum up, the style transfer model of art works based on deep learning proposed in this paper utilizes the obvious texture features of Chinese ink paintings as the input of CycleGAN model generator, and utilizes relativistic discriminator to improve the loss function and adversarial loss function of CycleGAN model discriminator, improving the CycleGAN model generator. On the basis of the improved CycleGAN model, this paper studies the transfer of ink painting style. Compared with the unimproved CycleGAN model, the model proposed in this paper can generate images with more Chinese ink painting flavor. The accuracy of GAN base in the verification set Sv is 94.3%, and the performance of GAN train and GAN test is better, with the average pass rate increased by nearly 10%. The innovation of this study is to realize the transfer of artistic style by using the deep learning algorithm so as to provide an information-based way for the transformation of different styles in art.



---

*Source: 5433623-2022-06-11.xml*