# A Method of Style Transfer for Chinese Painting

Cunjian Chen

► To cite this version:

Cunjian Chen. A Method of Style Transfer for Chinese Painting. 11th International Conference on Intelligent Information Processing (IIP), Jul 2020, Hangzhou, China. pp.266-272, 10.1007/978-3-030-46931-3_25 hal-03456972

HAL Id: hal-03456972

https://inria.hal.science/hal-03456972

Submitted on 30 Nov 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Method of Style Transfer for Chinese Painting

Cunjian Chen

Hangzhou Dianzi University, HangZhou, China
291296924@qq.com

**Abstract.** This paper introduces a style transfer method for traditional Chinese painting. We improved the traditional method by adding style characteristics and constraints unique to Chinese painting. By comparing Chinese painting with Western painting and natural pictures, we find that the features such as lines and textures in Chinese painting are quite different from other images. Therefore, these features are extracted and added to the original method in a restrictive manner. Finally, experiments prove that The method has a certain improvement effect on the style transfer result of Chinese painting.

**Keyword:** Style Transfer, Neural Network, Traditional Chinese Painting.

## 1 Introduction

The earliest origin of the term neural art is in Gatys [1]. This paper is different from the traditional image style artistic method. It pioneered the use of convolutional neural networks to learn oil painting style. And Gatys also detailed the steps of image style transfer based on VGG19, which described the formation of style expressions, texture combining methods, and loss functions in more depth [1-3].

Compared to Gatys using Gram matrix to calculate the overall style effect, Li proposed to use the local texture of the style image for style transmission [4,5]. Ulyanov D, proposed a texture combining algorithm based on feedforward network. Since the network is trained in advance, their algorithm execution is faster than the Gatys algorithm [6]. Moreover, some people have designed a more Fast networks need to be trained on some large datasets such as COCO datasets, and then applied to existing images that need to be artistic, which greatly saves training time [7]. Recently, Goodfellow et al. [8] proposed a Generative Adversarial Network (GAN). Deep neural
networks based on GAN ideas are gradually applied in the field of image style transfer [9-11]. For example, CycleGAN [12], DiscoGAN [13] and DualGAN [14] use the cyclic consistency loss to save the key information of the input and transformed images, and realize the image style transfer in the case of unmatched data.

Gu [15] proposed a feature reorganization style migration method based on vgg neural network. They link most parametric and non-parametric style migration methods by readjusting style loss. It has a good effect for general images. But not effective to Chinese painting, because Chinese painting is different from natural images and has special style characteristics. Therefore, this paper try to extract these features and use them to improve the original method to improve the effect of Chinese painting.

## 2 Style Description of Chinese Painting

Chinese painting is one of the oldest continuous art traditions in the world. It is derived from calligraphy, which is essentially a linear art, using strokes to evoke pictures and emotions. The painting techniques of Chinese painting are mainly expressed in four aspects: pen, ink, color and composition [16].

The use of pen is the essence of Chinese painting. Chinese painting is line-shaped and reaches its peak in the use of line technology. Thickness and thinness of the line, long and short, thick and light, light and heavy, and virtual and real are all different things. Important factors [17]. Chinese painting has many painting techniques, and lines are an important basis for these techniques.

Ink painting has always been a favorite type of painting by Chinese literati, and its ink charm has strong Chinese characteristics [18]. The key of ink painting is the harmony of water and ink. Water is the charm of ink painting. The flowing beauty of water combined with the deep ink color is like black and white dancing on paper, and light and shadow intersect. Ink is generally divided into five colors, which are burnt, thick, dark, light, and clear.

Composition generally refers to the arrangement of objects on the screen, reasonable allocation of space, and good screen management is an important basis for painting [19]. Most of the Chinese paintings refer to the structure of the article, which emphasizes the difference between the real and the false, with a clear distinction between primary and secondary. Although composition is a wild art creation process, it is not ruleless. Many long-term developments in Chinese painting have established many
compositional techniques that conform to Chinese aesthetics. Such as full composition, "Zi" composition, corner composition and so on.

# 3 Feature Extraction

This paper improves the method in Gu [15], extracts the style features of Chinese painting. Then adds these features to the feature matrix.

In order to extract features that can effectively reflect the artistic style of images, This chapter refers to a large of art research literature and some articles on feature extraction and classification of Chinese painting[20,21]. According to the description in the second section, the general artistic style is expressed in line drawing, frequency, and texture features. Therefore, this article chooses these three characteristics as the artistic style description of Chinese painting. The main methods are:

## 3.1 Line Feature Extraction

Lines are the most basic component of traditional Chinese painting. Painters use lines to extract, generalize, and abstract natural things and natural scenes. Represent a three-dimensional space with a two-dimensional plane. This experiment uses curvature to represent the fluidity $F_{line}$ of the line, as shown in Equation 1:

$$F_{line} = \frac{(1+f_x^2)f_{yy}+(1+f_y^2)f_{xx}-2f_xf_yf_{xy}}{(1+f_x^2+f_y^2)^{3/2}} \quad (1)$$

where x and y represent the coordinates of a pixel in the image, and f (x, y) is the gray value of the pixel. $f_x$, $f_y$, $f_{xy}$, $f_{xx}$, $f_{yy}$ are the first, second, and mixed partial derivatives of f (x, y), respectively, and $F_{line}$ is the Gaussian curvature of the pixel.

## 3.2 Texture Feature Extraction

Texture feature is a visual phenomenon reflected image, which reflects the surface structure of tissue having a slowly varying or periodic change of the flow properties of the surface of the object. This experiment uses LBP shown in Equation 2:
$$ \mathrm{LBP}(x_c, y_c) = \sum_{p=1}^{8} s(I(p)-I(c)) * 2^p \quad (2) $$

where p represents the p-th pixel point other than the central pixel point in the 3 × 3 window; I (c) represents the gray value of the central pixel point, and I (p) represents the gray value of the p-th pixel point in the field ; s (x) is shown in Equation 3:

$$ s(x) = \begin{cases} 1, & x \ge 0 \\ 0, & \text{otherwise} \end{cases} \quad (3) $$

## 3.3 Frequency Feature Extraction

The frequency of an image is an indicator of the intensity of grayscale changes in the image, and is the gradient of grayscale in plane space. Different frequency information has different functions in the image structure. The main component of the image is low-frequency information, which forms the basic gray level of the image, and has little effect on the structure of the image. The intermediate frequency information determines the basic structure of the image, which forms the main edge structure of the image. The edges and details are further enhancement of the image content on the IF information. This experiment uses Fourier transform to extract the frequency map.

## 3.4 Add Feature Constraint Algorithm

The three features mentioned above are combined into three channels, and a new feature with a certain width and height is added to the feature matrix extracted by the vgg19 network in the original method to form a constant width and height. New feature matrix. Can be tested in different vgg19 network layers.

# 4 Experiment

## 4.1 Build Environment

The experiment uses the caffe framework to build the network. caffé is a deep learning framework that combines expressiveness, speed, and modular thinking. Large-scale industrial applications in academic research projects, start-up prototypes and even vision,
speech and multimedia. The experimental hardware platform is Intel Core i7-7800K CPU, 8G memory, NVIDIA GeForce RTX2070 GPU.

In order to verify the validity of the method, we select some natural pictures and some Chinese paintings on the Internet as experimental materials. The subjective evaluation is used as the evaluation criterion in the experiment.

## 4.2 Experimental Process and Results

**Experiment 1.** Choose a spectacular architectural picture as the content picture and a Chinese ink painting as the style picture. Through a large number of experiments, we found that the effect of adding constraint features to the third layer of the vgg19 network is better than other layers. Therefore, this experiment only operates on the third layer of the network. Figure 1 shows the content image, style image, the composite image generated by the original method, and the composite image generated by our method. Comparing the experimental results, The results of the original method have a great impact on the content, especially the background, and the image has a serious distortion phenomenon. ours method optimizes these defects. However, the problems existing in the original method cannot be completely eliminated. It is speculated that the style transfer based on the non-parametric method cannot effectively generate a transition for a content map with a large difference between the subject and background colors.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** synthesis results of Architectural drawings and Chinese ink painting</figcaption></figure>

**Experiment 2.** Choose a nature photo as the content image, and several famous Chinese paintings as the style images. After experimental analysis, it is still the best choice for the third layer operation of the network. The experimental results show that ours method also has certain improvement effects. In Figure 2, a nature photo as content, and the style image is one of the top ten Mandarin Chinese For the Thousand Miles of Rivers and Mountains, the result generated by this method is more coordinated overall, and the details are also better.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Synthetic results of nature photos and Thousand Miles of Rivers and Mountains</figcaption></figure>

In addition, since the evaluation criteria of the synthesized results are subjective evaluations, 10 different people were found to conduct subjective evaluations on the result shown in Figure 3. in this experiment. The evaluation results are shown in Table 1. It can be seen that the method in this paper has certain optimization effect on the synthesis result of Chinese painting as a style image.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Synthetic results of nature photos and famous Chinese paintings</figcaption></figure>

<table><caption>**Table 1.** 10 different people praise the synthesis result(Number of people who think the method in this paper is better / Total number)</caption><thead><tr><th>style</th><th>land-<br>scape paint-<br>ing</th><th>All the<br>Pretty<br>Horses</th><th>Han Xizai<br>Ban-<br>quet map</th><th>Thousand<br>Miles of<br>Rivers and<br>Mountains</th><th>Tang<br>Palace<br>Ladies</th><th>Ink<br>painting</th><th>Han<br>Palace<br>spring</th><th>Five<br>Oxen</th></tr></thead><tbody><tr><td>Prairie rate</td><td>0.8</td><td>0.6</td><td>1.0</td><td>0.9</td><td>0.9</td><td>0.6</td><td>0.7</td><td>0.9</td></tr></tbody></table>
## 5 Conclusion

This paper shows a style transfer method for traditional Chinese painting. By improving previous experimental schemes, unique style features in traditional Chinese painting were extracted and added to the feature matrix extracted by the vgg network to achieve the effect of increasing constraints. In the end, a good migration effect was obtained, but it was still unable to achieve the effect of false and real, making people feel that they should draw from the hands of celebrities, and has a certain degree of adaptability to the input image. The main reason is that the paintings in Chinese painting are extremely random and have no regular texture. At the same time, because the evaluation of artistic style images is more subjective, it cannot be objectively evaluated like photographic works. Therefore, we should focus on processing input images and classifying the input images to find the most appropriate content image and style image to match in future. And establish an objective evaluation system for Chinese painting images.

## References

1. Gatys L A, Ecker A S, Bethge M. A Neural Algorithm of Artistic Style[J]. Computer Science, 2015.

2. Gatys L A, Ecker A S, Bethge M. Texture synthesis using convolutional neural networks[C]// International Conference on Neural Information Processing Systems. MIT Press, 2015.

3. Gatys L A, Ecker A S, Bethge M. Image Style Transfer Using Convolutional Neural Networks[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 2016.

4. Li C, Wand M. [IEEE 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) - Las Vegas, NV, USA (2016.6.27-2016.6.30)] 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) - Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis[J]. 2016:2479-2486.

5. Johnson J, Alahi A, Fei-Fei L. Perceptual Losses for Real-Time Style Transfer and Super-Resolution[J]. 2016.

6. Ulyanov D, Lebedev V, Vedaldi A, et al. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images[J]. 2016.

7. Johnson J, Alahi A, Li F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution[M]// Computer Vision - ECCV 2016. 2016.

8. GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]. The 27th International Conference on Neural Information Processing Systems, Montreal, Canada, 2014: 2672-2680.
9. Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[J]. Computer Science, 2015.

10. Arjovsky M, Chintala S, Bottou, Léon. Wasserstein GAN[J]. 2017.

11. GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of wasserstein GANs[C]. The 31st International Conference on Neural Information Processing Systems, Red Hook, USA, 2017: 5769–5779.

12. Zhu J Y, Park T, Isola P, et al. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[J]. 2017.

13. KIM T, CHA M, KIM H, et al. Learning to discover cross-domain relations with generative adversarial networks[C]. The 34th International Conference on Machine Learning, Sydney, Australia, 2017: 1857–1865.

14. Yi Z, Zhang H, Tan P, et al. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation[J]. 2017.

15. Gu S, Chen C, Liao J, et al. Arbitrary Style Transfer with Deep Feature Reshuffle[J]. 2018.

16. Liu Xiangpeng. Anxiety and Breakthrough—A Personal Exploration Journey of Modern Ink Figure Painting [D]. Shandong Normal University, 2004.(in chinese).

17. Huang Qianyan. On the application and emotional expression of lines in the teaching of Chinese painting [J]. 2017, 709(07):100-101.(in chinese).

18. Wu Hongxing. Ink Danqing, Chinese Painting Divine Charm [J]. Jingdezhen Ceramics, 2018, 178(06):18-19.(in chinese).

19. Wang Wenyan. Spatial Forms and Implications of Painting Compositions [D]. Shandong Normal University, 2012.(in chinese).

20. Jiang S, Huang Q, Ye Q, et al. An effective method to detect and categorize digitized traditional Chinese paintings[J]. Pattern recognition letters, 2006, 27(7):p.734-746.

21. Jun-Jie C, Ya-Juan D U. Feature extraction and classification of Chinese Painting[J]. Computer Engineering and Applications, 2008, 74(2):26.