# Learning of Art Style Using AI and Its Evaluation Based on Psychological Experiments

Cong Hung Mai, Ryohei Nakatsu, Naoko Tosa, Takashi Kusumi, Koji Koyamada

► To cite this version:

Cong Hung Mai, Ryohei Nakatsu, Naoko Tosa, Takashi Kusumi, Koji Koyamada. Learning of Art Style Using AI and Its Evaluation Based on Psychological Experiments. 19th International Conference on Entertainment Computing (ICEC), Nov 2020, Xi'an, China. pp.308-316, 10.1007/978-3-030-65736-9_28. hal-03686030

HAL Id: hal-03686030

https://inria.hal.science/hal-03686030

Submitted on 2 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Learning of Art Style Using AI and Its Evaluation
## Based on Psychological Experiments

Mai Cong Hung¹,⁵, Ryohei Nakatsu², Naoko Tosa³, Takashi Kusumi⁴,
Koji Koyamada²

¹Graduate School of Faculty of Science, Kyoto University, Japan

²Academic Center for Computing and Media Studies, Kyoto University, Japan

³Graduate School of Advanced Integrated Studies in Human Survivability, Kyoto
University, Japan

⁴Graduate School of Education, Kyoto University, Japan

⁵RIKEN, Japan

hungmcuet@gmail.com, nakatsu.ryohei@gmail.com, tosa.naoko.5c@kyoto-u.ac.jp,
kusumi.takashi.7u@kyoto-u.ac.jp, koyamada.koji.3w@kyoto-u.ac.jp

**Abstract.** GANs (Generative adversarial networks) is a new AI technology that has the capability of achieving transformation between two image sets. Using GANs we have carried out a comparison between several art sets with different art styles. We have prepared four image sets; a flower image set with Impressionism art style, one with the Western abstract art style, one with Chinese figurative art style, and one with the art style of Naoko Tosa, one of the authors. Using these four sets we have carried out a psychological experiment to evaluate the difference between these four sets. We have found that abstract drawings and figurative drawings are judged to be different, figurative drawings in West and East were judged to be similar, and Naoko Tosa's artworks are similar to Western abstract artworks.

**Keywords:** GANs, Cycle GAN, art history, transformation of art style, Impressionism, abstraction

# 1 Introduction

Recently a new technology of deep learning in AI called GANs (Generative Adversarial Networks) has been proposed [1], and various attempts to create artworks by AI have been carried out. However, many of these methods merely let AI learn the style of a particular painter and output images with the learned style. Is there a different approach to the relationship between AI and art? For example, can AI approach basic questions such as what beauty is that exists at the basis of art, and what the difference between Oriental and Western perceptions of beauty is?

In this paper, a new methodology for approaching the relationship between AI and art will be proposed, and the results of verification through psychological experiments will be shown.
## 2 Related Works

Recently a new learning method called GANs (Generative Adversarial Networks), that can perform deep learning with a relatively small number of training data, has been proposed [1]. GANs are composed of two networks; a generator network and a discriminator network. By performing learning as a zero-sum game between these two networks, deep learning can converge even with a relatively small number of learning data. By modifying this basic configuration, various GANs have been proposed and interesting results have been obtained.

<figure><img src="image_2.png" /><figcaption>**Fig. 1** Basic concept of Cycle GAN.</figcaption></figure>

Among them, Cycle GAN is a new method that enables mutual conversion between two image sets. Fig. 1 shows the basic concept of Cycle GAN [2]. In Cycle GAN, when two image sets (X, Y) are given, a transformation function G and an inverse transformation function F between them are considered. Also, two types of errors, Dx and Dy are considered; Dx is a difference between X and X' where X' is the transformation of X by applying G then F and Dy is an error caused by the difference between Y and Y' where Y' is a transformation of Y by applying F and then G. The training is carried out so that the sum of these two types of errors is minimized (Fig. 1).

The feature of Cycle GAN is that, even if there was no one to one correspondence between image sets X and Y, the conversion between them is possible. By using this feature, for example, by learning a group of landscape photographs and a group of paintings of a specific painter, mutual conversion between these image groups becomes possible. Figure 2 shows how a Monet-style landscape painting is created from a landscape photograph [2].

<figure><img src="image_3.png" /></figure>
<caption>**Fig. 2.** Conversion from a landscape photo to a Monet painting using Cycle GAN. ([3])</caption>

**3. Fluid art “Sound of Ikebana”**

The behavior of fluid is an important research subject in physics, and has been studied as "fluid dynamics." It is known that fluid creates extremely beautiful shapes under various conditions. As beauty is a fundamental element of art, it is natural to consider fluid dynamics as a basic methodology of art creation. Naoko Tosa, one of the authors, has been leading a project of creating “fluid art” by shooting the behavior of fluid with a high-speed camera.

One of the techniques for creating fluid art is the creation of Ikebana-like shapes when sound vibration is applied to paint or other fluids and the phenomenon is shot with a high-speed camera. The detailed process is as follows. A speaker is placed upward, a thin rubber film is put on top of it, and a fluid such as paint is placed on top of it, and the speaker is vibrated with sound, then the paint jumps up and various shapes are created. Based on this method Naoko Tosa created a video art called "Sound of Ikebana" [3]. In April 2017, she exhibited Sound of Ikebana using more than 60 digital billboards at Times Square in New York. Figure 3 shows the event.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** "Sound of Ikebana" at Times Square in New York.</figcaption></figure>

It was interesting that the video art mentioned above let the authors consider what beauty is and also what Japanese beauty is. When Tosa, exhibited her media art around the world as Japan Cultural Envoy named by the Agency for Cultural Affairs in Japan, many foreign art-related people indicated, "In Tosa's media art, which expressed the beauty hidden in physical phenomena, there is the beauty which Westerners did not notice until now, and which might be the condensed consciousness and sensitivity unique to Japan." Inspired by this, the authors would like to compare Western and Oriental artworks using AI.

**4. Framework of This Research**

As described in Chapter 2, Cycle GAN can be used to carry out the transformation between two image sets, even if there is no one-to-one correspondence between images belonging to each image set. So far, reference [3] merely states that landscape photographs could be converted into Monet-style paintings and vice versa. But the
authors consider that Cycle GAN could be applied to research investigating the relationship between art and beauty. This paper is based on this basic thought. Considering that art is an essential feature extracted from real objects or natural phenomena, it is possible to use Cycle GAN to convert between real objects or natural phenomena and art that extracts their essence.

In the long history of art, the painting originally tried to imitate nature. Western realism is the extension of this trend. As the times go down, however, the impressionism is born that tries to paint the light and its transitions perceived by human eyes, rather than trying to paint nature as it is. At this stage, the form of the objects depicted is still clear. Later, however, the history of Western painting was followed by Cubism and Surrealism, then followed by more recent abstract paintings. Nevertheless, it can be said that artists extract essential things they felt in their hearts from the surrounding nature and made them into abstract paintings.

On the other hand, the history of Oriental painting is characterized by the fact that the painted objects have been clear since ancient times. Rather, it is characterized by the direction of minimalism that removes color like ink painting and by the way of drawing emphasizing the characteristics of the object like Ukiyo-e in Japan and remains at the level of figurative painting compared to the West.

In such a situation, how is “Sound of Ikebana” described earlier positioned in the history of Oriental painting? It does not depict landscapes either human life and looks like abstract images and videos. Nevertheless, as mentioned earlier, many people overseas have said "The artwork has the feeling of Japanese beauty." Is it possible to use AI style learning and style conversion functions to find out how Sound of Ikebana is positioned compared to Western and Oriental figurative and abstract paintings?

In this study, this important and interesting issue is approached using Cycle GAN.

(1) Two types of image sets (image set A, and multiple image sets B1, B2, ...) are prepared. The image set A is to be converted into art. A multiple image sets B1, B2, ... consist of art images.

(2) Using Cycle GAN, mutual conversion (Fig. 4) of an image set A and multiple image sets B1, B2, ... are achieved to obtain conversion functions (G1, F1), (G2, F2), ...

<figure><img src="image_5.png" /><figcaption>Fig. 4. Mutual conversion between two types of image sets.</figcaption></figure>

(3) A psychological experiment is performed using the image sets G1 (A), G2 (A), ... that are obtained by performing the conversion G1, G2, ... to the image set A. Depending on the purpose of the psychological experiment, questionnaires such as "Can you evaluate it as art?" "Do you feel beauty?" will be filled by the subjects.
By doing this, it is possible to verify depending on different art styles what kind of information is extracted as essential information from real objects and made into artworks.

**5. Learning of Various Art Style and Transformation of Art Style**

The following image sets were prepared. (Resolution of all images are 256x256)
Image set A: 8069 flower images
Image set B1: 1072 Monet paintings in the Monet2Photo dataset [2].
Image set B2: 123 Kandinsky abstract paintings from Wiki Art.
Image set B3: 238 Chinese hand-painted flower paintings called “Gongbi” from Stanford University project "Chinese Painting Generation Using Generative Adversarial Networks."
Image set B4: 569 images selected from "Sound of Ikebana"

The image set B1 includes paintings mainly for flowers drawn by the Impressionist Monet as a representative example of the Western figurative paintings. As a representative example of Western abstract paintings, Kandinsky paintings were prepared as image set B2. Image set B3 includes flower paintings of Chinese hand-painted painting, called “Gongbi [5],” as a representative example of Oriental figurative paintings. Image Set B4 is a set of still images taken from the media art "Sound of Ikebana" created by one of the authors, Naoko Tosa. The Sound of Ikebana is a video artwork created by shooting physical phenomena with a high-speed camera. As it is made from a physical phenomenon, it should not be said that it originally contains "Japanese beauty." In this experiment, the art style of the Sound of Ikebana was compared with Western and Oriental representative painting styles.

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Examples of mutual conversion between flower photos and Sound of Ikebana style images. (a: Flower to “Sound of Ikebana” style, b: “Sound of Ikebana” to flower-like)</figcaption></figure>

The four types of different image set B1, B2, B3, B4 and the image set A including were mutually converted using the Cycle GAN. Figure 5 shows examples of the result of the mutual conversion between the Sound of Ikebana and the photograph of the flower.

The question here is why it is necessary to use AI for comparative evaluation of art styles. Why Monet art, Kandinsky art, Chinese Gongbi art, and Sound of Ikebana images were not directly compared and evaluated using psychological experiments. There have been some studies that evaluated artworks through psychological
experiments [4]. However, by using a copy of the original artwork, it can be relatively easy to identify the artist for each art. For example, knowing that a painting is Monet's art suggests that a subject has a prejudice of the work of Monet, a representative of the Impressionists in Western art history and that this will have a significant effect on evaluation experiments. To avoid this effect, there are research examples of using lesser-known works [4]. However, works of famous artists and art schools can be easily identified and would greatly affect evaluation experiments. On the other hand, using GAN allows for AI to learn an art style and to apply the art style on the input images with the art style from the input. Therefore, bias can be avoided in the evaluation experiment and this is the benefit of using AI to evaluate artworks.

# 6. Evaluation of Obtained Results Based on Psychological Experiment

The experiment described in Chapter 5 yielded results of performing various style conversions on flower images. By having people evaluate the results of applying various style transformations to various flower images, is it possible to know what art is, what is the beauty behind it, and the culture of beauty? Is there any suggestion on how people receive Japanese beauty and the corresponding Western beauty? That is the goal of this research. Since this is a subjective evaluation, a method used in psychological experiments, which is to present a target image to a subject, to conduct a questionnaire survey, and statistically analyze the results, was used.

## 6.1 Psychological experiment

Image group Group1, Group2, Group3, and Group4 are prepared by selecting two images from each of image sets G1(A), G2(A), G3(A), and G4(A), which are obtained by converting image set A into image sets B1, B2, B3, and B4. The resolution of each image is 256x256.

Twenty three Kyoto University students (12 male and 11 female, all are Japanese) were used as subjects. The gender ratio is almost half. Each of 8 images was printed out on A4 high-quality paper, and the eight images were presented to the subjects. The order of the presented images was set randomly for each subject. The subjects were asked to perform a seven-step subjective evaluation of the 6 items shown in Table 1. These items were selected to identify the difference between the Oriental and Western art styles.

<table><caption>**Table 1.** Adjective pairs used for evaluation</caption><tr><td>Individual - Ordinary</td><td>Bold - Careful</td></tr><tr><td>Dynamic - Static</td><td>Artistic - Non-artistic</td></tr><tr><td>Stable - Unstable</td><td>Oriental - Western</td></tr></table>
## 6.2 Analysis

The results of the subjective evaluations by the 23 subjects and for six items, which are Individual-Ordinary, Dynamic-Static, Stable-Unstable, Bold-Careful, Artistic-Non artistic, Oriental-Western, were averaged for each evaluation item, graphed, and t-tested. Figures 6 – 8 show the results of the averaged value and the standard error for each evaluation item. Also, the results of t-analysis (**:1%, *:5%) are shown on these figures.

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Subjective evaluation results for "individual-ordinary" (left) and "Dynamic-Static" (right). (Group1: Monet-style flower images, Group2: Kandinsky-style flower images, Group3: Chinese figurative art-style flower images, Group4: Naoko Tosa art-style flower images.)-</figcaption></figure>

<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /><figcaption>**Fig. 8** Subjective evaluation results for "Artistic-Non-artistic" (left) and "Oriental-Western" (right). (Group1: Monet-style flower images, Group2: Kandinsky-style flower images, Group3: Chinese figurative art-style flower images, Group4: Naoko Tosa art-style flower images.)-</figcaption></figure>

## 6.3 Consideration

### (1) Group 2 vs Group 4

Group 2 and Group 4 received similar evaluations. This indicates that there is a small significant difference between the style of Kandinsky and the style of Sound of Ikebana. Conversely, Group 4 is evaluated as having a significant difference from Group 1 and Group 3 for all items except "Oriental-Western". This indicates that the Sound of Ikebana is considered to be abstract rather than figurative. If the transition from figurative painting to abstract painting is the history of Western painting, the Sound of Ikebana can be said to be positioned in the history of the transition from figurative painting to abstract painting in the Orient.

### (2) Group 1 vs Group 3

Similarly, Group 1 and Group 3 received similar evaluations. In particular, the hypothesis that there is a significant difference of 5% level between the "Dynamic-Static" and "Artistic-Non artistic" items have been rejected. Group 1 is an image set with a style of Western Impressionism, and Group 3 is an image set with the characteristics of Oriental figurative painting. Each of the original art could be judged that each has its characteristics, But in a more essential part, these styles may have something in common.

### (3) Artistic or not

It is interesting to note that Group 1 and Group 3 are around or below the median value of 4 for "Artistic-Non artistic." Few people will rate Monet's original image as unartistic. Chinese hand-painted paintings have also been highly evaluated as elaborately depicting nature. However, Groups 2 and 4 are evaluated as being higher artistic than Groups 1 and 2. This is thought to be due to the young age of the subject. As the younger generation has more opportunities to watch abstract drawings and recent media art, they may have an aesthetic sense appreciating abstract paintings. Also, this means that our principle of using the converted photos with a specific art
style instead of the original artworks worked well. By using the original art image for
evaluation, it was relatively easy to identify who the artist was, or even the specific
work itself, and it is guessed that this had a significant effect on the evaluation.

(4) Oriental or Western

As shown in Fig. 8, the answers to the question of Oriental or Western are all
around the median of 4 except for Group 2. This indicates that the subject did not
identify whether Oriental or Western for the artworks in these groups, and gave a
response near the middle. Initially, the authors expected that the Sound of Ikebana
would be evaluated as "Oriental" because of the overseas evaluation that the artwork
contains Japanese beauty. But so far, such a result was not obtained. At the same time,
Monet-style images and Chinese Gongbi-style images were evaluated as intermediate.
This seems to indicate that at this time, the art style extracted by AI has not yet
reached a level to identify Western or Oriental impression.

7. Conclusion

In this paper, a new method of handling art with AI was described by using GANs
to investigate where is the difference in art style, and what is the essence of the
difference in aesthetic sense between Oriental and Western beauty.

By using the method proposed in this paper, subjects evaluate the styles of Western
and Oriental figurative and abstract paintings without bias created by identifying
specific artists and/or artworks. As a result, it was shown that the figurative drawings
of the Orient and the West are not very different. Also, it was shown that one of the
authors' works "Sound of Ikebana," has no significant difference from the abstract
drawings of the Western abstract drawings.

However, it is not enough to clarify why the Sound of Ikebana is evaluated by
Westerners as Oriental in the scope of this study. It is future work to clarify this. Also,
it is necessary to use Western subjects to know the difference of sensitivity between
Oriental and Western people.

Reference

1. Antonia Creswell, et al., “Generative Adversarial Networks: An Overview,” IEEE Signal Processing Magazine, Vol.35, No.1, pp.53-65 (2018).

2. Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” The IEEE International Conference on Computer Vision (ICCV), pp.223-2232 (2017).

3. Naoko Tosa, Yunian Pang, Qin Yang, Ryohei Nakatsu, “Pursuit and Expression of Japanese Beauty Using Technology,” Arts journal, MDPI, Vol.8, No.1, 38, DOI (2019).

4. K. Freedman, “Judgement of Painting Abstraction, Complexity, and Recognition by Three Adult Educational Groups,” Visual Arts Research, Vol.14, pp68-78 (1988).

5. https://en.wikipedia.org/wiki/Gongbi