Proceedings of 1st International Symposium on Color Science and Art 2018

Artificial Intelligence Artistic Painting Mirror
as Interactive Art Using Deep Neural Networks
Yasuo Kuhara
Tokyo Polytechnic University, 2-9-5 Honcho, Nakano, Tokyo 164-8678, Japan

input from the camera with artistic styles featuring thecharacteristics of various painters using artificial
intelligence called deep learning. When a viewer performs
various actions, they are transformed into images
expressed in a painting style in real time, and this isreflected on an organic EL display like a mirror. This styletransformation is performed by a convolution neuralnetwork learning visual perception such as object and facerecognition. In the examples in Figure 2, the right-handside is the style image and the left-hand side is thegenerated image. We can understand that the original
image is repainted with the style image, and not only color
sense but also fine space patterns are expressed
according to the painter's style. Works used for learning
are Da Vinci, Van Gogh, Renoir, Munch, Picasso, Matisse,
Mondrian, Sharaku, and Hokusai, as well as ancient
Egyptian mural paintings. Viewers can take various posesin front of the camera and enjoy their own figure drawn in aparticular painting style (see Figure 1).

ABSTRACTThis paper reports a summary of the project named“Basic research on interactive art using deep learning tocreate color expression,” which has been adopted as a
private university research branding project on theresearch of color science and art. In this project, I havedeveloped an interactive artform named “AI Artistic
Painting Mirror” using deep-learning neural networks. This
work transforms images from a camera into an artisticstyle movie in real time, as if it were a mirror, using deepneural networks that have learned various styles ofpainters in advance. The AI artistic painting mirror has
been exhibited at several galleries and festivals. It has
achieved stable performance and a good reputation.
1. INTRODUCTION
Artificial intelligence (AI) has interesting possibilities in
the field of art, where intuitions and feelings are important.
For example, applications that transform photographs into
artist-styled paintings are very popular. Several libraries
for artistic transformation by neural networks have beendeveloped and published on the Internet. For example, E.
Matsumoto developed “chainer-gogh” [1], which is an
implementation of the “Neural Algorithm of Artistic Style”
by Gatys et al. [2]. Furthermore, Y. Tomoto developed“chainer-fast-neuralstyle” [3], which achieved fastertransformation than chainer-gogh. As an application of
fast artistic style transfer using neural networks, I havedeveloped an interactive artform named “AI ArtisticPainting Mirror” (see Figure 1).
2. CONCEPTPainters express what they see with their own eyes asa painting based on their own unique sensibility. They
cannot explain it in words, because it is beyond thealgorithmic process and reflects the unique visualexperience that painters have been exploring. This work
is an interactive painting form, which creates an image

Fig. 1 AI Artistic Painting Mirror

Fig. 2 Sample Pictures23Proceedings of 1st International Symposium on Color Science and Art 2018
3. METHOD
3.1 System Configuration
For a calculation for deep learning, we should use ahighly specialized general-purpose computing on graphics
processing units (GPGPU) workstation, which is acomputer optimized for deep learning. For this equipment,
we employed HPC Systems’ HPC5000-XBWGPU4TS,
which has a Xeon processor, and four NVIDIA Teslaseries GPUs, in a specialized computer room for deeplearning in the facilities of our university. We can connect
and use it directly and through our university intranet.
3.2 Neural Networks
We use the chainer-fast-neuralstyle application by
Tomoto [3]. It is a Chainer implementation of “PerceptualLosses for Real-time Style Transfer” by Johnson et al. [4].
This method builds on “A Neural Algorithm of Artistic Style”
by Gatys et al. by training feedforward neural networks
that apply artistic styles to images [2]. Johnson’s methodcan stylize images hundreds of times faster than theoptimization-based method presented by Gatys. They
train one image transformation network model per one
style target. This models are trained on the Microsoft
COCO dataset, which has gathered images of complex
everyday scenes containing common objects labeledusing per-instance segmentations to aid in precise object
localization [5].
3.3 Performance
Response from camera input to display artistic images
is important as an interactive arrform. Using GPU
increases the calculation performance dramatically, ten or
hundred times more than normal CPU. Additionally, imageresolution relates to performance. A high-resolution image
creates beautiful graphics but the latency is slow. In thecase of high vision quality, the response time is 1 to 2seconds. On the other hand, in the case of VGA quality,
the response time is 0.2 to 0.5 seconds. In the exhibition,
we prioritize graphical beauty for a painting of fine art.

11, 2018. Chika means an underground pavement. At thisfestival, many families, children, and adults, accounting for975 visitors, the Mayor, and Ayukoro-chan as the city’s
mascot watched it and enjoyed their figures transformed
into artistic style (see the lower left and right of Figure 3).
5. PERSPECTIVE
We have developed the AI Artistic Painting Mirror. In
the exhibition, visitors enjoyed our interactive work morethan we expected. In the future, we have plans forcollaboration with other artists such as photographers andexhibition-holders in foreign countries. Neural networks
can learn any style of artistic graphics for adapting to
specific artists or museums. We hope our work in various
display styles will prevail.
6. ACKNOWLEDGMENT
This research was supported by “Polytechnic Color
Science and Art Creating Tomorrow, Studying the Future,
and Connecting the World through Color” as a private
university research branding project.
7. REFERENCES[1] E. Matsumoto, “chainer-gogh,”
https://github.com/yusuketomoto/chainer-fast-neuralstyle.
[2] Gatys et al., “A Neural Algorithm of Artistic Style,”
http://arxiv.org/abs/1508.06576, 2015.
[3] Y. Tomoto, “Chainer implementation of Perceptual
Losses for Real-Time Style Transfer andSuper-Resolution,”
https://github.com/yusuketomoto/chainer-fast-neuralstyle.
[4] J. Johnson et al., “Perceptual Losses for Real-Time
Style Transfer and Super-Resolution,”
https://arxiv.org/abs/1603.08155, 2016.
[5] T. Lin et al., “Microsoft COCO: Common Objects inContext,” https://arxiv.org/abs/1405.0312,
http://cocodataset.org/.

4. EXHIBITION
4.1 Color and Science Laboratory Gallery
I am attending the rotating exhibition entitled “Explorerof Color” of the International Research Center for Color
Science and Art at Tokyo Polytechnic University (see theupper left of Figure 3). This exhibition is held from
September 15, 2018 to April 19, 2019. Every month, two or
three hundred visitors including students from preschool tohigh school visit our gallery and enjoy this exhibition.
4.2 Festa in the Faculty of Arts
We attended the annual art event “Gather Art! Spread
Art! ” of the Faculty of Arts at Tokyo Polytechnic University(see the upper right of Figure 3). This exhibition was heldon October 7 - 24, 2018. More than two thousand visitors
enjoyed this exhibition.
4.2 Chika Fes of Atsugi City
Responding to a request from Atsugi City, we exhibited
this work at Chika Fes of Atsugi City on November 10 andFig. 3 Exhibitions

24