情報処理学会 インタラクション 2020
IPSJ Interaction 2020

3P-81
2020/3/9

CalliShadow: Interactive User Guidance for Calligraphic Practice
Zhizhou He1,a) Haoran Xie1 Kazunori Miyata1
Abstract: The study of calligraphy has always been a challenging problem for people without a Kanji cultural background. It isdifficult for novices to understand the balance in Chinese characters, the stroke order, and the stroke thickness. Previous studies
have been conducted to guide users to write calligraphy through electrical stimulations and have been made to evaluatecalligraphy fonts by recording calligraphy traces through tablet computers. However, the way of electrical stimulation can easilyinterfere with the beautiful font of learners, and it isn't straightforward for users to restore the real environment of traditionalcalligraphy learning. In this work, we propose a calligraphy support system by combining image matching, image blending, andprojection mapping. Providing users with multiple results similar to current writing, the users are instructed on how to write thecorrect strokes. Experimental results show that our system can effectively support calligraphy learners.
Keywords: image matching, image blending, calligraphy projection, stroke orders

1. Introduction
Among the practices of writing Chinese calligraphy characters,
most learners of calligraphy without any background may write
as shown in the left image of Figure 1. The traditional approachof calligraphy practice is to observe Chinese characters inpictures and copybooks. Through the observation and feature
extraction of the Chinese characters in the copybook (see inFigure 2.), the learners tried to imitate the writing methods offamous calligraphers like Xizhi Wang, Xun Ouyang andZhenqing Yan. However, due to the limited ability to understandand master the structure of fonts, the imitation of the fonts has not
only significant defects in balance but also the stroke thickness
cannot be well grasped. To solve this issue the prolonged andrepeated training are usually required. The repeated single
training approaches would increase the boredom of beginners andmake them lose the confidences. Meanwhile, the traditionalcalligraphy practice methods cannot inform learners of thecorrect stroke order. Besides, the writing style in copybooks is sodifferent from the learners that makes the learning be difficult.
The writing style of the typeface in a copybook may not besuitable for novice learners.

Figure 2. Copybook of Xun Ouyang
In our CalliShadow, a calligraphy support system, we use a
new way of projecting mixed images to support calligraphylearners. Compared with previous related research, our system
has the following advantages:⚫ Multi-layer matching calligraphy style recommendation.
⚫ Restore real traditional calligraphy practice environment.
⚫ Including shape detail of each stroke.
⚫ Including stroke writing orders

2. Related Works

Figure 1. Novice font(left) and font in copybook(right)
Therefore, our research goal is to propose a multi-styleselection, stroke order-guided calligraphy practice support systemfor beginners without changing the traditional calligraphy
practice tools.

1 Japan Institute of Science and Technology, Ishikawa, Nomi 923-1292, Japana) hezhizhou2019@gmail.com© 2020 Information Processing Society of Japan

There are some calligraphy support systems to guide thenovice in calligraphy practice. J. Nonami et al. constructed atranscription learning support system using Japanese brushwriting on the tablet without changing the traditional calligraphy
practice [1]. The proposed system recorded the calligraphy
writing trajectory of beginners in real-time. They designed thefont balance system with feedback on font balance to users
through the font balance system they designed. N. Muranaka et al.
adopted a semitransparent screen and a half mirror to simulate
VR space to improve the user's brush movement [2]. Beautifying
Font visualizes the pressure and speed of calligraphy strokes andintuitively guides beginners to learn calligraphy [3]. M.
Kobayashi et al. implemented virtual writing brush calligraphy
with leap motion [4].
There are some training systems for calligraphy and painting.
Calligraphy self-training system correct the beginner's wrong

1077

情報処理学会 インタラクション 2020
IPSJ Interaction 2020

way of writing by electrical stimulation [5]. ShadowDraw is asystem for guiding the free drawing of objects [6]. The proposedsystem updates outline shadows in real-time, instructing users tocontinue painting. Z. Lian et al. used the Coherent Point Drift(CPD) algorithm to realize the automatic deformation of Chinesecharacters [7].
Recently, GAN-based calligraphy graph matching approaches
have been explored. A proposed system extracted Chinesecharacter features based on node and edge detection techniques[8]. This system adopted deep learning matching algorithms tochange the font style of Chinese characters.
There are several ways to achieve the reproduction of
calligraphy traces. F. Peng et al. obtained the writing quality by
extracting the user's calligraphy stroke trajectory from the video
stream and comparing it with the example template [9]. A
calligraphy writing process system realized the animation toreproduce the writing process of ancient handwritten artwork[10].

3. System Design
Callishadow's system design consists of four parts: (1) SystemFramework, system operation process; (2) Database Generation;(3) The usage of matching algorithm between User's calligraphy
and database calligraphy font; (4) According to the degree ofmatching, select different image weights to blend the pictures.
3.1 System Framework

3P-81
2020/3/9

databases. Each set of databases is divided into Chinesecharacters “Yong” in different states.
Database 4 is composed of 20 "Yong" calligraphy pictures offamous calligraphers in Chinese history, and their calligraphy
styles are different. The first stroke of these 20 pictures extracted
as database 3. According to the stroke writing rules of theChinese character "Yong", 20 pictures are divided into 7 writing
states of the corresponding style "Yong" respectively and arestored in database 1. Database 2 consists of 20 "Yong" word
images with their corresponding stroke segmentation images.
(Segmentation of the second to seventh strokes)

Figure 4. A snippet of our database structure
Two detection group databases used to match the user's current
calligraphy writing trajectory in real-time. Two projection group
databases used to give users feedback on the most suitablewriting style and font stroke order. Figure 4 shows thecomposition and quantity of the database.
3.3 Matching Algorithm
HU moments theory has been proposed half-century ago [11].
When the density distribution function changed, the essence of
the image did not change, but the density distribution shifted.
Although the values of the moments may change at this time, theinvariant moments calculated from the moments still have atranslation, rotation, and scale invariance.
Our proposed system adopted the image features of the HU
moments with translation, rotation, and scale invariance.
In the continuous case, the image function is f(x,y), then the p+ q order geometric moment (standard moment) of the image isdefined as:(1)
The center distance of p + q order is defined as:

Figure 3. System Framework
Figure 3 shows the system framework of the proposed
calligraphy writing support system. Learners use calligraphy
brushes dipped in traditional Chinese calligraphy practice ink towrite calligraphy on white paper as a system input. The camera
captures every state of the user's calligraphy in real-time andmatches the user's font data with the font model of database one
in real-time and finds several models with higher matching.
According to the corresponding relationship established between
database 1 and database 2, find the next stroke picture
corresponding to the model. According to the level of matching,determine the weight of the image mixing, the matching is high,and the weight is high. The system gives the user a next strokesuggestion through projection as the output part of the entiresystem.
3.2 Database Generation
The data preparation of the support system consists of four

© 2020 Information Processing Society of Japan(2)
WhereX andY represent the center of gravity of the image.
(3)
For discrete digital images, use summation instead ofintegration:(4)(5)
N and M in Equations 4 and 5 represent the height and widthof the image, respectively.
The physical meaning of each moment:

1078

情報処理学会 インタラクション 2020
IPSJ Interaction 2020

3P-81
2020/3/9

0th moment (m00): the quality of the target area
1st moment (m01, m10): the centroid of the target area
2nd moment (m02, m11, m20): the radius of rotation of thetarget area
⚫ 3rd moment (m03, m12, m21, m30): the orientation and slopeof the target area, reflecting the distortion of the target
The normalized center distance is defined as:
⚫
⚫
⚫

(6)
Using the second- and third-order normalized center moments,seven invariant moments M1 ~ M7 are constructed:

groups 1 are familiar with Chinese characters, and participants ingroups 2 are entirely unfamiliar with Chinese characters. The 2
groups of experimenters used the traditional practice method ofobserving copybook pictures to practice calligraphy as a control
experiment. The calligraphy practice support system provides
projections of 1, 3, and 5 layers of images, as well as projectionsof gray scale and color font strokes.
The system provides full word writing support projection forexperimenters and the next writing support projection based on
the current font status of the experimenter. Figure 6 shows theexperimenter using colored stroke hints and grayscale stroke
hints.
We provide multiple layers of mixed picture projections to giveexperimenters more imitative choices in the process of practicing
calligraphy. At the same time, give experimenters more stroke
information to increase the fault tolerance rate of beginners incalligraphy.
Color and gray scale are to verify whether the color hints havean effect on the experimenter.

(7)
These seven invariant moments constitute a set of feature
quantities that are used to identify large objects in the image,such as characters, which have better results.
3.4 Image Weighted Blending
The proposed system sorts the database matching pictures fromhigh to low. The image mixture is blended in pairs from thelowest image to the highly matching image, and the weight of thetwo images is set to 50% each. The system mixes selected
matching pictures in the database multiple times to form a mixed
picture with gradually decreasing transparency (see Figure 5).

Figure 6. Comparison experiments of writing guidance projection
With color hints(left) and with gray-scale hints (right)

5. Results and Discussions
The questionnaire results show that the proposed support
system is helpful for the balance practice of calligraphy fonts incontrast to the traditional calligraphy practice mode. The
participants who are unfamiliar with Chinese characters foundthat the stroke support to help them understand the composition
and writing order of Chinese characters well (see in Figure 7).

Figure 5. Weighted blending images
Three layers color (left) and three layers gray-scale (right)

4. Experiments
In our experiments, there were 20 participants (5 female and 15men. The age range is between 22 and 33 years.), which are
divided into 2 groups of 10 people each. The participants in© 2020 Information Processing Society of Japan1079

情報処理学会 インタラクション 2020
IPSJ Interaction 2020

3P-81
2020/3/9

6. ConclusionCalliShadow system supportdegrees
76543210664Very helpful

3Helpful

1Not obvious
00 0Familiar

Not familiar

HelplessFig. 7. CalliShadow system support degrees
The results of experimental data show that both learners who
are familiar with calligraphy and learners who are not familiar
with calligraphy prefer colored single stroke hints. They thoughtthat colored fonts are easier to distinguish between different
matching degrees (see in Figure 8).

We proposed a calligraphy writing support system withcamera-projector system. The system Provide users with multiple
results similar to current writing, the users are instructed on how
to write the correct strokes. Evaluation experiments show that thesystem is helpful to understand the stroke order and font structureof calligraphy beginners who are unfamiliar with Chinesecharacters, and the system dramatically improves the learning
effect in the early stage of calligraphy practice.
As future work, the proposed system needs more accurate
algorithm to improve real-time matching due to the limitations of
the contour matching algorithm.

Reference[1][2]

System mode preferences
987654321086

Gray-scale
whole wordhints
Gray-scale
single strokehints

21 11 10Familiar

Not familiar[4]

Color wholeword hints
Color single
stroke hints

Figure 8. CalliShadow system mode preferences
The results show that the number of layers in the picture willcause disturbance to those who are familiar with calligraphy inFigure 9. Calligraphy novices would like to choose the numberof layers of mixed font image, based on their personal preference.

Blending image layers
preferences
876543210[3]7[5]

[6]

[7]
[8]

[9]

433 3

One layer
Three layers
Five layers

0Familiar[10]

Not familiar

Figure 9. Blending image layers preferences© 2020 Information Processing Society of Japan[11]

J. Nonami and Y. Takegawa, “Construction of a support
system for learning character balance in transcription forbeginners,” 2014 IEEE 3rd Glob. Conf. Consum.
Electron. GCCE 2014, pp. 26–30, 2014.
N. Muranaka, T. Yamamoto, and S. Imanishi, “ACalligraphy Mastering Support System Using Virtual
Reality Technology and its Learning Effects” vol. 123,
no. 12, 2003. IEE Jpn. Trans. Electron. Inf. Syst.
117-C(11), 1629-1634(1997) (in Japanese)
M. Nakamura, S. Yamaguchi, and S. Morishima,“Beautifying font: Effective handwriting template formastering expression of Chinese calligraphy,” ACM
SIGGRAPH 2017 Posters, SIGGRAPH 2017, 2017.
M. Kobayashi, Y. Tashiro, M. Nakajima, and T. Saitoh,“Development of a Japanese calligraphic brush writing
simulation system using leap motion,” 2018 Int. Work.
Adv. Image Technol. IWAIT 2018, pp. 1–4, 2018.
A. Morikawa, N. Tsuda, Y. Nomura, and N. Kato,“Double Pressure Presentation for CalligraphySelf-training,” ACM/IEEE Int. Conf. Human-Robot
Interact., no. 1, pp. 199–200, 2018.
Y. J. Lee, C. L. Zitnick, and M. F. Cohen, “ShadowDraw:
Real-Time User Guidance for Freehand Drawing,” ACM
Trans. Graph., vol. 30, no. 4, pp. 1–10, 2011.
Z. Lian and J. Xiao, “Automatic shape morphing for
Chinese characters,” vol. 1, no. 212, pp. 1–4, 2012.
P. Lyu, X. Bai, C. Yao, Z. Zhu, T. Huang, and W. Liu,
“Auto-Encoder Guided GAN for Chinese Calligraphy
Synthesis,” Proc. Int. Conf. Doc. Anal. Recognition,ICDAR, vol. 1, pp. 1095–1100, 2018.
P. Xu et al., “Evaluating brush movements for Chinesecalligraphy: A computer vision based approach,” IJCAIInt. Jt. Conf. Artif. Intell., vol. 2018-July, pp. 1050–1056,
2018.
L.-J. Yang, T.-C. Xu, X.-S. Li, and E.-H. Wu,“Feature-oriented writing process reproduction ofChinese calligraphic artwork,” pp. 1–4, 2014. SA '14
SIGGRAPH Asia 2014 Technical Briefs Article No. 5
M.- Kuei, “Visual Pattern Recognition by Moment
Invariants,” IRE Trans. Inf. Theory, vol. 8, no. 2, pp.
179–187, 1962.

1080