IIEEJ Transactions on
IIEEJ Transactions on
Image Electronics and Visual Computing

Image Electronics andVisual Computing

Special Issue on Extended Papers Presented in IEVC2019 Part Ⅱ
Special Issue on CG & Image Processing Technologies for Automation,
Labor Saving and Empowerment

Vol. 8, No. 2 2020

I.I.E.E.J

Editorial Committee of IIEEJ
Editor in Chief
Mei KODAMA (Hiroshima University)

Vice Editors in Chief
Osamu UCHIDA (Tokai University)
Naoki KOBAYASHI (Saitama Medical University)
Yuriko TAKESHIMA (Tokyo University of Technology)

Advisory Board
Yasuhiko YASUDA (Waseda University Emeritus)
Hideyoshi TOMINAGA (Waseda University Emeritus)
Kazumi KOMIYA (Kanagawa Institute of Technology)
Fumitaka ONO (Tokyo Polytechnic University Emeritus)
Yoshinori HATORI (Tokyo Institute of Technology)
Mitsuji MATSUMOTO (Waseda University Emeritus)
Kiyoshi TANAKA (Shinshu University)
Shigeo KATO (Utsunomiya University Emeritus)

Editors
Yoshinori ARAI (Tokyo Polytechnic University)
Chee Seng CHAN (University of Malaya)
Naiwala P. CHANDRASIRI (Kogakuin University)
Chinthaka PREMACHANDRA (Shibaura Institute of Technology)
Makoto FUJISAWA(University of Tsukuba)
Issei FUJISHIRO (Keio University)
Kazuhiko HAMAMOTO (Tokai University)
Madoka HASEGAWA (Utsunomiya University)
Ryosuke HIGASHIKATA (Fuji Xerox Co., Ltd.)
Tomokazu ISHIKAWA（Toyo University）
Masahiro ISHIKAWA （Saitama Medical University）
Naoto KAWAMURA (Canon OB)
Shunichi KIMURA (Fuji Xerox Co., Ltd.)
Shoji KURAKAKE (NTT DOCOMO)
Kazuto KAMIKURA (Tokyo Polytechnic University)
Takashi KANAI (The University of Tokyo)
Tetsuro KUGE (NHK Engineering System, Inc.)
Koji MAKITA (Canon Inc.)
Junichi MATSUNOSHITA (Fuji Xerox Co., Ltd.)
Tomoaki MORIYA (Tokyo Denki University)
Paramesran RAVEENDRAN (University of Malaya)
Kaisei SAKURAI (DWANGO Co., Ltd.)
Koki SATO (Shonan Institute of Technology)
Syuhei SATO（University of Toyama）
Masanori SEKINO (Fuji Xerox Co., Ltd.)
Kazuma SHINODA (Utsunomiya University)
Mikio SHINYA (Toho University)
Shinichi SHIRAKAWA (Aoyama Gakuin University)
Kenichi TANAKA (Nagasaki Institute of Applied Science)
Yukihiro TSUBOSHITA (Fuji Xerox Co., Ltd.)
Daisuke TSUDA (Shinshu University)
Masahiro TOYOURA (University of Yamanashi)
Kazutake UEHIRA (Kanagawa Institute of Technology)
Yuichiro YAMADA (Genesis Commerce Co.,Ltd.)
Norimasa YOSHIDA (Nihon University)
Toshihiko WAKAHARA (Fukuoka Institute of Technology OB)
Kok Sheik WONG (Monash University Malaysia)

Reviewer
Hernan AGUIRRE (Shinshu University)
Kenichi ARAKAWA (NTT Advanced Technology Corporation)
Shoichi ARAKI (Panasonic Corporation)
Tomohiko ARIKAWA (NTT Electronics Corporation)
Yue BAO (Tokyo City University)
Nordin BIN RAMLI (MIMOS Berhad)
Yoong Choon CHANG (Multimedia University)
Robin Bing-Yu CHEN (National Taiwan University)
Kiyonari FUKUE (Tokai University)
Mochamad HARIADI (Sepuluh Nopember Institute of Technology)
Masaki HAYASHI (UPPSALA University)
Takahiro HONGU (NEC Engineering Ltd.)

Yuukou HORITA (University of Toyama)
Takayuki ITO (Ochanomizu University)
Masahiro IWAHASHI (Nagaoka University of Technology)
Munetoshi IWAKIRI (National Defense Academy of Japan)
Yuki IGARASHI (Meiji University)
Yoshihiro KANAMORI (University of Tsukuba)
Shun-ichi KANEKO (Hokkaido University)
Yousun KANG (Tokyo Polytechnic University)
Pizzanu KANONGCHAIYOS (Chulalongkorn University)
Hidetoshi KATSUMA (Tama Art University OB)
Masaki KITAGO (Canon Inc.)
Akiyuki KODATE (Tsuda College)
Hideki KOMAGATA (Saitama Medical University)
Yushi KOMACHI (Kokushikan University)
Toshihiro KOMMA (Tokyo Metropolitan University)
Tsuneya KURIHARA (Hitachi, Ltd.)
Toshiharu KUROSAWA (Matsushita Electric Industrial Co., Ltd. OB)
Kazufumi KANEDA (Hiroshima University)
Itaru KANEKO (Tokyo Polytechnic University)
Teck Chaw LING (University of Malaya)
Chu Kiong LOO (University of Malaya) F
Xiaoyang MAO (University of Yamanashi)
Koichi MATSUDA (Iwate Prefectural University)
Makoto MATSUKI (NTT Quaris Corporation OB)
Takeshi MITA (Toshiba Corporation)
Hideki MITSUMINE (NHK Science & Technology Research Laboratories)
Shigeo MORISHIMA (Waseda University)
Kouichi MUTSUURA (Shinsyu University)
Yasuhiro NAKAMURA (National Defense Academy of Japan)
Kazuhiro NOTOMI (Kanagawa Institute of Technology)
Takao ONOYE (Osaka University)
Hidefumi OSAWA (Canon Inc.)
Keat Keong PHANG (University of Malaya)
Fumihiko SAITO (Gifu University)
Takafumi SAITO (Tokyo University of Agriculture and Technology)
Tsuyoshi SAITO (Tokyo Institute of Technology)
Machiko SATO (Tokyo Polytechnic University Emeritus)
Takayoshi SEMASA (Mitsubishi Electric Corp. OB)
Kaoru SEZAKI (The University of Tokyo)
Jun SHIMAMURA (NTT)Tomoyoshi SHIMOBABA (Chiba University)
Katsuyuki SHINOHARA (Kogakuin University)
Keiichiro SHIRAI (Shinshu University)
Eiji SUGISAKI (N-Design Inc. (Japan), DawnPurple Inc.(Philippines))
Kunihiko TAKANO (Tokyo Metropolitan College of Industrial Technology)
Yoshiki TANAKA (Chukyo Medical Corporation)
Youichi TAKASHIMA (NTT)Tokiichiro TAKAHASHI (Tokyo Denki University)
Yukinobu TANIGUCHI (NTT)Nobuji TETSUTANI (Tokyo Denki University)
Hiroyuki TSUJI (Kanagawa Institute of Technology)
Hiroko YABUSHITA (NTT)
Masahiro YANAGIHARA (KDDI R&D Laboratories)
Ryuji YAMAZAKI (Panasonic Corporation)IIEEJ OfficeOsamu UKIGAYA
Rieko FUKUSHIMA
Kyoko HONDA

Contact Information
The Institute of Image Electronics Engineers of Japan (IIEEJ）
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
Tel : +81-3-5615-2893 Fax : +81-3-5615-2894
E-mail : hensyu@iieej.org
http://www.iieej.org/ (in Japanese)
http://www.iieej.org/en/ (in English)
http://www.facebook.com/IIEEJ (in Japanese)
http://www.facebook.com/IIEEJ.E (in English)

IIEEJ Transactions on
Image Electronics and Visual Computing
Vol.8 No.2 December 2020
CONTENTS

Special Issue on Extended Papers Presented in IEVC2019 Part Ⅱ
78

Upon the Special Issue on Extended Papers Presented in IEVC2019 Part Ⅱ

Naoki KOBAYASHI

Contributed Papers
79

A Screen Shake Determination Method Based on 2D Motion Histogram
Analyses by Using Group Transition and Maximum Group Ratio in Gaze
Areas

Mei KODAMA

91

Digital Contents for Creating and Watching 3DCG of Vehicles Based on
Drawing their Pictures

Shinji MIZUNO

System Development Paper
100

Cooperative E-learning Applications Based on HTML-5 Canvas for JapaneseClassical Literature Education

Eri YOKOYAMA, Hiroshi SUNAGA, Makoto J. HIRAYAMA

Special Issue on CG & Image Processing Technologies for Automation, Labor Saving and Empowerment
109

Upon the Special Issue on CG & Image Processing Technologies for
Automation, Labor Saving and Empowerment

110

Bidirectional Mapping Augmentation Algorithm for Synthetic Images
Based on Generative Adversarial Network

Masanori SEKINO

Contributed Paper
Haoqi GAO, Koichi OGAWARA

Regular Section
Contributed Paper
121

Robust Sphere Detection in Unorganized 3D Point Clouds Using an Efficient
Hough Voting Scheme Based on Sliding Voxels

Jaime SANDOVAL, Kazuma UENISHI, Munetoshi IWAKIRI,
Kiyoshi TANAKA

Announcements
136

Call for Papers: Special Issue on Image-related Technology for Realizing Immersive Media

137

Call for Papers: Special Issue on CG & Image Processing Technologies Supporting and Expanding Human Creativities

138

Call for Papers: IEVC2021 Shiretoko (Shari), Hokkaido / Sept. 8-11, 2021

Guide for Authors
140

Guidance for Paper Submission

Published two times a year by the Institute of Image Electronics Engineers of Japan （IIEEJ）
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan
Tel：+81-5615-2893 Fax：+81-5615-2894 E-mail：hensyu@iieej.org http://www.iieej.org/

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

Upon the Special Issue on
Extended Papers Presented in IEVC2019 PartⅡ
Editor: Prof. Naoki KOBAYASHI
Saitama Medical University
The Institute of Image Electronics Engineering of Japan (IIEEJ) regularly holds
International academic events named “Image Electronics and Visual Computing (IEVC)”
since 2007, on every two or two and half years. The 6th International Conference on Image
Electronics and Visual Computing (IEVC2019) was held in Bali, Indonesia on August 21-24,
2019．From this time, the name of this event was changed from International Workshop to
International Conference, to promote the event more worldwide and more attractive forspeakers and attendees. The conference was successfully held with 109 presentations and 165
participants (including 39 foreigners from more than 10 countries).
There were two paper categories in IEVC2019: general paper and late breaking paper(LBP), and in general paper, there were two tracks: Journal track (JT) and Conference track(CT). In IEVC2019, 33 JT papers, 52 CT papers and 24 LBP were submitted.
Journal track is a newly introduced one and has the advantage to be able to publish thepaper on the journal (IIEEJ Trans. on IEVC) in the “Special Issue on Journal Track in
IEVC2019” on December 2019 issue, by submitting full paper version (8 pages) together withconference paper version to be peer-reviewed in advance. Actually seven papers wereadopted in the “Special Issue on Journal Track in IEVC2019” published on December 2019.
The special issue on “Extended Papers Presented in IEVC2019” to be published on June
2020 was openly called for all paper categories in IEVC2019, and five papers, including theJT papers not to be in time for the publication schedule of December 2019 issue, wereadopted in June 2020 issue. In this Part Ⅱ of the “Extended Papers Presented in IEVC2019”,
two general papers and one system development paper have passed the review process to bein time for the publishing schedule.
Finally, I would like to give great thanks to all the reviewers and editors for their timeand efforts towards improving the quality of papers. I would also like to express my deepest
appreciation to the editorial committee members of IIEEJ and the staff at IIEEJ office forvarious kinds of support.

78

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
IIEEJ

Paper

-- Special Issue on Extended Papers Presented in IEVC2019 Part Ⅱ --A Screen Shake Determination Method Based on 2D Motion Histogram Analyses
by Using Group Transition and Maximum Group Ratio in Gaze Areas
Mei KODAMA† (Fellow)† Hiroshima University<Summary> If videos have screen shake information, it is one of the important issues to prevent viewers
from VIMS (visually induced motion sickness). So far there are two major approaches to prevent them. First
approach is a visually sickness information extraction method by using bio-metric information, and since it
is necessary to extract it after the physical condition becomes poor, the processing delay is inevitable. On
the other hand, second approach is a motion information extraction method by using image processing invideos. However, it is reported that the processing time becomes longer when the detailed motion such as
global motion estimation is analyzed. Thus, a screen shake determination method, which had used the blockmatching method as a simple motion analysis, motion direction histograms, and this similarity, had been
proposed. However, there is still the problem that the accuracy of detecting screen shake decreases, when
the amount of screen shake is small in the conventional method. It cannot extract the direction informationof screen shake. To solve the problems, this paper proposes a novel screen shake determination method
based on 2D motion histogram analyses. In particular, there are three features: the use of gaze areas, thegroup transition analysis of maximum frequency, and the maximum group ratio analysis in this method.
A new evaluation value Ev is deﬁned in consideration of both the accuracy of no-swing and pseudo swing
images. Simulation experiments show that the Ev in the proposed method is at most 4.02 smaller than thatin the conventional method for the small screen shake. Therefore, it is revealed that the proposed method
improves the accuracy of detecting the small screen shake in the conventional method and can extract thedirection information of screen shake. Furthermore, it is shown that it solves the problem of setting thethreshold of the histogram correlation. An adaptive method for each gaze area, and an adaptive method foreach number of directions and divisions in motion vector space will deserve for consideration, but they areleft for further studies.
Keywords: visually induced motion sickness (VIMS), screen shake, motion vector, histogram, group transition, group ratio1.

in advance. So far there are two major approaches toprevent them. The ﬁrst approach 6)–9) is a visually sickness information extraction method by using bio-metric
information directly, and since it is necessary to extract
it after the physical condition becomes poor, the process-Introduction

Nowadays, since the number of videos viewed on theInternet is increasing, and there are many users whoview them, there is the problem of video quality associated with visually induced motion sickness (hereinafter,ing delay is inevitable. On the other hand, the second
approach is a motion information extraction method byusing image processing in videos. However, it was reported that the processing time became longer when thedetailed motion such as a global motion estimation was
analyzed 15)–18). When obtaining the global motion vec-

VIMS) 1)–3) as a biological safety problem. In addition
to 2D images, 3D images, 360-degree images, and virtual space images, this problem had been studied 4)–16).
In particular, it is one of the important issues to prevent
viewers from VIMS. For example, if videos include screen
shake scenes taken by amateurs, it is necessary to takemeasures for viewing. Speciﬁcally, it is necessary to takemeasures such as displaying a subtitle of warning or ﬁltering the scene when screen shake scenes are detected,
or deleting the videos included the screen shake scenes

tors 17),18) related to the screen shake, spatial image information is needed, and the processing delay occurs. Even
if the high-speed calculating method of vector data areemployed, a huge amount of calculation is required, compared with the general motion calculation method such

79

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

determination. In other words, it can detect the scene
corresponding to screen shake by using the temporal cor-as the block matching. Thus, a screen shake determination method, which had used the block matching method

relation of the histograms. Then, he focused on general
block matching method for extracting screen shake, andas a simple motion analysis, motion direction histograms,
and these similarities, had been proposed 15),16). This basic idea using motion histograms is an extension of theprevious video identiﬁcation systems based on a motion
histogram analysis 19),20) and it supports the real-time

proposed a screen shake determination method to prevent
VIMS. It was a histogram-based approach using motion
vectors obtained by simple block matching method, which
was based on a ﬁxed block size. Since it also used theamount of the motion vectors which were separated fromprocessing. However, there are still the problem thatthe accuracy of detecting screen shake decreases and the
problem of how to set common parameters for various
sequences, when the amount of screen shake is small in
the conventional method. Then, to solve the problems,
this paper proposes a novel screen shake determination
method based on 2D motion histogram analyses. In particular, there are three features: the use of gaze areas,
the group transition analysis of maximum frequency, andthe maximum group ratio analysis in this method.
The previous studies are described in section 2, andthe details of the proposed method are shown in section
3. The determination accuracy is evaluated by comparing
the conventional method with the proposed method in thesimulation experiments and the results are considered in
section 4. Finally, this paper concludes in section 5.

2.

the motion direction histogram, the accuracy was higher
than that of a determination method with only the histogram correlation. However, there was a problem thatthe results were not output as continuous frames. Then,the time extension method, which detected a continuous
warning scene from warning candidate frames, was proposed, and it improved the accuracy of detecting VIMS.
However, there were still some basic problems that theaccuracy was reduced in the small screen shake, and thenormal movement of camera, e.g., pan or tilt, was falsely
detected as screen shake. For these undetected frames
or false detection, an improved time extension method,which had used the histogram correlation with direction
only and had considered the start and the end frames as
the screen shake scene simply, had been studied 21). To

Previous Study

simply improve the detection accuracy for VISM, there
were the previous determination methods 15),16) by using
Bhattacharyya distance 22) (hereinafter, BD) of the histograms with only the direction and the frequency change
for each direction. Then, among the previous methods,
this extension method with the highest accuracy is employed as the conventional method to make it easier toevaluate the accuracy of detecting screen shake under thesame conditions in this study. The principle of themis simply described, the procedure of the conventional
method is shown, and the problems are organized.

A motion extraction method with the global motion
compensation had been studied to analyze the detailed
camera motions 17),18). However, the real-time processing
is extremely diﬃcult in these methods due to a lot of motion parameters in camera operations, e.g., pan, tilt, andso on. On the other hand, general video processing, e.g.,video coding and motion analysis, generally uses a blockmatching method based on a parallel motion model. We
need a simple motion extraction of screen shake specialized for VIMS rather than a motion extraction with highaccuracy of detecting objects, and it is also required toadapt to the real-time processing.
By the way, a searching method with spatial information of a query video and a retrieval method with motion
vectors of a query video had been studied as a high-speed
video searching method

2.1

Principle of screen shake detection

The principle of screen shake detection had been proposed 15),16). In the moving pictures, which do not include scene change, since there is continuity of motion
between frames due to a moving subject or a movementof a camera as features of video scene, the histogram of
motion vectors in frame number m − 1 (fm−1 ) is generally similar to that in frame number m (fm ). On the19),20). Those retrieval methods
use the characteristics of the similarity of motion information. Then, considering the reduction of the processing
time, the author proposed a determination method withhistograms of motion vectors based on a block matching method as a general motion compensation processing 15),16). This method can make good use of the featureother hand, in the moving pictures, which include scenechange or screen shake information (e.g., hand shake, or
shake caused by photographing environment: in a vehicle, such as car, bus, and train, and in a stadium), it isof motion vectors based on the principle of the similarity

80

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）Video

togram in time domain is calculated. Histogram correlation with BD is employed.

Analysis parameters
number of directions, quantization
number, threshold values, etc.

Simple detection process of scene change
Calculating process of motion vectors
motion vectors, motion side information(a)(e) The screen shake is determined by using the thresholdvalue of each histogram correlation as the feature of(b)

Generating process of each motion histogram based on
motion group classified by direction and coding sideinformation for motion vectors in time domain(c)Calculating process of histogram correlation based oneach motion histogram in time domain(d)Screen shake determination process by using
the threshold value of each histogram correlation(e)

Screen shake determination process of caution frameby considering the start and the end frames(f)the similarity for each frame.
(f) Finally, the caution frame with screen shake is determined by using the start and the end points extracted as candidate results in the screen shake scene.
The number of frames between the start and the end
frames is determined by the threshold value T hf rame .
Thereafter, the processing of steps (c) to (f) is repeated for all input frames, and the ﬁnal results aresequentially obtained.

Final results

Fig. 1 Processing ﬂow of the conventional method byusing direction information

Here, the ﬁrst method consisting of steps (a) to (e) is
deﬁned as ‘CM1’ and the second one consisting of all steps
is deﬁned as ‘CM2’ as two types.

considered that the histogram in fm is not similar to thehistogram in fm−1 , when screen shake occurs due to external factors. Hence, the author focuses on the changeof the histograms in motion vectors in the conventional
method, and BD is employed as the degree of the similarity of the histogram.
2.2

2.3

Problems

The screen shake determination method uses thechange of the distribution of motion vectors obtained byeach frame. A large screen shake can be extracted in
the conventional method by using histograms of motion
directions only, whereas a small screen shake cannot be
extracted in the conventional method, which uses the determination method with BD and the frequency change
additionally 15),16). Since the amount of motion vectors

Outline of the conventional method

Figure 1 shows the outline of the conventional
method 21), and the procedure is shown as follows:(a) Simple detection process is applied in order to extracta scene change frame. For example, the time correlation value, or the diﬀerence of the luminance histogram between frames can be used.
(b) Motion vectors based on a block matching method are
calculated, macro block (MB) of vector detection unit
in a simple method is deﬁned as 16×16 pel and thematching search range is deﬁned as ± 32 pel. The
motion side information such as motion vectors andundetected motion is output.
(c) Each motion group histogram is generated for eachdirection and the motion type (coding side information) by using motion vectors for the whole screen intime domain. Here, the motion information group is
deﬁned as a group of direction (GD) when the motion
vector of which norm is greater than 0 can be calculated. As the other cases in a block matching method,
there are two states of motion estimation: ‘no motion’ information (NMI) and ‘not detected’ information (NDI). Two groups are deﬁned as a group of NMI

cannot be analyzed suﬃciently, the shake cannot be extracted in the conventional method 21). In all the previous
studies 15),16),21), since both the direction and the motion
amount are not used simultaneously, it cannot be found
how the scene is shaking. When the input images havethe object’s motion and the movement of the camera withscreen shake, there is also a problem that it is diﬃcult to
detect screen shake unless the amount of shake motion islarge. If the threshold value of BD is set small in order
to increase the determination sensitivity of BD, there isa problem that false detection increases, e.g., the object’s
motion may be falsely detected as screen shake. More importantly, although it is possible to detect the occurrenceof shaking, it is not able to detect in which direction thescreen is shaking in the previous studies 15),16),21). Then,
as described problems, the author focuses on the accuracy improvement in the case where the amount of screen
shake is small, or where screen shake is added to generalcamera movement. To solve the problems and to obtainthe screen shake direction, this paper ﬁrstly proposes anovel screen shake determination method by using boththe histogram of motion directions and the histogram of(GNMI) and a group of NDI (GNDI), respectively. A
motion histogram consists of GD, GNMI, and GNDI.
(d) Each histogram correlation based on each motion his-81

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

quantized amount of motion vectors 21). They are called(b)quantized histograms of both direction and size of motion
vectors (hereinafter, QHs) in this paper.

3.

(a)

Isy

Proposed Method

(c)
㼈㼈
㼈

fm-1

fm
Isx

For the problems described above, the author proposesa novel screen shake determination method by using 2D
motion histograms to improve the decrease of accuracy ofdetecting small screen shake in the conventional method.
In particular, the accuracy decreased in the conventional
method due to a general camera motion, e.g., pan, tilt.
As illustrated by Fig. 2, he considers that the motion
added to the object motion due to screen shake is extracted. Here, in Fig. 2, the bird area surrounded by thedotted black line indicates the object in fm−1 , the bird

Fig. 2 Relationship among the general motion, theshake motion, and the extracted motion due toscreen shake between fm−1 and fm
Area B

Rx

Rx
Ry

Area CIsy

Area E

Area B

Ox
Oy

Isy

Area C
Ry

Isx

Isx

(i) gaze area method 2 (am2) (ii) gaze area method 3 (am3)Fig. 3 Deﬁnition of two types of gaze area methods

area surrounded by the dotted purple line indicates theobject according to the general motion in fm , and thehistogram cannot be analyzed suﬃciently if the frequencyof the histogram is low or it is not given as an appropriate
number, as Sturages 23), Scott 24), Freedman et al. 25), andblack background area surrounded by the red or the lightgreen line indicates the object according to the generalmotion and screen shake in fm . Figure 2 (a) shows the

Shimazaki et al. 26) had studied in general histograms.
Each problem will be evaluated in the evaluation experiments described below.

object motion according to the general motion, however,
the object motion is extracted as the state of Fig. 2 (b) or
Fig. 2 (c) due to screen shake, where Isx and Isy denotethe horizontal and vertical screen sizes, respectively.
Then, he focuses on how the screen shaking information
appears on the motion histograms. Especially, he pays attention to the sensitivity of the histograms. This paper
proposes a screen shake determination method by using
QHs based on gaze areas in order to improve the sensitivity of detecting histogram change. Since QHs consistof 2D histograms for the direction and the size of motion,
this method can analyze not only the change of direction
but also the change amount. Therefore, he also proposesa motion analysis method using group transition due toscreen shake. So far only the motion direction was con-

3.1

Deﬁnition of gaze areas

This method ﬁrstly employs area information separated
by the edge area that does not include the object area
and the central area that includes it. The conventional
method targets the whole screen area (i.e., all areas ofMBs); however, the proposed method does each gazearea. In the conventional method, let ‘am1’ be the area
method that employs the whole screen and the area type(at) is given by ‘All’. When a screen shake occurs, thestate of the motion vector detection in each frame changes
according to the amount of shake. For example, when acamera shakes from left to right, the motion is extracted
as the reverse direction to the camera shake. It is diﬃcult to extract it in the edge area of a frame. Actually,
the motion information may change from the detectedmotion information to no motion compensation (no MC)
information as coding information, i.e., not detected motion information. Therefore, due to not detected motion information, the histogram analysis may not be pro-sidered in the previous histogram, and it was diﬃcult toobtain the direction information of screen shake. However, the proposed method uses the motion analysis withthe group transition to the adjacent group in QHs, and it
can extract the direction information. Additionally, gazeareas are used in order to increase the motion detection
sensitivity. However, since the frequency corresponding
to each bin in the histogram decreases if the gaze area becomes smaller or the division number of QHs increases,

cessed accurately. Then, he proposes two types of gazearea method as a screen segmentation method, considering screen areas where motion vectors are detected accurately. As illustrated by Fig. 3, they are deﬁned as ‘am2’and ‘am3’. In particular, gaze areas are deﬁned such as
an edge area and a central area considering an object
and a background as a simple ROI (Region Of Interest).

there is not only the problem that it becomes diﬃcult
to use the similarity information based on motion histogram but also the problem that the reliability of thehistogram analysis decreases. That’s because the motion

82

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
y
G2

G0

y
G0
G1
x

0

G3

G7

G4
G5

y
G0

G6
G2
G5
G3 G 1
x
0G
4

G9

G5
G6

G8

(i) DQ(4, 1)G4(ii) DQ(4, 2)

G3

G1
0

x

G8

G7

G9

(iii) DQ(8, 1)

vision areas based on each direction and size parameter,where direction parameter (Dn ) is set to 4 or 8, and sizey
G11 G10
G4 G 3 G2G
G1 G 9G13 50x
G6 G 7 G8
G14 G
G16

G0 G

G2

12

parameter (Qn ) is set to 1 or 2. Here, a combination of
QH-parameters is expressed as DQ(u, v), where u = Dn ,v = Qn . In Fig 4, x and y denote the amount of move-

15

G17

(iv) DQ(8, 2)

Fig. 4 Examples of division areas based on each direction and size parameterVideoments in the horizontal and vertical directions, respectively. The outer square shows the motion search area,
and the origin coordinate shown by a white dot meansa zero vector and the group of zero vectors is expressed
as G0 . Each division area shows each GD deﬁned in section 2. Here, the numbers are given in the order of rotation and small. Outside the square area indicated bygray color means undetected motion and the group is expressed as the ﬁnal group number, e.g., G5 , G9 , G17 . QHs

Analysis parameters

Number of directions, quantization
number, threshold values, etc.

Simple detection process of scene change
Calculating process of motion vectors
motion vectors, motion side information
Gaze area setting process

Area information,area type, etc.

(a)
(b)
(c)

Generating process of each motion histogram based on
motion group classified in each area quantizedby direction and size, and coding side information
for motion vectors in time domain(d)Calculating process of histogram correlation based oneach motion histogram in time domain(e)Screen shake determination process by using
the threshold value of each histogram correlation(f)Group transition analysis by usinggroup of maximum frequency(g)Group occupancy rate analysis by usinggroup of maximum frequency(h)Screen shake determination process by integrated
processing of group analysis results(i)

Screen shake determination process of caution frameby considering the start and the end frames(j)based on their groups are calculated.
3.3

The processing ﬂow of the proposed method is shownby Fig. 5. It consists of ten steps, and the details areshown as follows:(a) Simple detection process is applied in order to extracta scene change frame in the same manner as step (a)in the conventional method.
(b) Motion vectors based on a block matching method are
calculated in the same way.
(c) As shown in section 3.1, three types of gaze area methods are deﬁned as am1, am2, and am3. To utilize themotion features extracted in each area eﬀectively, eachtype can be adaptively used, considering the amountof motion; however, that is a further study. If the gazearea method is not used, am1 is used simply.
(d) Each motion group histogram is generated based on
motion group classiﬁed in the area quantized by direction and size of motion vectors according to each area

Final results

Fig. 5 Processing ﬂow in the proposed method

In am2, a background area (Area B) and a central area
(Area C) are deﬁned for the object in a frame. In am3,
an edge area (Area E), Area B, and Area C are deﬁned.
QHs for each area method are calculated. Here, Isx andIsy denote the horizontal and vertical screen sizes, respectively, Rx and Ry denote the width and the heightof Area B, respectively, and Ox and Oy denote the width

method in time domain. Examples of quantized areas
are illustrated by Fig. 4.

and the height of Area E, respectively.
3.2

The processing ﬂow

Deﬁnition of division areas for QHs(e) Each histogram correlation based on each motion histogram in time domain is calculated as BD.
(f) The screen shake is determined by using the thresholdvalue in the same way.
(g) The author focuses on the motion group transitionof maximum frequency in the calculated histogram.
Group transition patterns are prepared in advance.

In the conventional method, the direction informationis only employed. For example, there is a problem thatthe change of the histogram does not appear if the screen
shake is on the same axis as the motion direction of theobject , or if the amount of the object’s motion is large
and the amount of screen shake is small. Then, to beable to analyze the motion direction and the amount of

Two types are deﬁned: ptype1 denotes motion grouppatterns by using the opposite directions and ptype2
denotes adjacent motion group patterns. For example,
when the pattern is expressed by using a group number in Fig. 4 (i) with DQ(4, 1), ptype1: (1,3,1), (2,4,2),

motion at the same time and identify the direction of
screen shake, this method secondly employs QHs. The
direction and the size divisions in motion vector spacefor QHs are deﬁned. Figure 4 depicts examples of di-83

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
Table 1 Experimental conditions
Items
Test sequence

Details of contents
three types: s1(Track train),s2(Red leaves), s3(Drama set) in ITE test
sequences 27) , where s3 includes one scenechange frame between frame numbers 245
and 246.
Sequence typeno-swing image, pseudo swing image
Image size
504 × 280 pel
Number of frames
450 frame
Pseudo swing image frame numbers to which pseudo shaking
is added: 60–120, 180–240, 300–360
Swing angle typefour types ( θ= 0, π/4, π/2, 3/4π rad )(θ)( see Eq.(1) )Swing parameter (S) three types (S=16, 32, 64 pel ( see Eq.(1) )Gaze area method
am1(at: All), am2(at: A1-1, A1-2),(am)
am3 (at: A2-0, A2-1, A2-2)
Comparison
The conventional method: CM1, CM2
The proposed method:
PM1, PM2
methodfor each gtype1, gtype2
QH-parameters
Dn =4,8 and Qn =1,2
Other parameters(Isx , Isy ) = (504, 280) pel, (Rx , Ry ) =(32, 32) pel, (Ox , Oy ) = (64, 64) pel,T hmax1 = 10 %, T hf rame = 15 frame
Evaluation method accuracy rate, Ev(i) s1(ii) s2(iii) s3

Fig. 6 Examples of the test images

In particular, there is the major diﬀerence between theconventional method and the proposed method in thethree following points: the utilization of gaze areas, grouptransition analysis, and group ratio analysis of maximum
frequency. In the conventional method, it is necessary to
set a strict threshold because the shaking movement is notdirectly captured, whereas, in the proposed method, themovement can be directly detected by the group analysis
regardless of whether or not input images include generalmotion (e.g., a movement of an object and a backgroundmovement), if there is a shaking scene. In other words,
a histogram correlation analysis is used as a kind of preﬁlter in this method. Here, the ﬁrst method consisting ofsteps (a) to (i) is deﬁned as PM1, and the second method
consisting of all steps is deﬁned as PM2 as two types ofthe proposed method.

(3,1,3), (4,2,4), (1,0,1), etc., and ptype2: (1,2,1),(1,3,1), (1,4,1), etc. Here, in this study, group transition state is simply evaluated by two changes, however
the number of changes is a future issue. Additionally,
only the change of maximum group ratio T hrate =50%

4.

or greater is employed. The change from the current
group to the other group caused by the screen shakecan be analyzed, and consequently the shake directions can be obtained.
(h) To grasp the change of motion in the same motion
group, the group ratio of maximum frequency is analyzed by the threshold value T hmax1 . For the change
ratio greater than T hmax1 , a decision is made whether

Evaluation Experiments

To evaluate the accuracy of detecting screen shake inthe proposed method, the evaluation experiments are performed. Firstly, the experimental conditions of the conventional method and the proposed method in this study
are shown. Next, the results of the experiments and theconsideration are described.
4.1

the ratio increases or decreases. It is possible to judge
whether a group change has occurred due to a largechange in group ratio, and consequently this step canﬁnd the direction in which it is shaking.
(i) To compare with the conventional method ( i.e., only
step (e) ), the author considers four types of processing
methods for screen shake analysis: dm1: the process-Conditions of simulation experiments

The experimental conditions are shown in Table 1. As
shown in Table 1, two sequence types: no-swing images
and pseudo swing images in each sequence type are employed in the evaluation experiments. All these images
are generated by downsizing and trimming the original
images in ITE test sequences 27). Note that these se-

ing method of only step (g), dm2: that of only step(h), dm3: that of both steps (g) and (h), and dm4: theintegrated processing from steps (f) to (h). In thesegroup analysis methods, it is possible to select which
method is easy to detect the screen shake, which cannot be detected by the conventional method.
(j) Finally, the caution frame with screen shake is determined in the same manner as step (f) in the conventional method. Thereafter, the processing of steps (e)quences have not only the movement of the object but
also that of the camera. Figure 6 depicts examples of
the test images.
First, the pseudo swing images are generated under theconditions indicated by Table 1. The detailed conditions
are shown as follows:• A pseudo swing function is deﬁned by using a sinefunction to make it easier to evaluate the accuracy ofdetecting the screen shake. When the original coordinates in the original image are given by (xo , yo ) andthe displacement coordinates in the pseudo image due

to (j) is repeated for all input frames, and the ﬁnal
results are sequentially obtained.

84

SD [pel]IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

0

30

60

90

120

150

80
40
0
-40
-80

180

210

240

270

300

330

360

390

420

method with the proposed method, it is necessary to consider whether the result of the obtained accuracy is cor-

FN

450

(i) Sθ(16,0)
horizontal
vertical
0

30

60

90

120

150

180

210

240

270

300

330

360

390

420

rect. In brief, there is a trade-oﬀ relationship between
each accuracy of no-swing images and pseudo swing images. Therefore, an evaluation index is required to simultaneously indicate whether each accuracy is high or not.
Then, since the ideal goal is 100 % in accuracy rate, theEuclidean distance between 100 % and the obtained accuracy is also used as the performance evaluation. In particular, this paper focuses on 2D accuracy rate space between the accuracy of no-swing images and that of pseudo
swing images, and the Euclidean distance between the 2D
coordinates of (100,100) and the calculated 2D coordinates is used as the evaluation value. Let it be Ev, and it
is calculated by Eq. (2), where the accuracy rate obtainedin each no-swing image is given by Aro and that in each
pseudo swing image is given by Ars .

FN

450

(ii) Sθ(32,0)SD [pel]

80
40
0
-40
-80

of screen shake detection by comparing the conventional

horizontal
verticalSD [pel]

80
40
0
-40
-80

horizontal
vertical
0

30

60

90

120

150

180

210

240

270

300

330

360

390

420

FN

450

(iii) Sθ(64,0)

Fig. 7 Examples of the pseudo swing function

to screen shake are given by (xd , yd ), they are represented by Eq. (1).


xd
yd

=
=

xo + S sin(4πt) × cos θ
yo + S sin(4πt) × sin θ(1)Where S denotes the amplitude parameter, t =F N/30 (F N : frame number), and the shaking direction is represented by θ (0 ≤ θ < 2π rad). Here, thefrequency parameter is set to 2 under the condition
that the number of pixels of the position gap at theextreme value is 1 or greater in Eq. (1).
• Twelve pseudo swing images based on Eq. (1) are generated by using three amplitude parameters (S) and
four direction parameters (θ). The examples of thepseudo swing function are shown in Fig. 7, where SD
denotes the amplitude and F N denotes the frame
number. In this study, pseudo shake information is
not added to the whole sequence, and it is partiallyadded to three places at every 60 frames. Here, a combination of swing parameters is expressed as Sθ(p, q),
where p = S, q = θ.
Next, three area types of gaze area methods are deﬁned by am1–am3. All(am1), A2-0(am3) for Area E, A11(am2) and A2-1(am3) for Area B, and A1-2(am2) andA2-2(am3) for Area C are used as area types(at). Twotypes of gtype1 and gtype2 are deﬁned as the group anal-
Ev= (100 − Aro )2 + (100 − Ars )2

Therefore, it can be evaluated that the accuracy of detecting screen shake is improved if the value of Ev becomes smaller.
4.3

Simulation results and consideration

In the conventional method, there is a trade-oﬀ problem in the accuracy of detecting the screen shake. The
movement of the object or the background is considered
as general motion information. For example, when thescreen shake is simultaneously detected for screen shake
images and no screen shake images with and without thegeneral movement, it is extremely diﬃcult to ﬁnd an optimum threshold in the determination method by using a
BD threshold. That’s because the threshold value should
be set large in the conventional method not to detect thegeneral movement except for screen shake. That is to say,the higher the sensitivity, the greater the false detection
rate, whereas the lower the sensitivity, the lower the rate.
In the conventional method, this problem is a trade-oﬀ
problem, and it leads to the problem of the reduction ofysis method in the proposed method. Gtype1 denotes
the group transition analysis in the maximum group,and gtype2 denotes that in the maximum group excluding GNMI. Note that, strictly speaking, the conventional
method is used only based on am1 with DQ(4, 1) andDQ(8, 1). However, to evaluate the gaze area methods
and QH-parameters, the results in DQ(4, 2) and DQ(8, 2)
are included in the results of the conventional method,
and the results of am2 and am3 are done. Other param-

detection accuracy. However, the proposed method does
not depend on the threshold setting for BD. Then, heevaluates whether the proposed method can improve thetrade-oﬀ problem by the simulation experiments.

eters and the threshold values are shown in Table 1.
4.2

(2)

(0 ≤ Aro ≤ 100, 0 ≤ Ars ≤ 100)Deﬁnition of evaluation value(a) Detection state by group transition and maximum rate analysis

Each accuracy rate is calculated by using no-swing andpseudo swing images in the conventional method and the
proposed method. However, to evaluate the accuracy

The details of the detection state, i.e., the changesof the group percentage of maximum frequency and the85

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
Group
percentage
of maximum
frequency[%]

100

50
0

All

Group change point

Maximum change point: |v|>Thmax1

G5
G2
G1
G0
G3
G4 All0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450Groupof maximum
frequency

Group

Group
percentage
of maximum of maximum
frequency frequency[%]100

Frame number(i) s1Group

Group
percentage
of maximum of maximum
frequency frequency[%]

100
50
0

All

Group change point

Maximum change point: |v|>Thmax1

G5
G2
G1
G0
G3
G4 All0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450

50
0

All

Group change point

Maximum change point: |v|>Thmax1

G17
G12
G11
G10
G9
G4
G3
G2
G1
G0
G5
G6
G7
G8
G13
G14
G15
G16 All
0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450

Frame number

Fig. 10 An example of each group change in pseudo
swing images ( s1 in Sθ(16, 0) and am1 with
DQ(8,2) )Frame number(ii) s2

Maximum change point: |v|>Thmax1

Frame number(iii) s3

Fig. 8 Examples of each group change in no-swing images ( am1 with DQ(4,1) )All

Group change point

Maximum change point: |v|>Thmax1

Group change pointGroupof maximum
frequency

Group

G9
G6
G5
G2
G1
G0
G3
G4
G7
G8

50

All

Maximum change point: |v|>Thmax1

G5
G2
G1
G0
G3
G4 All0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 45050Group change point

Maximum change point: |v|>Thmax4Group change point

Maximum change point: |v|>Thmax5Group change point

Maximum change point: |v|>Thmax6

A2-0

0

A2-1

A2-1

50
0

A2-2

G9
G6
G5
G2
G1
G0
G3
G4
G7
G8 A2-2
0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450

Fig. 11 An example of each group change in pseudo
swing images ( s1 in Sθ(16, 0) and am3 with
DQ(4,2) )

100

All

Group change point

Frame number

Frame number(ii) s2

0

A2-0

100

100

0

0

50

Group
percentage
of maximum
frequency[%]

Group

Group
percentage
of maximum of maximum
frequency frequency[%]

0

G5
G2
G1
G0
G3
G4 All0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450

Frame numberGroup
percentage
of maximum of maximum
frequency frequency[%]

G9
G6
G5
G2
G1
G0
G3
G4
G7
G8

50

(i) s1

Group

50

100

100

Group
percentage
of maximum of maximum
frequency frequency[%]Grouppercentage
of maximum
frequency[%]Group change point

G5
G2
G1
G0
G3
SC
G4 All0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450

Grouppercentage
of maximum
frequency[%]

0

All

Group
of maximum
frequency

100

50

Group
of maximum
frequency

Group

Group
percentage
of maximum of maximum
frequency frequency[%]

100

Maximum change point: |v|>Thmax1

G5
G2
G1
G0
G3
SC
G4 All0 30 60 90 120 150 180 210 240 270 300 330 360 390 420 450

low area shows the true shaking images in Fig. 8 andFig. 9. Note that s3 sequence has a scene change andit is represented by ‘SC’. in Fig. 8 and Fig. 9. Figure 10
and Fig. 11 depict the examples of each group change in
am1 with DQ(8,2) and am3 with DQ(4,2), respectively.
As shown in Fig. 8, the object movement in frame or
the movement of camera appears as group transition andFrame number(iii) s3

Fig. 9 Examples of each group change in pseudo swing
images ( Sθ(16, 0) and am1 with DQ(4,1) )group of maximum frequency, in no-swing and pseudo
swing images are shown in Fig. 8 and Fig. 9, respectively.
Each result is obtained by the proposed method of am1
with DQ(4,1). Each orange line indicates the frame ofgroup transition (group change point), each violet line
indicates the frame with the variation of the maximum
value ≥ T hmax1 (maximum change point), and each yel-maximum value change in the no-swing images for eachsequence. On the other hand, Fig. 9 shows that the movement caused by shaking occurs in the pseudo swing images. In particular, it can be found that the number ofgroup changes increases when screen shake occurs. However, as shown in Fig. 9 (i), when the amplitude of swing

86IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）(a) s1

method can detect some of them. In particular, the im-(a) CM1,All(b) CM2,All(c) PM1,All(d) PM2,All0(b) s2(c) s3

provement is remarkable in s1 and s3 sequences, How-30 60 90 120 150 180 210 240 270 300 330 360 390 420 450
Frame number

ever, there are still some frames that cannot be detected(a) CM1,All(b) CM2,All(c) PM1,All(d) PM2,All

030 60 90 120 150 180 210 240 270 300 330 360 390 420 450
Frame number

030 60 90 120 150 180 210 240 270 300 330 360 390 420 450
SC
Frame numberin s2 sequence despite the screen shake frames even if the
proposed method is used. One of the reasons is that an
appropriate vector cannot be estimated for the movementof the background tree because the camera is moving diagonally.

(a) CM1,All(b) CM2,All(c) PM1,All(d) PM2,All(i) no-swing images(a) s1(a) CM1,All(b) CM2,All(c) PM1,All(d) PM2,All0(b) s2(a) CM1,All(b) CM2,All(c) PM1,All(d) PM2,All(c) s3(a) CM1,All(b) CM2,All(c) PM1,All(d) PM2,All(b) Evaluation of accuracy improvement in Ev

30 60 90 120 150 180 210 240 270 300 330 360 390 420 450
Frame number

030 60 90 120 150 180 210 240 270 300 330 360 390 420 450
Frame number

030 60 90 120 150 180 210 240 270 300 330 360 390 420 450
SC
Frame number

In the conventional method and the proposed method,each accuracy of detecting screen shake is analyzed indetail by using an Ev as an evaluation value.
Firstly, the comparison results of each Ev in eachmethod using am1 for each threshold of BD are shownin Table 2. In Table 2, ‘Ave’ denotes the average valueof Ev in each sequence, and this study discusses the effectiveness based on the average value in the conventional
method and the proposed method. Note that each PM2
with gtype1 and gtype2 is employed; only the results of
CM2 and PM2 are shown since they are basically more effective than CM1 and PM1, respectively; only the resultsof am1 are shown; the results of dm1–dm3 are omitted
since dm4 is equal to or greater than them in the accuracyof all the simulation experiments.
As shown in Table 2, the highest accuracy value is 4.70
in CM2, and the highest accuracy values are 2.84 and 2.75
in PM2 with gtype1 and gtype2, respectively for Dn = 4.

(ii) pseudo swing images

Fig. 12 Comparison of screen shake determination results ( pseudo swing images in Sθ(16, 0) and
am1 with DQ(4,1) )is small, the group transition does not appear. Whereas,
as shown in Fig. 10 and Fig. 11, even if the amplitude issmall, the group transition appears in am1 with DQ(8,2)
and in am3 with DQ(4,2). Next, it is required to extract
screen shake in the same group when the group transition
does not occur. Then, to solve the problem, the proposed
method focuses on the change of the group ratio in thesame group of maximum frequency. By Fig. 9–Fig. 11,
the change can be found. However, the group transitions
don’t appear when the number of division areas is small
such as DQ(4,1), whereas, since the group transitions appear if the large number of division areas are used suchas DQ(4,2) and DQ(8,2), a screen shake scene can beextracted. Am2 and am3 are more eﬀective than am1 in
s1 sequence as gaze area methods in this experiment. By
the results, there is no problem of the frequency of QHs
in each gaze area method.
In practice, the detection results are evaluated in theconventional method and the proposed method. Figure 12 shows the comparison of the screen shake determination results in no-swing and pseudo swing images forConsequently, it is improved by up to 1.95 in PM2. It
is found that the quantization extension method is more
eﬀective in each method using CM2, and gtype2, which
deals with the maximum group except for GNDI, is more
eﬀective than gtype1 in the proposed method using PM2.
However, the average of Ev in Dn =4 is more superior

each test sequence in each method. Note that the threshold value is set to 0.35 in a BD determination method,

than that in Dn =8. That is because if the threshold value
is set large, the change in the histogram is less likely toappear in the determination method using only BD, andat the same time, the accuracy deteriorates due to thetrade-oﬀ problem between no-swing images and pseudo
swing images. It is necessary to consider the parameter setting of Dn and Qn in the determination method

and the proposed method with gtype1 is employed. Since
all no-swing images don’t have screen shake, it should not
be detected. However, it is falsely detected in CM1 andPM1 in s2 sequence, whereas the detection errors are improved in CM2 and PM2. As shown in Fig. 12, there areundetected frames of screen shake in pseudo swing images for the conventional method; however, the proposedusing BD and motion analysis separately, and it is a future issue. On the other hand, although not listed inTable 2, by the results of Ev in A2-2 of am3, the accuracy in PM2 with gtype2 is 4.01, and it is 1.13 better
than the accuracy of 5.14 in CM2. In particular, PM2with gtype2 is more eﬀective than that with gtype1. As
shown in Table 2, the BD threshold value corresponding

87

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

Table 2 Comparison results of each Ev in the conventional method and the proposed method using am1 ( Sθ(16, 0 ) )
Th Q
n
1
0.20

2
1

0.25

2
1

0.30

2
1

0.35

2
1

0.40

2
1

0.45

2
1

0.50

2

CM2
s1
s2
s3
3.12
4.60 11.22
Ave: 6.32
2.90
4.60 11.40
Ave: 6.30
6.03
2.28 7.37
Ave: 5.22
5.80
2.28 6.03
Ave: 4.70
27.45
2.28 7.37
Ave: 12.37
11.38
2.28 7.37
Ave: 7.01
28.12
6.25 7.37
Ave: 13.91
28.12
2.01 7.37
Ave: 12.50
36.61
9.60 7.37
Ave: 17.86
34.60
4.91 7.37
Ave: 15.62
38.84 10.27 7.37
Ave: 18.82
38.84
9.60 7.37
Ave: 18.60
40.18 17.19 8.71
Ave: 22.02
40.18 17.19 7.37
Ave: 21.58

Dn = 4
PM2 with gtype1
s1
s2
s3
1.56 6.16 9.19
Ave: 5.64
0.67 6.16 13.26
Ave: 6.70
3.12 4.60 6.51
Ave: 4.74
2.90 4.60 8.05
Ave: 5.18
4.02 2.28 2.23
Ave: 2.84
5.80 2.28 2.23
Ave: 3.44
4.02 2.28 2.23
Ave: 2.84
5.80 2.28 2.23
Ave: 3.44
4.46 6.25 2.23
Ave: 4.32
11.38 2.01 2.23
Ave: 5.21
5.36 9.60 2.23
Ave: 5.73
12.05 5.13 2.23
Ave: 6.47
5.80 10.27 2.23
Ave: 6.10
13.84 9.82 2.23
Ave: 8.63

PM2 with gtype2
s1
s2
s3
1.56 6.16 9.19
Ave: 5.64
0.67 6.31 13.26
Ave: 6.75
3.12 4.60 6.51
Ave: 4.74
2.90 4.81 8.05
Ave: 5.25
4.02 2.28 2.23
Ave: 2.84
5.80 2.50 2.23
Ave: 3.51
4.02 2.28 2.23
Ave: 2.84
5.80 2.50 2.23
Ave: 3.51
4.46 2.01 2.23
Ave: 2.90
11.38 2.23 2.23
Ave: 5.28
5.36 0.67 2.23
Ave: 2.75
12.05 5.36 2.23
Ave: 6.55
5.80 0.67 2.23
Ave: 2.90
13.84 7.81 2.23
Ave: 7.96

CM2
s1
s2
s3
3.12 22.59 11.52
Ave: 12.41
0.67 22.59 14.74
Ave: 12.67
4.91 14.99 8.55
Ave: 9.49
0.67 14.99 10.59
Ave: 8.75
26.12 14.99 7.37
Ave: 16.16
10.49 14.99 7.37
Ave: 10.95
27.90 4.33 7.37
Ave: 13.20
27.90 4.33 7.37
Ave: 13.20
27.90 3.04 7.37
Ave: 12.77
27.90 3.04 7.37
Ave: 12.77
35.49 3.04 7.37
Ave: 15.30
35.49 3.04 7.37
Ave: 15.30
40.18 7.59 7.37
Ave: 18.38
40.18 7.59 7.37
Ave: 18.38

Dn = 8PM2 with gtype1
s1
s2
s3
0.89 23.58 17.17
Ave: 13.88
0.89 23.58 28.57
Ave: 17.68
0.89 22.59 11.73
Ave: 11.74
0.89 22.59 15.85
Ave: 13.11
0.89 14.99 8.55
Ave: 8.15
0.89 14.99 12.88
Ave: 9.59
1.56 14.99 7.37
Ave: 7.97
5.80 14.99 13.48
Ave: 11.43
1.56 10.98 7.37
Ave: 6.64
11.38 10.98 9.39
Ave: 10.58
1.56 9.96 7.37
Ave: 6.29
11.38 9.96 9.39
Ave: 10.24
2.01 9.96 7.37
Ave: 6.44
12.05 9.96 9.39
Ave: 10.47

PM2 with gtype2
s1
s2
s3
0.89 23.73 17.17
Ave: 13.93
0.89 23.73 28.57
Ave: 17.73
0.89 22.74 11.73
Ave: 11.79
0.89 22.74 15.85
Ave: 13.16
0.89 15.15 8.55
Ave: 8.20
0.89 15.15 12.88
Ave: 9.64
1.56 15.15 7.37
Ave: 8.03
5.80 15.15 13.48
Ave: 11.48
1.56 11.16 7.37
Ave: 6.70
11.38 11.16 9.39
Ave: 10.64
1.56 10.12 7.37
Ave: 6.35
11.38 10.12 9.39
Ave: 10.30
2.01 10.12 7.37
Ave: 6.50
12.05 10.12 9.39
Ave: 10.52

Table 3 Results of each Ev at the highest accuracy in each method and comparison results of an Ev gain between CM2and each proposed method
Test Seq.
s1
s2
s3
am
(S, θ)CM1 CM2 PM1 PM2 Ar1 Ar2 CM1 CM2 PM1 PM2 Ar1 Ar2 CM1 CM2 PM1 PM2 Ar1 Ar2(16,0)
1 21.93 0.67 2.01 0.67 -1.34 0.00 28.16 2.01 11.84 0.67 -9.83 1.34 33.04 6.03 3.80 2.01 2.23 4.02
(16,0)
2 22.70 1.56 15.19 1.34 -13.63 0.22 28.48 2.01 10.27 0.67 -8.26 1.34 32.59 5.80 7.59 5.58 -1.79 0.22
(16,0)
3 21.06 0.67 8.93 0.67 -8.26 0.00 28.70 2.01 6.92 0.67 -4.91 1.34 32.40 5.58 7.39 5.36 -1.81 0.22
(32,0)
1 17.04 0.67 8.26 0.67 -7.59 0.00 27.25 2.01 11.62 0.67 -9.61 1.34 33.93 1.56 2.90 1.34 -1.34 0.22
(32,0)
2 16.51 0.67 3.80 0.67 -3.13 0.00 26.61 1.56 6.47 0.89 -4.91 0.67 32.81 1.56 2.90 1.34 -1.34 0.22
(32,0)
3 18.18 0.67 2.68 0.67 -2.01 0.00 26.02 0.67 3.35 0.67 -2.68 0.00 32.37 0.67 2.90 0.67 -2.23 0.00

to the highest accuracy value in each PM2 is larger thanthe value in each CM2; therefore, the proposed method
has the advantage that the accuracy is not easily aﬀectedby the threshold settings of BD, e.g., it is able to be usedin T h of around 0.30–0.45 and Dn =4. Furthermore, theproposed method is superior to the conventional method.
As shown in Table 3, in Sθ(16,0), Ar2 is 0.22 in am2
in s1 sequence, Ar2 is 1.34 in am1, am2, and am3 in s2
sequence, Ar1 is 2.23 in am1 in s3 sequence, and Ar2
is 4.02 in am1 and Ar2 is 0.22 in am2 and am4 in s3
sequence. In Sθ(32,0), Ar2 is 1.34 and 0.67 in am1

gaze area method in PM2 with DQ(4, 2) is more eﬀective than CM1 with am1 and Dn =4. However, there isand am2 in s2 sequence, respectively, and Ar2 is 0.22
in am1 and am2 in s3 sequence. Additionally, Ar2 is

a problem of the parameter setting in both gaze area parameters and QH-parameters. In this experiment, theaccuracy is higher when Qn is set large than when Dn

0.22 in am2 ( Sθ(32,π/2) ) and am3 ( Sθ(64,π/2) ). On
the other hand, he considers the case where the proposed
method is inferior. Ar2 is -0.09 in am1 ( Sθ(16,π/4) ),Ar2 is -1.34 in am2 ( Sθ(32,π/4) ), and Ar2 is -0.89 in

is set large. This evaluation is related to the accuracy of
motion vectors for each sequence, and it is a future study.

am1 ( Sθ(16,3π/4) ). In the other cases, Ar2 is 0 and theSecondly, Table 3 shows the results of each Ev at
the highest accuracy in each method and the comparison results of an Ev gain between CM2 and each proposed method, where Ar1 denotes the gain between CM2
and PM1, and Ar2 denotes the gain between CM2 andaccuracy in CM2 is the same as that in PM2. Of course,
since PM1 does not include the time extension processing(see (j) in section 3), PM1 is inferior to CM2. However, it
is greatly improved over CM1 in all cases. Consequently,
if all gaze area methods can be adaptively employed, theproposed method is superior to the conventional method
in all cases. In particular, it is revealed that the pro-

PM2. Here, the results in only the cases of Sθ(16,0) andSθ(32,0) are shown in Table 3. In the experimental results, the author pays attention to the case where the88IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
Table 4 Results of each Ev at the highest accuracy inthe proposed method using dm3

further study, there is an adaptive method in each gazearea parameter and each QH-parameter.

area type
S
16
32
64

am1
All
8.41
12.05
7.07

am2
A1-1 A1-2
14.29 14.17
7.66 11.24
9.00
9.89

A2-0
14.57
6.18
8.33

am3
A2-1
13.37
10.71
9.35

5.

A2-2
11.72
10.14
8.95

Conclusions

If videos have screen shake information, it is one of
the important issues to prevent viewers from VIMS. A
screen shake determination method, which had used theblock matching method as a simple motion analysis, motion direction histograms, and this similarity, had been
proposed. However, there was still the problem that theaccuracy of detecting screen shake decreases, when theamount of screen shake was small. The conventional
method could not extract the direction information of
screen shake. To solve the problems, this paper proposed a novel screen shake determination method basedon 2D motion histogram analyses. In particular, there
were three features: the use of gaze areas, the analy-

posed method is better than the conventional method if
the screen shake is small, e.g., S = 16. However, there isa future issue in the improvement of accuracy reductionin the shaking direction (θ=π/4), it is necessary to study
an adaptive gaze area method, and a theoretical analysis
is required for the relationship between the shaking direction and the moving direction of the object to accurately
grasp the performance limit of the proposed method.
Finally, he considers the results of motion detectionof screen shake in the proposed method. In the conventional method, a determination method using a BD
analysis could ﬁnd whether the screen was shaking; however, it could not detect in which direction the screen
was shaking. On the other hand, as shown in Fig. 8–11, since the proposed method uses the group transition
and the maximum group ratio, it is a crucial point tobe able to extract a scene of screen shake corresponding
to simple shaking information deﬁned as swing patterns.
To evaluate the eﬀectiveness of the group analysis in the
proposed method, he focuses on dm3 corresponding to
only the group analyses, and the Ev by using only dm3
for each gaze area method is calculated. The results ofeach Ev at the highest accuracy in the proposed method
using dm3 are shown in Table 4, where θ = 0. As shownin Table 4, the values of the highest accuracy are 8.41 in
S = 16, 6.18 in S = 32, and 7.07 in S = 64. It is foundthat the screen shake can be detected only by the group
analyses, and the motion direction of the screen shake canbe obtained according to the accuracy. The accuracy cansis of group transition of maximum frequency, and thegroup ratio analysis of maximum frequency group in the
proposed method. A new evaluation value Ev was deﬁned in consideration of both the accuracy of no-swing
images and pseudo swing images. Simulation experiments showed that the Ev in the proposed method was at
most 4.02 smaller than that in the conventional method( Sθ(16, 0) ). Furthermore, the direction of the screen
shake could be extracted by the proposed method. Therefore, it was revealed that the proposed method not only
improved the accuracy of detecting the small screen shakefor the conventional method, but also it had the advantage of extracting the direction of the screen shake.
As further studies, an adaptive method for each gazearea parameter and each QH-parameter and a theoreticalevaluation will be required.

References
1) S. Chiba: “Health Hazard by Video Image”, Journal of ITE,
Vol. 56, No. 6, pp. 926–927 (2002). (in Japanese)be further improved by using the gaze area method, e.g.,
am1 is eﬀective in S=16 and am3 is eﬀective in S=32. An

2) N. Sugita, M. Yoshizawa, A. Tanaka, K. Abe, T. Yambe, S.
Nitta, S. Chiba: “Evaluation of the Eﬀect of Visually-Induced
Motion Sickness Based on Pulse Transmission Time and HeartRate”, Proc. of SICE 2004 Annual Conference, Vol. 3, pp.
2473–2476 (2004).

adaptive gaze area method using the frequency distribution can be also considered. Hence, since the proposed
method can detect the occurrdance of screen shake andits direction at the same time, it is more eﬀective thanthe conventional method. Furthermore, it is importantto accurately grasp the performance limit of the proposed
method, and a theoretical evaluation is a future task.
As mentioned above, it is revealed that the proposed
method by using QHs and a gaze area method is more
eﬀective than the conventional method. However, as a3) I. Tsubaki, T. Morita, K. Aizawa, T. Saito: “The Analysis ofOscillatory Motion of Videos Aﬀected by Camera Shaking for
Visually Induced Motion Sickness”, IEICE A, Vol. 89, No. 3,
pp. 262–267 (2006).
4) M. Abe, M. Yoshizawa, N. Sugita, A. Tanaka, S. Chiba, T.
Yambe, S. Nitta: “Estimation of Eﬀects of Visually-Induced
Motion Sickness Using Independent Component Analysis”,
Proc. of 2007 29th Annual International Conference of theIEEE Engineering in Medicine and Biology Society (2007).
5) H. Watanabe, H. Ujike: “The Activity of ISO/Study Group on

89

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）“Image Safety” and Three Biological Eﬀect”, Proc. of Second
International Symposium on Universal Communication (ISUC’08), pp. 210–214 (2008).

tiﬁcation Method Using Histogram Information of Coded Motion Vectors”, The Journal of IIEEJ, Vol.43, No.4, pp.499–507(2014). (In Japanese)

6) A. Tanaka, N. Sugita, M. Yoshizawa, M. Abe, T. Yambe: “Dynamic Characteristics between the Subjective Score of Motion
Sickness Discomfort and Video Global Motion”, Proc. of 2010
Annual International Conference of the IEEE Engineering in
Medicine and Biology, pp. 1368–1369 (2010).

20) M. Kodama: “A Video Content Identiﬁcation Method Using Moving Regions Information by Histogram of Motion Vectors”, The Journal of IIEEJ, Vol.42, No.5, 625–632 (2013). (InJapanese)21) M. Kodama: “A Consideration on Accuacy Improvementof Screen Shake Determination Methods Using Motion Histograms”, Proc. of IIEEJ International Conference on Image Electronics and Visual Computing 2019 (IEVC2019), 1P-9
(2019).

7) H. Ujike: “Developing an Evaluation System of Visually Induced Motion Sickness for Safe Usage of Moving Images”, Synthesiology English Edition, Vol. 5, No. 3, pp. 139–149 (2012).
8) T. Kiryu, A. Iijima: “A Multi-Timescale Autonomic Regulation Model for Interpreting Visually Induced Motion Sickness”,
Proc. of 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), pp. 254–257 (2014).

22) A. Bhattacharyya: “On a Measure of Divergence between Two
Statistical Populations Deﬁned by their Probability Distributions”, Bulletin of the Calcutta Mathematical Society, Vol.35,
pp.99–109 (1943).

9) N. Kobayashi, R. Iinuma, Y. Suzuki, T. Shimada, M. Ishikawa:“Using Bio-Signals to Evaluate Multi Discomfort in Image
Viewing - Balancing Visually Induced Motion Sickness andField of View -”, Proc. of 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6198–6201 (2015).

23) H.A. Sturges: “The Choice of a Class Interval”, Journal of theAmerican Statistical Association, Vol. 21, No. 153, pp.65–66(1926).
24) D.W. Scott: “On Optimal and Data-Based Histograms”,
Biometrika, Vol. 66, Issue 3, pp.605–610 (1979).

10) P. Bala, D. Dionı́si, V. Nisi, N. Nunes: “Visually Induced Motion Sickness in 360◦ Videos: Comparing and Combining Visual Optimization Techniques”, Proc. of 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct(ISMAR-Adjunct) (2018).

25) D. Freedman, P. Diaconis: “On the Histogram as a Density Estimator: L2 Theory”, Zeitchrit für Wahrscheinlichkeitsheorie
und Verwandte Gebiete, Vol.57, No. 4, pp.453–476 (1981).
26) H. Shimazaki, S. Shinomoto: “A Mthod for Selecting the Bin
Size of a Time Histogram”, Neural Computation, Vol.19, No.
6, pp.1503–1527 (2007).

11) S. Wibirama, K. Hamamoto: “Investigation of Visually Induced Motion Sickness in Dynamic 3D Contents Based on Subjective Judgment, Heart Rate Variability, and Depth Gaze Behavior”, Proc. of 36th Annual International Conference of theIEEE Engineering in Medicine and Biology Society, pp. 4803–
4806 (2014).

27) ITE/ARIB HDTV Test Materials –Second Edition, ITE, https://www.ite.or.jp/contents/chart/ippan en.html (2020).

(Received June 11, 2020)(Revised October 12, 2020)

12) S. Wibirama, T. Wijayanto, H.A. Nugroho, M. Bahit, M.N.
Winadi: “Quantifying Visual Attention and Visually Induced
Motion Sickness During Day-Night Driving and Sleep Deprivation”, Proc. of 2015 International Conference on Data andSoftware Engineering (ICoDSE) (2015).

Mei
KODAMA (Fellow)
He received the B.E, M.E, and Ph.D degrees,
all in Engineering from Waseda University,
Tokyo, Japan, in 1992, 1994 and 1997, respectively. He was the Research Associateof Waseda University from 1995 to 1998. He
joined in Hiroshima University from 1998 to2001 as the Associate Professor at Center forTechnology Research and Development of Hiroshima University. He is currently working as the Associate Professor at Graduate
School of Advanced Science and Engineering, School of Integrated Arts and Sciences,and Information Media Center of Hiroshima
University. His research interests include
audio-visual communication, especially, scalable video coding, image processing, video
content distribution system and video communication services. He received Best Author Award of the Journal of ITE from ITE in
2013, Best Journal Paper Award from IIEEJin 2018, and Excellent Journal Paper Award
from IIEEJ in 2014 and 2020. He has been an
editor in chief of Journal of IIEEJ as well as
IIEEJ Trans. on Image Electronics and Visual Computing since 2016. He is a memberof IIEEJ, IEICE, IPSJ, ITE, IEEE and ACM.

13) A. Sugiura, K. Akachi, A. Yoshida, C. Ito, S. Kondo,
K. Tanaka, H. Takada: “Experimental Study on Control of
Visually Evoked Postural Responses by Galvanic Vestibular
Stimulation”, Proc. of 12th International Conference on Computer Science and Education (ICCSE) (2017).
14) Y. Onuki, I. Kumazawa: “Reorient the Gazed Scene Towards
the Center: Novel Virtual Turning Using Head and Gaze Motions and Blink”, Proc. of 2019 IEEE Conference on Virtual
Reality and 3D User Interfaces (VR) (2019).
15) M. Kodama:“A Consideration on a Determination Method for
Screen Shake with Histograms in Motion Vectors”, Proc. of
5th IIEEJ International Workshop on Image Electronics andVisula Computing (IEVC2017), 1P-7 (2017).
16) M. Kodama:“A Screen Shake Determination Method Using
Histograms of Motion Vectors in Video Scenes”, IIEEJ Trans.
on Image Electronics and Visual Computing, Vol.6, No.1, pp.1–
12 (2018).
17) X.C. Chen, N. Chaddha: “Lossy Adaptive Coding of Motion
Vectors with Global Camera Motion Compensation”, Proc. ofConference Record of The Thirtieth Asilomar Conference on
Signals, Systems and Computers (1996).
18) H. Jozawa, K. Kamikura, A. Sagata, H. Kotera, H. Watanabe:“Two-Stage Motion Compensation Using Adaptive Global MC
and Local Aﬃne MC”, IEEE Trans. on Circuits and Systems
for Video Technology, Vol.7, Issue 1, pp.75–85 (1997).
19) M. Kodama: “A Consideration of A High-Speed Video Iden-

90

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
IIEEJ

Paper

-- Special Issue on Extended Papers Presented in IEVC2019 Part Ⅱ --Digital Contents for Creating and Watching 3DCG of Vehicles
Based on Drawing their Pictures
Shinji MIZUNO† (Member)† Faculty of Information Science, Aichi Institute of Technology<Summary> In this paper, the author developed a method to generate 3DCG models of trains and cars
by drawing their simple pictures with pens on papers. The author also developed a system that allows us
to see the generated 3DCG models of trains and cars running in a 3DCG diorama three-dimensionally. The
author created digital contents applying the developed method and system, and exhibited them at events
hosted by a railway company and an automobile company. With this content, the users could run 3DCG
models of train or car in the CG diorama and watch them immediately by just drawing pictures of vehicles
on papers with pens. More than 300 children experienced the contents at each event, and enjoyed creating
3DCG vehicles by drawing and watching them. The author conﬁrmed that the proposed method was usefulto create interactive contents which attract many children at events.
Keywords: 3DCG, drawing, train and car, trick art, digital content

1.

hand drawn pictures running in a 3DCG diorama threedimensionally.
Both trains and cars are very popular as entertainment
and education contents. Miniature trains and cars arevery popular among children’s toys7),8). There are many

Introduction

Recently, the use of digital contents has greatly increased in various situations such as events, shops, restaurants, stages, and so on1)–3). They have realized adding

animations on trains and cars9)–11), and museums12)–14).
Furthermore, since railway / automobile companies have
many opportunities to hold events for children and families, the demand for digital contents on trains and cars is

new values to familiar life and arts, creating unrealistic
spaces, and have been attracting many people from children to the elderly.
In such events, digital contents that can generate 2D/ 3DCG based on drawing or coloring are very popular and often used. In usual drawing and coloring, only
papers and pens are necessary, children can easily expe-

considered to be high.
A content for creating 3DCG models of trains and cars
based on hand drawn pictures has been also developed.
“Paper Train” and “Paper Racer” generate 3DCG models of trains and cars by drawing their side views in thesame way as the propsed method in this paper15). However, only the side of trains or cars are textured, and thefront and top of the vehicles have inappropriate colors.
“SKETCH RACING” also generates 3DCG models of
trains and cars by drawing their side views16). However,rience them, and adults can also enjoy them. Therefore,various digital contents based on drawing pictures on papers with pens are developed4)–6). These contents are
particularly in high demand for events targeting families,
and such contents with various target objects of drawing
and expression methods are still expected.
In this research, the author developed two novel 3DCG
contents: “I CAN DRAW” and “Toyota City Creation”
for children based on drawing objects on papers withthe created 3DCG cars are based on 3D models prepared
beforehand, and the shapes of the generated models are
not always as shown in their pictures. Also, the generated trains and cars are only placed in a general 3DCG
scene in both contents. On the other hand, textures areappropriately applied not only to the side but also to thefront and top of the vehicles, and the shapes of the generated models are exactly the same as the pictures of theirpens. The target objects of drawing are trains in “I
CAN DRAW”, and cars in “Toyota City Creation”. The
author developed methods to generate 3DCG models of
trains and cars from hand-drawn simple pictures of themfor each content. The author also developed a systemthat allows us to see the trains and cars generated from91

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）•whole 3DCG diorama•cockpit viewprojector•a part of 3DCG diorama
cameraPC
3DCG models
of vehicles

PC

projector•a part of 3DCG diorama
projector
KinectLAN
projector

projector

picture•a part of 3DCG diorama

3DCG vehicles generation system•a part of 3DCG diorama

3DCG diorama display system

Fig. 1

Overview of our content

side views in our method. And the generated vehicles canbe observed three-dimensionally as if they were actuallyrunning in a 3DCG diorama in front of us. In “Toyota
City Creation”, it is also possible to interact with the
3DCG diorama.

2.

a method for generating a more detailed 3D shape bydrawing contours inside in addition to the outside contour18). Zeleznik et al. proposed a method for generatinga 3D shape by analyzing the contour lines of a bird’s eyeview of an object19). Kondo et al. proposed a method forgenerating a three-dimensional shape by analyzing the
shadow and contour of an object20). Eitz et al. proposed

Overview of the Contents

a sketch-based 3D shape modeling by retriving from alarge amount of shape data21).

The contents developed in this research are “I CANDRAW” for drawing and generating trains, and “Toyota
City Creation” for drawing and generating cars. Bothcontents consist of a system that generates 3DCG models of trains and cars from hand drawn pictures, and asystem that displays and interacts with a 3DCG diorama
in which trains and cars generated from pictures are running. Figure 1 shows the overview of our contents.
The system for generating 3DCG models of vehicles
consists of a PC and a camera. This system takes animage of the hand-drawn vehicle in the paper using thecamera, analyzes the image, and generates a 3DCG modelof the vehicle from the image in real time.
The system for displaying 3DCG diorama consists of a
PC and 5 projectors. This system receives 3DCG vehicles’ data via the network from the system for generating
3DCG models. In addition, the system builds a city diorama by 3DCG, and runs the received 3DCG vehicles inIn the content of this paper, the user creates a 3DCG
model of the vehicle by drawing. The users of this content are mainly children, and it is necessary to be able
to easily generate 3DCG models of cars while enjoying
drawing on papers with pens. From the perspective of thecontents in this paper, the methods of the references17)
and18)are suitable for modeling curved surfaces and are
not suitable for modeling vehicles. The methods of thereferences19)and20)require drawing while imagining a 3D
shape, which is too diﬃcult for children. The method of
the reference21)would be unrealistic because it requires ahuge amount of data for the child’s freely drawn shapes.
As mentioned in the former section, the methods of thereferences15)and16)generate a 3DCG of a vehicle by drawing, but the texture and shape are incomplete. Also,
drawing only the side view of a car vehicle may not giveenough satisfaction.
In this paper, the author considered the balance between the enjoyment of drawing and the ability for chil-the diorama. The 3DCG image of the whole scene is divided and projected by four projectors on a ﬂoor.

3.

Generating 3DCG Models of Vehicles
from Hand-Drawn Pictures of Vehicles

dren to easily generate a 3DCG of vehicle. That is, theshape and color of the side view of a vehicle can be freely
drawn, and in addition, the front or top surfaces of thevehicle can be drawn. Then, it is possible to generate
3DCG of various vehicles by only using the pictures.

3.1

Policy of generating 3DCG models in the
proposed contents
Many methods have been proposed to generate 3Dmodels from hand-drawn sketches. Igarashi et al. proposed a method for automatically generating a 3D shapefrom a hand-drawn outside contour17). Li et al. proposed

92

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

Fig. 2

3.2

Fig. 3

Examples of pictures of train

Fig. 4

Examples of pictures of car(a) train(b) car
The template sheets for drawing vehicles

Drawing pictures of vehicles

In the contents of this paper, the user draws a pictureof a train or a car with color pens on a paper. The users
are assumed to be children including infants. Thus, theauthor has prepared template sheets on which the minimum elements necessary for drawing a train or a car are
drawn beforehand (Fig. 2). It is not necessary to use
the template as long as the positional relationship between the side view and the front view in drawing a trainor the side view and the top view in drawing a car ismatched. Details are described in the next section.
In the case of a train, the user draws a train seen fromthe side on the right and seen from the front on the left of
the template sheet. In the side view area of the template
sheet, the rough shape of the train and the wheels are
drawn beforehand as a guide. In the front view area, arectangle is drawn beforehand. Users who are not goodat drawing can draw the appropriate train by drawing
the outer shape of the train as well as the guide. Users
who are good at drawing can draw the side view of the
train in any shape. The outer shape should be drawn
with dark lines such as black or blue. This is to reliably

using various colored pens inside the car (Fig. 4).
3.3

Generating 3DCG models of vehicles frompictures

After drawing a vehicle, the pitcture is taken with a
web camera. The paper is placed horizontally when atrain is drawn, and it is placed vertically when a car isdrawn. The system applies adaptive thresholding to the
picture to extract two closed regions from the picture withthe area sizes larger than a threshold. Then the positional
relationship between the two regions is examined by using
the center of gravity of each region. If the regions are at
the left and right of the picture, it is determined to be a
train, and if they are above and below the picture, it isdetermined to be a car.
Figure 5 shows the process of generating a 3DCG
model of a train from a picture. The region on the rightis the side view of a train and the region on the left is thefront view. First, the system ﬁnds the bounding boxes ofextract the side view region, the front view region, andthe top view region of a vehicle from the picture takenwith a web camera, which is the ﬁrst step of generatinga 3DCG model described in the next section. The user
can draw freely using various colored pens inside the train(Fig. 3).
In the case of a car, a user draws a picture of the careach region (Fig. 5(b)). The region of the side view isalso approximated to a polygon. Then, the side view isdivided into the wheel region and the body region basedon the lower point of the front view. In the side view, thefront shape region is determined based on the straightline extending from the lower left corner of the body region (Fig. 5(c)). The slope of the straight line is 1.0,
which was determined experimentally. The 3D shape ofthe front surface of the train is modeled by using vertices

seen from the side and the top. The area for the side viewand the area for the top view of a car are prepared in thetemplate sheet. In the side view area of the template
sheet, the bottom of the car and the wheels are drawn
beforehand as a guide. In the top view area, a rectangle
is drawn beforehand. The shape of the car is generally
more varied than the train, so the guide of the template
sheet is minimal. The user can draw various shapes ofcars on the template sheet. The user can draw freelyof the polygon inside the front shape region. Then, thebounding boxes of the side view are put on both sides ofthe front surface, and a rectangle for a roof is also put.
As a result, a 3DCG base model of a train is generated

93

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

mapping

front shape region

body region
wheel region(a) An original picture 1(b) The bounding boxes of(c) The wheel region, the(d) The base model of theeach region, and approximating the side view region to a polygon

body region, and thefront shape region of thepicture 1

train from the picture 1

front shape region

mapping

body region

wheel region(e) The 3DCG model of the(f) An original picture 2

train from tne picture 1

Fig. 5(g) The wheel region, the(h) The 3DCG model of the

body region, and thefront shape region of thepicture 2

train from the picture 2

The process of generating a 3DCG model of a train from the picture(Fig. 5(d)). The installation height of the roof is determined by the height of the inclusion rectangle in the front
view. The width of the roof is determined by the width of
the bounding box of the front view, and the length of theroof is determined by the length of the bounding boxes ofthe side view and the edge of the front shape region. Finally, the side view image is mapped to the rectangles ofboth sides of the base model as textures with masks. The
3DCG base model is cut out in the shape of the side view
image of the train. The front view image is also scaled tothe height of the front surface and mapped to the surface.
As a result, a 3DCG model of the train is completed (Fig.
5(e)). Figure 5(f)–(h) is another example. Although the(a) An original picture(b) The bounding boxesof each region, andapproximating
theside view region to apolygonfront shape region in Fig. 5(g) is larger than in the caseof Fig. 5(b), the process is the same. The 3D shape of thefront surface of the train is modeled by using vertices ofthe polygon inside the front shape region, and the front
view image is scaled and mapped to the front surface. In
this example, the front view image is elongated in the vertical direction. The upper and lower parts of the smoke
have only sides.
In the case of a car, a 3DCG model is generated in(c) The base model of the car(d) The 3DCG model of thecar

Fig. 6

The process of generating a 3DCG model of a carfrom the picture

the side view and the front view (the top view) of a vehicle
are arranged horizontally when the picture is taken bya web camera. In the case of a train, the body region
and the wheel region are divided based on the positional
relationship between the side view and the front view.

almost the same process as for a train (Fig. 6). There isno setting of the front shape region like a train, and theshapes of the front, top, and rear surfaces of the model are
generated based on the polygon that approximates theside view. Then the top view image is scaled and mapped
onto the front-top-back model by parallel projection.
In the method proposed in this paper for generating a

Therefore, if the picture is tilted greatly when taking witha web camera, an appropriate 3DCG model of vehicle may
not be generated.

3D model of a vehicle from a drawing, it is assumed that94

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）(a) A track root image(b) A mountain area image
Fig. 8(c) A water area image

A two-car train(d) The generated 3DCG
dioramaFig. 7

4.

Generating the 3DCG diorama

Displaying a 3DCG Diorama and Vehicles Running in It

4.1 Generating a 3DCG diorama
The 3DCG display system of our content has a 3DCG
diorama for running 3DCG models of vehicles. The

Fig. 9terrain of the 3DCG diorama is composed of a twodimensional grid (128×72), and the geographical featureof the 3DCG diorama and the color are automaticallygenerated by giving binary images which are showing the
track, the mountain area, and the water area, respectively(Fig. 7). The altitude of the mountain area is determined based on the Euclidean distance transformation of
the mountain area image. CG trees are automaticallyplaced randomly in the mountain area. CG buildings
are placed manually. The track is approximated as equilateral polygons for applying track textures and guiding
vehicles.
The 3DCG display system receives 3DCG model dataof each vehicle generated by the 3DCG model generation
system via network. In this implementation, the author
used the standard ﬁle sharing function of OSX to receive

The whole 3DCG diorama divided into four areas andprojected by four projectors

same time. Each car is composed of about 200 triangles,
and the diorama is composed of about 36,000 triangles.
An ordinary PC (MacPro / MacBookPro) can render a
3DCG scene composed of these data in real time (30fps).
4.2

Three-dimensional
3DCG diorama

displaying

of

the

The image of the 3DCD diorama is projected by ﬁve
projectors connected to the display system. Four projectors are used to project 3DCG images on a ﬂoor. The
whole 3DCG diorama is divided into four areas, eachof which is individually projected by the four projectors(Fig. 9). The size of the image by one projector is about
460×260 (cm).
At this time, an anamorphosis method: one of methods for trick art is applied to the 3DCG diorama images
projected on the ﬂoor. A picture drawn based on the
anamorphosis method looks distorted when viewed fromthe front, but it can be properly observed when viewed
from a speciﬁc place and it can give the observer a threedimensional eﬀect. The system of this paper realizes the

3DCG model data via network. Each data is composedof the polygon of outer shape of a vehicle’s side view and
texture images (the side view, the front view, the topview).
Each time the data is received, a 3DCG model of avehicle is reproduced and placed at a point on the trackof the 3DCG diorama, and starts moving. The speed of
the vehicle is determined by the direction of the side of
the polygon approximating the track at that point. Thus
the vehicle moves along the track. In the case of the train,
a two-car train is realized by arranging two 3DCG train

three-dimensional eﬀect on the 3DCG diorama by applying the anamorphosis method.
Figure 10 (a) shows the process of generating a 3DCG
image with a principle of anamorphosis. To generate animage with anamorphosis from a 3DCG model, the viewpoint for observation and the screen in the real space aredetermined in advance, and the CG viewpoint and projection plane that reproduce them are set in a 3DCG space.

models in diﬀerent directions (Fig. 8).
Multiple vehicles can run on the track at the same time.
In this implementation, up to 50 vehicles can run at the95

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

3DCG image for anamorphosisuser

CG viewpoint
projection plane

real space

3DCG space(a) The process of generating a 3DCG image with a principle of anamorphosis(b) An image for projection(c) Seeing the projected 3DCG diorama
from the speciﬁc place

Fig. 10

Seeing the 3DCG diorama three-dimensionally based on the principle of the trick art

points are set to a height of 110 (cm) on the assumptionthat children will observe them. This was decided basedon the average height of the 3rd grade elementary school
students is about 128(cm)22), and the eye position when

Fig. 11looking at the ﬂoor is 15–20 cm below the head top in
our experiment.
Another projector is used to project the whole 3DCG
diorama image on a wall. It is also used to project images
seen from the cockpit of the train or the car running in the
3DCG diorama (Fig. 11). With these images, the user
can enjoy the atmosphere running in the 3DCG diorama

Scene from a cockpit of a car

Then, a 3D model is placed in the 3DCG space and a CG
image is generated. The generated image is distorted,
but when displayed on a predetermined screen in the real
space and observed from a predetermined viewpoint, theimage is correctly observed. As a result, the anamorphosis eﬀect is obtained, and the CG image is observed
stereoscopically.
In this implementation, four viewpoints in the real
space to observe each ﬂoor image are determined in ad-

as a driver.
4.3

Interaction

The 3DCG display system can interact with the user’s
action. A Kinect sensor is installed at the center of thefour images projected on the ﬂoor, and the system acquires the positions and hand movements of users withina certain area around the projection images.

vance. The system reproduces a projection plane anda CG viewpoint in the 3DCG space as well as the relationship between each projection area and each viewpointin the real space, and generates 3DCG diorama images
for projection (Fig. 10(b)). When a user sees the 3DCG
diorama images projected on the ﬂoor from the predetermined viewpoints for each image, the images are observed
as if the 3DCG diorama were built three-dimensionally on
the ﬂoor by the anamorphosis method (Fig. 10(c)). The

When user shakes the hand, objects are generated andthrown from the corresponding position in the 3DCG diorama. In this implementation, users can ﬂy petals interactively to make ﬂowers bloom in the 3DCG diorama because the theme of the event is for recovery from disaster.
As a result, the user can feel as if ﬂying objects to the 3D
diorama.

trains and cars are also observed as if they were actuallyrunning in the 3DCG diorama. Note that, the four view-

96

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）(a) Children drawing pictures of trains(b) Taking a hand-drawn picture of a (c) A 3DCG train appearing in the
train with a camera and generatinga 3DCG train

3DCG diorama from above(d) 3DCG trains running in the 3DCG (e) Visitors observing the 3DCG dio- (f) Visitors watching the driver’s view
dioramarama

Fig. 125.

image

Exhibiting “I CAN DRAW” at an event hosted by a railway company

Exhibiting Contents in Events

The author implemented the 3DCG modeling systemand the 3DCG diorama display system for two contents: “I CAN DRAW” and “Toyota City Creation”.
The 3DCG modeling system was implemented in C++
on Mac Book Air (2GHz Core i7) using OpenGL and
OpenCV. The resolution of the web camera is 1920×1080(pixels).The 3DCG diorama display system was implemented in C++ on MacPro (3.5GHz 6-core Xeon E5) using OpenGL and OpenCV. The reason of using MacPro
was to connect ﬁve projectors.
The author exhibited the content “I CAN DRAW” fordrawing and generating trains at an event hosted by arailway company for the company’s 30th anniversary heldin Toyota, Japan on February 3, 2018. Figure 12 showsthe event, and Fig. 13 shows examples of 3DCG modelsof trains generated by children. About 350 children cre-Fig. 13

3DCG models of trains generated by children

pleased that the event was a great success with a large
number of children participating.
The author exhibited the content “Toyota City Creation” for drawing and generating cars at an event hosted
by a an automobile company in Toyota, Japan on May 13,
2018 (Fig. 14). About 330 children created 3DCG models of cars by drawing pictures of car. The children drewa wide variety of cars, and the 3DCG models of cars generated from the pictures were also very unique (Fig. 15).
Since a lecture on ﬁsh was held during the same event,many 3DCG cars with ﬁsh motifs were created. In theinteraction of ﬂying petals to make the 3DCG diorama
bloom, not only children but also many adults enjoyedthis interaction.
The author received a report from CBC Creation, Inc.
that the automobile company was pleased that many employees of the company participated in the event withated 3DCG models of trains by drawing pictures of train.
The children who experienced it seemed to be very happy
while being surprised that 3DCG models of trains weregenerated from pictures they drew. Then, while watching the 3DCG diorama, the children looked for the trains
generated from their pictures with their family, and they
were delighted and chased when they found their trains.
Some children walked around the diorama, following thetrains they had generated.
The author received a report from CBC Creation, Inc.:a management of this event that the railway company was

97

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）(a) Children drawing pictures of cars(b) Taking a hand-drawn picture of a (c) The 3DCG diorama projected withcar with a camera and generating a
3DCG carﬁve projectors on a ﬂoor and on a wall(d) 3DCG cars running in the 3DCG dio- (e) A user interacting with the 3DCG (f) Flowers that bloomed in 3DCG diorama

dioramaFig. 14

Fig. 15

rama due to interaction with users

Exhibiting “Toyota City Creation” at an event hosted by an automobile company

3DCG models of cars generated by children
Fig. 16

6.

their children and enjoyed drawing.

“Rakugaki Cruise”

Conclusion

The vehicle 3DCG generation system was also used inanother content “Rakugaki Cruise” at a commercial event
held at a shopping mall in Okinawa from February 28 toApril 12, 202023). The content itself is an extension of the

In this paper, the author developed a method to generate 3DCG models of trains and cars by drawing theirsimple pictures with pens on papers. The author also
developed a system to see the generated 3DCG models

two contents introduced above, such as being able to create a 3DCG city by drawing, and being able to immersethe 3DCG scene, but the part of generating a 3DCG vehicle is the same (Fig. 16). The method of generatingof trains and cars running in a 3DCG diorama threedimensionally. The author created digital contents byapplying the proposed methods, and exhibited them at
events hosted by railway companies and automobile companies. Many children enjoyed the contents.
In the contents of this paper, participants can draw
pictures of cars and run them in the 3DCG diorama, butthey cannot create the 3DCG diorama itself. As mentioned at the end of section 5, “Rakugaki Cruise” hasa 3DCG model of a vehicle from a drawing proposed inthis paper has worked successfully for over a month at
the commercial event, over 500 people generated vehicles
from their drawing, and the content has been very well received by visitors and event organizers. Therefore, it canbe said that the proposed method is useful suﬃciently.

realized that participants can draw and create buildings

98

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）in the 3DCG diorama, and the author would like to de-21) M. Eitz, R. Richter, T. Boubekeur, K. Hildebrand, M. Alexa:“Sketch-Based Shape Retrieval”, ACM Transactions on Graphics, Vol. 31, No. 4, Article 31 (2012).

velop a drawing interface that allows participants to draw
and generate terrain of the 3DCG diorama.
To realize more complex 3DCG models such as airplanes, buildings and characters, and more complex interactions are future works.

22) Ministry of Education, Culture, Sports, Science and Technology: “Annual Report of School Health Statistics Research2019”, https://warp.ndl.go.jp/info:ndljp/pid/11293659/www.
mext.go.jp/component/b menu/other/ icsFiles/aﬁeldﬁle/2019/03/25/1411703 03.pdf (2019).

Acknowledgements

23) S. Mizuno: “Proposal and Development of a Content to Immerse and Move in a Hand-drawn 3DCG space”, IPSJ SIG
Technical Report, Vol. 2020-DCC-25, No. 2 (2020).

I would like to thank CBC Creation, Inc. for supporting the exhibition of the two contents developed by us at
the events.

(Received March 9, 2020)(Revised August 9, 2020)

References

Shinji MIZUNO (Member)He received the Ph.D degree from Nagoya
University in 1999. He was a Research Associate at Toyohashi University of Technology
from 2000 to 2009. He is currently a professor at Faculty of Information Science, Aichi
Institute of Technology.His research interests
include computer graphics, image processing,virtual reality, and interactive arts. He is amember of IIEEJ, IPSJ, the Society of Art
and Science, VRSJ, and ACM SIGGRAPH.

1) Ikebana Sogetsu ﬂow × FLOWERS BY NAKED Live Performance, Old Imabashi ward, https://ﬂowers.naked.works/
event/4285/ (2019).
2) MOMENTFACTORY:TABEGAMI
SAMA,
https://momentfactory.com/work/all/all/tabegami-sama (2017).
3) SIRO-A: TECHNO CIRCUS,
tokyo/home-en (2019).

https://www.technocircus.

4) teamLab: “Sketch Aquarium”, https://www.teamlab.art/w/
aquarium/ (2013).
5) S. Mizuno, M. Isoda, R. Ito, M. Okamoto, S. Sugiura, M.
Kondo, Y. Nakatani, M. Hirose: “Sketch Dance Stage”, Proc.
of SIGGRAPH 2015, Posters (2015).
6) X. Cao: “Wonder Painter: Turn Anything into Animation”,
Proc. of SIGGRAPH 2018, Real-Time Live! (2018).
7) TAKARA TOMY, PLA RAIL, https://www.takaratomy.co.
jp/english/products/plarail/what/index.html (2020).
8) TAKARA TOMY, TOMICA, https://www.takaratomy.co.jp/
english/products/tomica/index.html (2020).
9) Ludorum plc, CHUGGINGTON, https://www.chuggington.
com/ (2020).
10) Gullane Limited, THOMAS & FRIENDS, https://www.
thomasandfriends.com/en-us/ (2020).
11) Disney, Cars, https://cars.disney.com/ (2020).
12) East Japan Railway Culture Foundation, The
Railway Museum, http://www.railway-museum.jp/ (2020).
13) Transport Heritage NSW Limited, NSW Rail Museum, https://www.nswrailmuseum.com.au/ (2020).
14) Toyota Motor Corporation, Toyota Automobile Museum,https://www.toyota.co.jp/Museum/english/ (2020).
15) RICOH, Paper Train & Paper Racer, http://www.ricoh.co.jp/
rental/paper app/ (2017).
16) Little Planet:
SKETCH RACING,
attraction/sketch racing/ (2019).

http://litpla.com/

17) T. Igarashi, S. Matsuoka, H. Tanaka: “Teddy: A Sketching
Interface for 3D Freeform Design”, Proc. of SIGGRAPH ’99,
pp. 409–416 (1999).
18) C. Li, H Pan, Y. Liu, X. Tong, A. Sheﬀer, W. Wang: “Robust
Flow-Guided Neural Prediction for Sketch-Based Freeform Surface Modeling”, ACM Transactions on Graphics, Vol. 37, No.
6, Article 238 (2018).
19) R. C. Zeleznik, K. P. Herndon. J. F. Hughes: “SKETCH: An
Interface for Sketching 3D Scenes”, Proc. of SIGGRAPH ’96,
pp. 136–170 (1996).
20) K. Kondo, H. Shizuka, W. Liu, K. Matsuda: “A Sketch Interpreter System with Shading and Cross Section Lines”, Journalfor Geometry and Graphics, Vol. 9, No. 2, pp. 177–189 (2005).

99

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
IIEEJ-- Special Issue on Extended Papers Presented

System Development Paper

in IEVC2019 Part Ⅱ--

Cooperative E-learning Applications Based on HTML-5 Canvas
for Japanese Classical Literature Education
Eri YOKOYAMA† (Member) , Hiroshi SUNAGA† , Makoto J. HIRAYAMA† (Fellow)† Osaka Institute of Technology<Summary> This paper proposes two e-learning applications specially designed for classical Japanese
literature classes. The ﬁrst one is a groupware allowing users to put comments on the handscroll images
where a part of the handscroll is shown as one scene, and users can put memo cards on it and the scene
with them moves when scrolled. The devised point of the application is that message cards can be placedin any point of the handscroll, and can be moved and modiﬁed, and the processed data can be stored in the
database. The other is a jigsaw puzzle game using classical literature images. The shape of each piece issame and rectangular, but users must look at the detail of each piece to accomplish the puzzle game, and it
helps the students watch the literature images seriously. Students who have actually used these applications
say they have been helpful in learning literature and it can be said that they eﬀectively work to help get
unmotivated students interested in literature classes. It was also acknowledged that they have enhanced
their interest in programming techniques as well, through playing these games.
Keywords: HTML-5, e-learning, literature application, handscrolls and paintings on a folding screen

1.

Introduction

In the faculty of Information Science and Technology
in Osaka Institute of Technology, as the social sciences
and humanities ﬁeld in the general education course,
there are two literature classes, namely “Elementary Literature” and “Literature with Visual Representation”.
Their syllabi describe that these literature courses arelinked tightly with the Information Technology as thecourses of this faculty and aim at letting students feel
literature familiar with them.
By introducing Web/smartphone applications into theclasses, it is expected to enhance students’ concentration on historical content and to further understand themeaning and backgrounds of literature works through discussion and mutual communications by group working.
The content includes illustrated handscrolls and pictures
drawn on a folding screen. They are included into applications and manipulated by students.
The main purpose of this research is applying methods
for increasing students’ interests and motivations to learn
Japanese classical literatures. Since most of students inscience and engineering departments are not familiar withstudies in humanities and have little interests and motivations to learn. We are trying to increase students’100

interests and motivations to learn Japanese classical literature by developing and using cooperative e-learning
applications related to edutainment technologies.
This paper proposes e-learning applications to activate the classes in literature. We also expect a synergistic eﬀect, where students get close to programmingor software development for their forthcoming graduation researches through the manipulation of these applications. The applications are based on the computing
services framework1), where Cloud/Web services, existing applications, and software components are combined
and necessary parts are programmed to satisfy the requirements2)–4). This paper proposes such e-learning applications, and shows the eﬀects of introducing them tothe literature classes. The rest of this paper is structured as follows. Section 2 discusses requirements andapplication images. Section 3 describes implementation
techniques and section 4 shows the eﬀects of introducing
these applications into literature classes. Finally, section
5 concludes this paper.

2.

Requirements

2.1 Features of literature classes“Elementary Literature” deals with both proses andverses by using image content, video content, and printed
images as well as explanation by voice. Here, by overlook-IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

ing at works from the ancient/medieval times to the Edoera, “The Tale of Genji (Genji Monogatari)” and othermasterpieces such as “Ogura Anthology of One Hundred
Tanka by One Hundred Poets (Hyakunin Isshu)”, students are led to understanding of the backgrounds orsituations of the story. “Literature with Visual Representation” uses image content from a literature databasenamed “The Largest Digital Reference on Pre-Modern
Japanese Works” of “National Institute of Japanese Literature”5) or the web-site of “National Diet Library Digital
Collections”6). In particular, it is important for students
working on a printed image, e.g., an illustrated handscroll, to add text or information on it.
We have changed these analogue ways to digital
environments. Image content has been included in
Web/smartphone applications to be manipulated andgiven new values by students. Also students very carefully study image content in the form of group working
or game. Hopefully, they may learn basic knowledge andunderstand what ancient people think of through visualcontent and interesting way of learning avoiding excessive explanation only on original texts. Among various elearning applications we have developed, this paper deals
with two applications that are excellent in terms of questionnaire results as shown in section 4, namely a groupwork application using handscrolls and a puzzle game using folding screens.
2.2

Current situations of classical literature
educations

The reason why we have developed and introduced suche-learning applications and settled the syllabi is that wewould like to propose a solution to improve the current
miserable situations of Japanese classical literature educations. According to investigations performed by the
National Institute for Educational Policy Research7), only
29.3% of the junior high school students surveyed positively answered to the question “Do you like classical literature?”. This was one of the questions in the questionnaire performed in 2013 on academic performance andlearning situations. Also, to the question “Do you likeclassical literature and Chinese classics?”, performed forhigh school students in 20058), 71.2% and 72.1% of themnegatively answered respectively.
Then, according to this report8), the main reason was
the excessive teaching on basic and elementary knowledge and techniques, thus leading to boredom and dislike of learning. The solutions were as follows. Teachers

101

Fig. 1 Group-work application on illustrated handscrolls

should educate so that their students can touch how ancient people see, feel, think of things at that time by not
only teaching original texts but also devising educational
tools. Then, teachers should enhance and deepen thisteaching method to augment the interest and motivationof the students.
Considering these reports, we started to try the active
usage of IT materials to the classes so that students may
eventually acquire elementary knowledge and techniques
for classical literatures with proactive manner.
2.3
2.3.1

Overview of applications
Group-work application using handscrolls

This application uses illustrated handscroll content,
named “Shigisan Engi Emaki”, oﬀered in the “National
Diet Library Digital Collections” site. “Shigisan Engi”
means the history of Shigisan Chogosonshi-ji Temple in
the Mt. Shigi. This content consists of three volumes,
each of which has height of about 30cm and width of
800–1,400 cm. The site oﬀers them as 26–39 jpg ﬁles.
As shown in Fig. 1, the application has the display
area, a slider bar, text area and several buttons. In this
implementation, the display size is 600×1,500 px. In caseof Volume 1, 26 jpg ﬁles are combined by hand to one
jpg ﬁle of a long length. The currently displayed image
is changed in accordance with the slider value.
In the conventional work, handscrolls are ordinarily displayed by using buttons9)and some pictures with longlength can be scrolled by slider or mouse operation10). In
this paper, we devised to use message cards which canbe placed in any point of the handscroll, and they can be
moved and modiﬁed, and processed data can be stored in
the database. Such functions are our originality.
If a user wants to write some text on the currently

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

displayed image, the user inputs text to the text area. Byclicking a point on the display, a box area i.e., a message
card is created at that point. The characters are vertically
written. Even if the display area is changed by the slider,
the relative position of the card is kept. The color of thecard and text can be later modiﬁed by using buttons.
Each user can select a color, thus each card can be
identiﬁed so that teacher can previously give basic information and students can add what they feel, think of,
and want to show. This application can be extended so
as to paste photo ﬁles.
To share information with users including the teacher,a special formatted record data are devised. Records aremanaged by a Web database server which provides services for recording, updating and retrieving. By thesefunctions users can submit their own opinions or information, receive those of the others, and build up a consensus.
Technically, the store/restore function and database connection function are provided. Also, a currently displayed
image can be converted to a png ﬁle.

2.3.2

Fig. 2 Jigsaw puzzle of Genji Monogatari pictures
drawn on a folding screen for questioning

Fig. 3 Puzzle of Genji Monogatari Emaki for answering

Puzzle game using folding screens

Pictures drawn on a folding screen for Genji Monogatari are transformed to a Jigsaw puzzle like game. Fig.
2 shows the area for setting a question, and Fig. 3 forthe answering area. Note that jigsaw-cut is not used foreach piece, but rectangle-cut is used for the easy implementation and image content itself is of the largest importance for education. An original image is separated
into small pieces each of which is randomly disposed.
There are many jigsaw puzzle games found in programming sites11),12), in which one picture ﬁle is divided into
pieces and randomly disposed. In our application, as we
attach importance to teaching and learning, the shape ofeach piece is set same (simply, square or rectangle) and
let the users look into the details of image pieces to learn
the historical and literal backgrounds of the era. A user
touches a piece in the questioning area and sets it at a position in the answering area. If the user wants to change
the position in the answering area, a touched piece canbe set at another position. When completed, all positions
are judged from the viewpoint of whether or not they arecorrect. In case of an incorrect answer, one piece can be
put back.

102

Fig. 4 Setting a part of ﬁle to canvas area

3.

Techniques to Construct Applications

3.1 Group-work application using handscrolls
3.1.1 Display and slider control
The display area is constructed by using the HTML5 Canvas element13), which is managed by JavaScript14)
programs. A Canvas area is created by the canvas tag,whose height and width attributes are set as 600×1,500
px, respectively (Fig. 4) in this implementation. The
original ﬁle size is 34,311×1,240 px, which has been madeby adding separated jpg ﬁles, and a part of this ﬁle is cut
and shown as a scene in accordance with the slider value.
The slider bar is created by an input tag with the“range” type. By moving the slider, a range value canbe extracted, where the max and min values correspondto the left and right ends, respectively. By dividing theabsolute value of the slider by the length between min
and max values, a relative position can be determined.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

In the ﬁgure, the aspect ratio of the display is 1:2.5,
while that of the original ﬁle is 1:27.6. To show the pieceof the picture ﬁle designated by the slider, the parametersof the “drawImage function” are set as follows. The yaxis value of the original ﬁle to be cut is 0. That of x-axis
is the position calculated by the slider relative value. In
the same way, the height is 1,240 px, and the width is27.6 px. Each time a user moves the slider, the picture
ﬁle is redrawn.
3.1.2 Message card setting
Figures and text are overwritten on the canvas, i.e., thecurrently displayed piece of the handscroll ﬁle. If a user
clicks a point on the Canvas, x- and y-axes values are obtained. A rectangle is drawn by the “ﬁllRect function” of
JavaScript by setting the top-left coordinate of the card,and its width and height values as shown in Fig. 4. Text
is drawn by the “ﬁllText function”. As this function only
permit horizontal writing, we make a subroutine by which
the given characters are divided to a single character andeach is disposed vertically in the card area by calculating
the number of characters and the card size.
The values of each card are kept as an object and thenpushed into a list (an array of a variable size). In case of
JavaScript, an object is deﬁned as a function containing
only variables without methods as shown in Fig. 5. This
corresponds to a composite data type representing theproperties of one message card.
3.1.3 Modiﬁcation of message card
When the slider is moved and a new scene is drawn, all
the records are extracted from the list, and all the message cards are redrawn in accordance with data in eachrecord. The record contains the slider and x-axis value.
The original handscroll ﬁle is cut to show the backgroundscene designated by those values. Also, the record contains relative x-axis positions on this scene. By using
them, all the message cards are overwritten on the scene.
If a user wants to move an existing card or modify current message or color, ﬁrst the designated card must be
identiﬁed. When a user clicks a positon in the display
area, x- and y-axes coordinate values are obtained by
the event listener function programmed for collision detection processing. This function successively compares
the stored x/y coordinate values in each record with theclicked coordinate values. If the clicked coordinate values
are within the range of “x + width” and “y + height”,
this message card is identiﬁed, that is, the detection is
successful. If the user wants to modify the text or color,these data in the record are replaced from the older ones.

103

Fig. 5 JavaScript class for the record

Fig. 6 Example of the special data format

Fig. 7 Example of the Json data

If the user wants to delete this card, this record is removed
from the list. Also, the card position can be moved.
3.1.4

Application programming interface

As this application is driven by JavaScript programs on
the browser, all the records will disappear if the browser
is closed. To keep them, we provide a database function
in a Web server and deﬁne Web APIs. If the user push
the save button, the program sends the records to the
database server by means of the Ajax15) communication.
The request is of the form of a URL, its parameters must
be a character sequence.
Therefore we introduce a special data format for the
Web access. All the records are extracted from the list,and every data element in them is separately read andconcatenated to a long character sequence as shown in
Fig. 6. This sequence is added to the URL with theargument name (record).
The example of Fig. 6 includes three records. If we
explain the ﬁrst one, it has values “0, 800, 300, 120, 210,
1, ‘ﬀ0000’, ‘This is a handscroll’ ”, which mean the scene
number, the x-axis value from the lest end of the ﬁle, they-axis value, the width of the card, the height of the card,
the sequence number, the RGB value, and the message,
respectively.
When this sequence is transferred to the server, it is
kept in the data base as one record value. When it isIIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

returned to the browser from the server, the handlingprogram decomposes it by ‘|’, and further decomposeseach separated element by ‘,’ to data values, RGB anda message. These are used to replicate the scene on thecanvas.
The server provides functions to insert, update, delete
and read the designated table. This server provides
REST16) interface, and uses jersey17) as the library. As
for the database, sqlite18) is used because it is lightweighted. To the read request, the response is formedin the Json19) format. An example is shown in Fig. 7.
3.1.5

Character and scene database

Fig. 8 Display of major element in handscroll

To let students understand scenes more deeply, we introduce a database explaining major characters, animals,
objects and landscapes, as shown in Fig. 8. Each picture is cut from the handscroll and visually processedso as to become clearer by using Photoshop. Also, eachis attached metadata explaining the name of the person
or object, or the scene with historical/literary meanings.
The keyword category search function is provided as well
as a keyword search function.
As shown in Fig. 9, the picture ﬁles can be displayed.
To show the contents in accordance with the user device,
the Responsive Web Design (RWD) approach is introduced. The layout employs a ﬂuid and proportion-based
grid structure, which is changed by judging the viewing
environment through CSS3 media queries. Pictures form
up in four columns when viewed in a PC display, threecolumns for a tablet, and two for a smartphone.

Fig. 9 Smartphone interface

Moreover, voice input, voice recognition and textto-speech functions are implemented by using Amazon
Alexa20) and Amazon Echo21). These functions improveusability and aspects of entertainment.
3.2

Puzzle game application

3.2.1

Picture ﬁle handling

A jigsaw-like puzzle is consisting of many pieces eachof which corresponds to the answer picture. From theprogramming viewpoint, it is very easy if separated ﬁlesfor each part are previously prepared, while the separation work is time-consuming and boresome. We propose
a method using only the original picture ﬁle.
As shown in the lower part of Fig. 10, a picture ﬁleis managed in a grid form (6 × 3 in this example), eachpiece of which is given an index (i, j) and identiﬁed byusing this index. From this original picture, each pieceis dynamically copied to the answer area as shown in theright part of Fig. 10. As the HTML Canvas capabilities

104

Fig. 10 Puzzle handlinginclude the partial copy function as discussed in section
3.1.1, at the time of game initialization, all the pieces ofthe original ﬁle are copied and positioned randomly on
the question grid area.
More speciﬁcally, each pieces of the original ﬁle is givena sequence number, 0 · · · 17 in this case. This sequencenumber corresponds to grid coordinate (0, 0) · · · (i, j) · · ·of the original ﬁle. The numbers, 0 · · · 17, are randomly
sorted as an array and kept until the end of the game.
This array is used as the conversion array, and copied

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）as the question array. In the question area, each partial

piece in the answer area can be moved by a touch process-ﬁle of the piece is drawn in accordance with this order.

ing event handler, the user can rearrange the position toThis game is implemented for both the Web browser andcome closer to the right answer. In addition, explanation

Android. The drawing method for the former is basicallysame as described in section 3.1.1.
In case of Android, the Bitmap class is prepared forthe ﬁle handling. The Bitmap instantiated from a ﬁlecan be cut partially by designating the pixel coordinateof the left-top, width (px), and height (px). This partial
bitmap can be placed to any position by designating theseparameters. For example, assume that grid coordinate(0, 0) is given sequence number 11, which is randomly
generated. Number 11 corresponds to grid coordinate

pages are prepared. These pages as well as the game itself
help the students to more deeply understand the scenes,
total story, and historical backgrounds. The game variations can easily be increased for teacher by utilizing theoriginal image ﬁles and their explanation texts.

(1, 5), when the image is composed of 6 × 3 grids. The
original position is designated by the left-top coordinate,i.e., x = 0 (px) and y = 0 (px), the grid width, and thegrid height, and thus a partial bitmap is created. The
new position is designated by the coordinate of the newposition, x = 5 × width (px), y = 1 × height (px), thegrid width, and the grid height. In this way, each piece is
set randomly to the question area from a single original
picture ﬁle.
3.2.2 Game processing
The answer area is managed in the same way. Also,
the answer area is given grid coordinate (0, 0) · · · (i, j) · · ·from top-left to bottom-right. The initial values of thequestion area are the same as the conversion array. So
the randomly positioned pieces (image) are drawn in thequestion area. On the other hand, all the initial valueof the answer area are set to −999 (out-of-scope value),
which means that no image is drawn in the area.
If a user touches a position on the question area, theevent listener informs its handler program of the touchedcoordinate values. The grid coordinate values of i and jwill be calculated by using width and height of the piece.
At this time, the piece number as well as the touchedstatus are memorized, and the sequence number of thispiece is given −999. By redrawing the question area basedon the question array, the touched position is not shown.
Then the user touches a position in the answer are,and the answer array is given the piece number at thisposition. By redrawing the answer area based on theanswer array, the selected image is shown.
When all the pieces are set on the answer area, the answer array is compared with the conversion array. If all
the numbers match correctly, the game is over. If there
is more than one mismatch, the user can reconsider. The
last piece is brought back to the question area. As each

105

4.

Evaluation of Applications

4.1

Eﬀects of the applications

4.1.1

Group-work application

We introduced applications to “Elementary Literature”
and “Literature with Visual Representation”. This application helps students to understand picture content
deeply and express their ﬁndings about what the content
means, what the characters are doing, what situations
are described. This is a kind of active learning, wherestudents try to ﬁnd something important or even trivial
in a spontaneous way and learn a research and opiniondelivery process.
A student ﬁrst searches for a character or a place he/she
is interested in by scrolling the handscroll by using theslider or jump buttons. Then, he/she writes a comment
about it, selects color, and attaches the message card on
the display area. If he/she ﬁnds a new fact or has new
opinion by investigating references or group discussion,
he/she may use new color, which can record the transitionof understanding. The teacher can show a speciﬁc scene
and easily move to another scene. Also, the teacher can
prepare what he/she want to tell in the form of message
card. Formerly it was diﬃcult to hand a paper handscrollof a long length or many sheets of cutouts to all the students but every student can manipulate the total part of
the handscroll in a large class. In addition, it promotes
remote studies by accessing to the server from outside of
the class room.
We performed questionnaires survey on the handscroll
application in January 2018 towards 219 students in the“Elementary Literature” class, and 42 students in the“Literature with Visual Representation” class. The students ﬁrst used the application before receiving explanation about the handscroll, and then we performed theﬁrst survey. At the next class, we lectured the handscroll
as an ordinary class, and after it we performed the second
survey.
The ﬁrst survey included two questions. The ﬁrst one
is whether or not the impression of each student about

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

Fig. 11 Survey on eﬀects of handscroll application

Fig. 13 Survey on eﬀects of folding screen puzzle application
Fig. 12 Survey on eﬀects of handscroll application in
conjunction with lecture

the handscroll has been changed. As the second one, thesame question to the classical literature was posed. Asshow in Fig. 11, these results mean that only the usageof the application did not lead to augmentation in theinterests of the students.
The second survey examines the eﬀects of the application in conjunction with the lecture. As show in Fig. 12,
three questions were posed. The ﬁrst one was about theeﬀectiveness to understand the handscroll. The next one
is whether or not the impression about the handscroll
has been changed. The last one is whether or not the
impression of each student about the classical literature
has been changed. In comparison with the previous questions posed before the lecture, the results imply that theuse of the application in conjunction with the lecture iseﬀective to augment the interests and understandabilityof the students.
4.1.2

Folding screen puzzle application

We observed students in the two classes as described in
section 4.1.1. The folding screen picture, “Genji Monogatari Emaki”, is diﬃcult to understand without knowing what the scene implies. By transforming the original
picture to may pieces of puzzle, each student must concentrate on details of the scene, particularly the four borders, and come naturally to understand what the scene
describes. As the Genji pictures have a characteristic, i.e.,interior description, this application leads the students tounderstand deeply and eﬀectively.
We also performed questionnaires on the jigsaw puzzle application in April 2019 towards 45 students in the106“Literature with Visual Representation” class, as shownin Fig. 13. The ﬁrst question was whether or not theapplication changed the impression of the folding screen
picture, and 40.0% answered positively that they becamemuch more interested than before. The others were neutral, and there was no negative answer. The second
question was whether or not the application changed theimpression of classical literatures, and 37.8% answered
positively and the others answered neutrally. The third
question was whether or not they understood how to appreciate pictures, and 46.7% answered positively, while
the others answered neutrally. The fourth question waswhether or not the application let the students more eﬀectively learn the classical literature by receiving explanations after playing the puzzle game, and 20.0% answered“very eﬀective”, 62.2% “eﬀective”, and the others answered the same. The ﬁfth, ﬁnal, question was whether or
not the students would like to appreciate other pictures,and 15.6% answered very positively, 57.8% positively, andthe others answered the same. By performing chi-square
tests to these data, signiﬁcances were validated for alldata at 5% level.
Considering the answer of the fourth question, whichobtained about 80% positive answers, to ﬁnd neighboringpieces, the shape, tincture, or continuity of lines must be
checked carefully and the situation must be understood.
The students proceeded with this work with special concentrations and memorized pieces. This action very muchhelped the understanding of the teacher’s explanations
after the game. Some students added a comment thatthey could understand the structure of the old Japanesebuilding and interior arts through this game. Also, theresult of the ﬁfth question shows more than 70% would

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

like to enthusiastically study other classical pictures. In

and lead to a future image of education ﬁelds in the dig-the other questions, it seems that the use of this gameitalized environments. First, it can transform the teach-

had a level of eﬀects while the attitude of about 60% of

ing style from a text-reading-centered one to a kind of

the students answered was the same.
With respect to free comments, there were answers toenhance the above mentioned results. Student answeredthat they would like to try more complicated puzzle, orthat the game made students more familiar with classicalliteratures. These comments show that the game playing contributes to nourish the attitude of the students toappreciate the depth of the scene the picture implies.

project-based one aimed at learning through studying on
one s own initiative. Secondly, from the IT literacy viewpoint, it helps students to acquire capabilities of extracting information from content and expressing what they
understand and analyze. Finally, it cultivates their perceptiveness to connect rich heritages from classical literatures with present lives and information technologies.

4.1.3

5.

Overall eﬀects

As described in section 2.2, the Ministry of Education, Culture, Sports, Science and Technology (MEXT)
expects teachers to help their students better perceive
how ancient people saw, felt, and thought in the past.
Considering the above mentioned evaluation results, ourapplications would ﬁll such requirements. There are threepoints we would like to emphasize.
(1) Lectures using the applications can shorten the distance between classical literature and students. By
touching image content through tablet or Web applications, it is expected to motivate students who are
reluctant to study the subject. In a lesson about traditional Japanese folding screens, for example, theteacher can focus on the part he/she attaches importance to without having to go into details about
techniques to appreciate the works.
(2) The Genji puzzle application creates a traditional
Japanese cultural space. It gives the students anopportunity to experience how ancient people might
have seen, felt, and thought at the time, which the
MEXT prioritizes for literature education.
(3) Each student can adaptively manipulate the applications in accordance at his/her own pace. In particular, the Shigisan group work application can show
and deliver his/her opinions or awareness to the workby the message card function. As study progresses,
such opinions can be revised or enhanced. Also, by
the group work function, data can be shared in theclass, while cooperation spirit is likely to be fosteredin the process. Thus, our applications are expected
to help students deepen their study in multiple ways.
4.2

Further study items

It is expected that the introduction of IT applications
to literature classes will help the below-mentioned planning of desirable lectures for classical literature classes

107

Conclusion and Future Work

We have developed e-learning applications for classicalliterature classes and introduced into lectures. These help
students to understand classical handscrolls and pictures
on a folding screen more deeply and enjoyably. After all,students acquire precise knowledge, and discuss actively
with each other, through the acquisition of the manipulation of the applications.
Technically, by using the HTML-5 and Android functions new types of teaching applications or self-learning
applications have been created. The manipulation to thedisplay image attracts attention to precise scenes, character actions, objects surrounding or possessed by them,and backgrounds. The dynamic behavior of our application motivates students to learn more deeply, think ofdetails of the situation, and investigate the teaching materials without teacher’s excessive inclusion, compared withthe conventional applications dealing only statistic texts
and images.
Eﬀects to the classroom are ﬁrstly improvement in attitude to the learning of unmotivated students by the combination of ordinary explanation and the applications. In
particular, as the handscroll application provides a tryand-error approach for both individual and group work,where students can freely think of comments, select colorand dispose the message card to the most impressive position, they proceed with their own work or group workeﬀectively.
We have developed and incorporated other applications. All of them have dynamic properties that leadto the synergistic eﬀect of visual and auditory perception. For example, although the learning of Hyakunin
Isshu (100 poems) is ordinarily apt to only use visual
images and text, in our application images move in thescreen and the poem is read aloud. Quest games arealso implemented and used for further precise knowledge
acquisition. We will also report eﬀects of a variety of ap-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

plications except for the two applications picked up thistime.

References

1) H. Sunaga, A. Morikawa:“Memory Usage Reduction Method
for Bitmap Drawing in Android Programming - Evacuation
Guidance Application -,” IEICE Technical Report SC201818(2018).
2) IEEE, IEEE Technical Committee on Services Computing,
http://tab.computer.org/tcsvc/index.html (2020).
3) H. Sunaga, Y. Yamato, H. Ohnishi, M. Kaneko, M. Iio, M. Hirano: “Service Delivery Platform Architecture for the NextGeneration Network”, Proc of ICIN 2008 International Conference on Intelligent Networks, Session 9-A (2008).
4) H. Sunaga, K. Nakajima:“Data Management for 3D Floor Plan
Drawn on HTML-5 Canvas - Floor Plan Data Management,”
IEICE Technical Report SC2017-15 (2017).
5) National Institute of Japanese Literature, Database of PreModern Japanese Works, https://kotenseki.nijl.ac.jp/ (2020).
6) Center for Open Data in the Humanities, Dataset of JapaneseClassics, http://codh.rois.ac.jp/pmjt/ (2020).
7) Ministry of Education, Culture, Sports and Technology(MEXT) National Institute for Educational Research(NIER), National Assessment of Academic Ability 2013,https://www.nier.go.jp/13chousakekkahoukoku/data/
research-report/13-questionnaire.pdf (2013)8) Ministry of Education, Culture, Sports and Technology(MEXT), Survey on Operational Situation of Curriculums,
Analysis and Improvement per Subject (Japanese and Integrated Japanese) in 2005 (2005).
9) National Diet Library, National Diet Library Digital Collections Legend of Mount Shigi, http://dl.ndl.go.jp/info:ndljp/
pid/2574276,2574277,2574278 (2020).
10) web-emaki.com,(2020).

Urashima

Taro,

http://web-emaki.com/

11) Ponksoft, jQuery based Jigsaw Puzzle, http://ponk.jp/jquery/basic/jigsaw (2020).
12) ao-system, Online Jigsaw Puzzle Making, https://ao-system.
net/jigsawpuzzle/ (2020).
13) WHATWG and W3C, HTML-5, http://www.HTML-5.jp/(2020).
14) ECMA, JavaScript (ECMAScript), ECMA-262, Edtion 10(2019).
15) W3C, Asynchronous JavaScript ＋ XML, Ajax Tutorial,” http://www.w3schools.com/ajax/default.asp (2020).
16) Java Community Process, The JavaTM API for RESTful Web
Services, JSR-000311 JAX-RS (2020).
17) Eclipse Foundation, Jersey, http://jersey.java.net/ (2020).
18) Hipp, Wyrick & Company, Inc.(Hwaci), SQLite, http://www.
sqlite.org (2020).
19) IETF, The JavaScript Object Notation (JSON) Data Interchange Format, RFC7159 (2014).
20) Amazon, Amazon Alexa, http://alexa.amazon.com/spa/index.
html (2020).
21) Amazon, AmazonEcho, https://www.amazon.com/ (2020).

(Received September 12, 2019)(Revised July 31, 2020)

108

Eri
YOKOYAMA (Member)
She received Ph.D in Literature, from Nara
Womens’s University in 2016. Currently, she
is an assitant professor at Faculty of Information and Science and Technology, Osaka
Institute of Technology. Her research interest includes classical literature especially for
Emaki (illustrated hand scrolls).

Hiroshi SUNAGAHe received the Ph.D. degree in information sciences fromTohoku University in 2002.
Currently, he is a professor of Osaka Institute of Technology, where he has contributed research and development of Web
and smartphone applications since 2009. Before then, he worked on developmental research of switching node systems, IP telephony technologies, and Peer-to-Peer communications in NTT. He is a member of IEICE
and IEEE.

Makoto J. HIRAYAMA (Fellow)
He received B.E. and D.E. from Waseda University in 1985 and 1996 respectively, M.B.A.
from Leicester University in 2004, and B.L.A.
from the Open University of Japan in 2008.
He worked at Hewlett-Packard Japan 1985–
1994, also at ATR Auditory and Visual Perception Research Laboratories 1989–1992 andATR Human Information Processing Laboratories 1992–1994, at Hewlett-Parckard Laboratories Japan 1994–1999, and at Kanazawa
Institute of Technology 1999–2013. Since
2013, he is a professor at Osaka Institute ofTechnology. His research interest includes
speech production and multimedia applications.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

Upon the Special Issue on CG & Image Processing Technologies
for Automation, Labor Saving and Empowerment
Editor: Masanori SEKINO(Fuji Xerox Co., Ltd.)A declining birthrate and an aging population are a common problem in developed
countries. In order to solve the labor shortage, it is necessary not only to expand theworkforce but also to improve labor productivity. Therefore, improvement in productivity is
expected through automation and labor saving technologies using evolving image
recognition technology, increasing information from IoT devices, and cheaper robots.
At the call for submission, we received 12 papers, 8 in Japanese and 4 in English.
Concerning English papers, one of them is adopted and included in this issue, and other threepapers are still under review. The fields covered in the submitted papers are diverse andinteresting, such as defect detection at production sites, long-term care support, sports scene
recognition, CG image conversion, and pathological image segmentation. Despite the variety ofcovered area, many of them discuss robustness against environmental changes anddisturbances, and the stability of machine learning, and ensuring robustness. Therefore,stability seems to be a common issue in social implementation of such technology.
In our journal, we are soliciting “system development papers” and “practical papers”,
which are the categories of the papers that make it easy to present the outcome of research
focusing on the practical and applied aspects of technology. As the technology recruited in thisspecial issue will well match with these paper categories, we hope your further submission of
the papers related to this issue especially in these categories.
Finally, I would like to appreciate the reviewers and editors for careful review and all theefforts to improve the quality of papers. Also, I would to thank to the editorial committee
members of IIEEJ and the staff at IIEEJ office for various kinds of support.

109

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
IIEEJ-- Special Issue on CG & Image Processing Technologies for Automation,
Labor Saving and Empowerment --Paper

Bidirectional Mapping Augmentation Algorithm for Synthetic Images
Based on Generative Adversarial Network
Haoqi GAO† (Student Member) , Koichi OGAWARA†† The Faculty of System Engineering, Wakayama University<Summary> In training deep neural networks for supervised learning tasks, we often use data augmentation methods to increase training dataset sizes. Furthermore, this technique is particularly useful when thesize of the training datasets is small, such as when the content of the training datasets includes privacy issues
that cannot be made public, or the categories in the train datasets are unbalanced. During the training
process, small training datasets will lead to model overﬁtting. Nowadays, data augmentation methods using
Generative Adversarial Network (GAN) and Neural Style showed to provide performance improvements forthe task of supervised learning. However, the traditional GAN is easy to cause the network to collapse,
which makes the generation process free and uncontrollable. Hence, it may cause the network model tofail to produce deterministic results, which makes the application limited. In this paper, we propose an
improved GAN-based data augmentation method for image classiﬁcation tasks. We compare our modelwith the latest GAN model, and the results show that our algorithm is eﬀective. On generated images,
when applying synthetic images to facial expression attribute classiﬁcation task, our method achieves 72.5%
accuracy rate on the FER2013 PrivateTest datasets and 71.2% accuracy rate on the FER2013 PublicTest
datasets.
Keywords: data augmentation, generative adversarial network, neural style, image classiﬁcation

1.

Introduction

Nowadays, training models using neural networks usually require a large number of labeled datasets. However,the process of labeling data is deﬁnitely time-consuming
and expensive, and manually labeling the data can easily
lead to many errors. In contrast, synthetic datasets do
not require manually annotated data, which reduces theburden of both the data collection and labeling process.
Theoretically, we can synthesize an unlimited number oftraining datasets. In addition, the synthetic samples allow us to precisely control the rendering process of theimages, such that the datasets have diﬀerent characteristics. Nevertheless, how to narrow the “domain gap” between synthetic datasets and real-world datasets is still amajor challenge.
Among them, face analysis has always been a research
hotspot in the ﬁeld of computer vision and neural networks due to its theoretical signiﬁcance and great practical application value. While the technology of faceresearch is mature, it is still deﬁnitely challenging forfaces captured by a camera in an unconstrained realworld environment. Real-world face datasets usually face110

the following challenges: (1) The number of speciﬁc facedatasets (such as face data with large oﬀset angles orcomplex expressions) is not suﬃcient for the training of
the model. (2) Due to privacy reasons, there are only afew public datasets available at present. (3) Preparationand labeling of training datasets usually require a great
deal of time and expensive investment. In this way, according to the main challenges in the given research ﬁeld,
this paper introduces the topic of synthetic face datasets.
With the rapid development and advancement ofgraphic imaging technology, researchers have been using
some 3D models1) to generate synthetic datasets. For
many tasks, synthetic datasets2),3) were used to minimize the cost and risk of training the network models. Nonetheless, it is still technically challenging to construct high-quality synthetic-to-real datasets. Though
the model is well-trained, it is not well-suited for realworld test datasets. Brieﬂy, if synthetic datasets are signiﬁcantly diﬀerent from real datasets, it will aﬀect thedecision made based on the model. Therefore, if the synthetic datasets are similar to the real datasets, the modeltrained with the synthetic datasets can be used more effectively on the real test datasets.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

ness of data augmentation by performing minor modi-,QSXW

ﬁcations on training datasets (such as image cropping,rotation, and ﬂipping)6). Up to now, Generative Adversarial Network (GAN)7) and Neural-style transfer8) techniques have been proposed as a powerful technique to do
the data augmentation. In this section, we brieﬂy review
some representative works.

2XWSXW,QSXW

2.1

Traditional transformation is based on a series of wellknown aﬃne transformations to process training datasets,such as rotation, ﬂipping, scaling, and distortion, as well
as some simple image processing methods, such as light-

2XWSXW
Fig. 1

Traditional transformations

Our experiment results

Our research plan considers the privacy of the facedatasets. Through FaceGen4), we can obtain large-scale
datasets from various angles and various facial expresses.
Nevertheless, even when FaceGen synthesized datasets
achieve high accuracy in network training, it is not sufﬁcient to validate on real-world test datasets, so we consider adding feature information of real datasets to synthetic datasets. This paper, however, is diﬀerent from
CycleGAN5) which transfers from mode A to mode B.
We hope that the generated face needs to maintain thecharacteristics of the synthetic face and the real face atthe same time.
In our translation model, there will be two mapping
functions. The top of Fig.1 represents one mapping: theinput is synthetic images, our network attempts to generate an image as real as possible. The bottom shows
another mapping: with real-world images as input, ourmodel generates a synthetic image that incorporates feature information of the input.
Our works and contributions will be summarized asfollows:

ing color transformation, contrast transformation, andadding noise6). We use these methods to generate newdatasets from a limited number of datasets. This can alleviate the problem of overﬁtting of the neural network
to a certain extent, and improve the generalization ability of the model. Nevertheless, compared to the original datasets, the increased datasets have not been fundamentally addressed by the problem of under-ﬁtting of
the datasets. Simultaneously, this augmentation method
requires manually adjusting the transfer function and corresponding parameters, which are usually dependent on
empirical knowledge. It is hard to achieve the best performance with those data augmentation methods.

1. Our network improves the sense of the reality of synthetic data and narrows the “domain gap” between
synthetic datasets and real datasets.

As GANs have been proposed in 2014, there have been
issues such as the diﬃculty of training, the diﬃculty of
convergence, and the lack of diversity in the generated
samples. Since then, many researchers13)–16) have tried

2. We used FaceGen-based synthetic facial datasets with
diﬀerent facial appearances (like smiles, anger, fear,
and neutral) as our input images. Our model istrained with these facial expression datasets separately. At test time, if you input an image, the output
is a synthetic image with diﬀerent facial expressions.

2.

Related Work

Since we all know, the algorithms based on deeplearning are more eﬀective when providing more training datasets. Past studies have shown the eﬀective111

2.2

Background on GANs and style transfer

Consider the limited problem of traditional transformation methods. Image synthesis is a relatively new research topic at present9),10). Generative Adversarial Nets
(GANs) has been a powerful technique to generate newimages for training6). GANs have been applied to a variety of tasks and have achieved remarkable results in all
aspects of image generation11),12).

to solve those problems and proposed improvements. Like
W-GAN13) uses the Earth-Mover instead of the JensenShannon divergence to measure the distance between thereal sample and the generated sample distribution. Conditional GAN (CGAN)14) proposes to add additional information, which can be labels or other ancillary information. SimGAN15) complements the adversarial loss witha self-regularization L1 loss that penalizes large changes
between the synthetic and reﬁned images to make thesynthetic datasets more realistic and can be used to enrich

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
5HVB %ORFN'HFRQYROXWLRQ&RQYROXWLRQ'‫ݕ ݕ‬*‫ݔ‬ሺ*‫ݕ‬ሺ‫ݕ‬ሻሻ

5HVB%ORFN*‫ݔ‬ሺ‫ݔ‬ሻ

5HVB%ORFN

濧
&RQY

&RQY

5HVB%ORFN

5HVB%ORFN

5HVB%ORFN

6W\OHIHDWXUH

5HVB%ORFN

5HVB%ORFN

5HVB%ORFN&RQY

6W\OHLPDJH漏GLIIHUHQWHPRWLRQ漐

[
[
[ [
The architecture of our proposed GAN

unlabeled real datasets. The pix2pix framework16) proposes a generator to learn the mapping function between
two paired images. CycleGAN5) introduces a cycle generator confrontation network and puts forward a transfer
between diﬀerent image domains without the need forspeciﬁc image pairs. Another hot and creative researcharea is that of style transfer17), which combines style andcontent of an image skillfully through a neural network
to form an interesting image. These depend on features
extracted from a pre-trained neural network.
Furthermore, there has been a great deal of work in
the area of generating synthetic face datasets by using
GANs18)–20). These methods can address problems suchas privacy of data, class imbalance, and denoise of images.
For instance, Wan et al.21) proposes FM-GAN for face
generation of ﬁne-grained multi-attribute. STGAN22)
treats the face synthesis problems as transferring stylesof one face to another. PGGAN23) is a breakthrough forsynthesizing high-quality realistic face images. PairedCycleGAN24) relies on a cycle-consistent generative adversarial network method for editing a portrait photo.
FaceID-GAN25) used expression features to modify facial
expression.

112濙*‫ݕ‬ሺ‫ݕ‬ሻ

3UHWUDLQHG 5HV1HW
5HVB%ORFN

5HVB%ORFN

5HVB%ORFN&RQY*‫ݕ‬ሺ*‫ݔ‬ሺ‫ݔ‬ሻሻ

Fig. 2濙'‫ݔ‬ሺ‫ݔ‬ሻ*‫ݕ‬

5HDO ‫ݕ‬

&RQY

濧
&RQY

5HVB%ORFN

5HVB%ORFN

5HVB%ORFN

5HVB%ORFN

5HVB%ORFN6\QWKHWLF ‫ݔ‬

&RQY

*‫ݔ‬

3.

&RQWHQWIHDWXUH/RVVIRU
JHQHUDWRU

Method Overview

Figure 2 shows the overall architecture of our proposed network with both synthetic images and real images as input. Consider our datasets: synthetic imagesof x ∈ X and real images of y ∈ Y as well. In this work,
we use two generators: Gx , Gy and two discriminators:
Dx , Dy . Branches of both the synthetic and real image
generation processes are speciﬁed by the blue and purple
channels, respectively.
We adopt CycleGAN5) network architecture andWasserstein adversarial loss13) to learn the mapping
for the generator network. The generator network is
composed of four convolutional layers, several residual
blocks26), and transpose convolutional layers. The number of residual blocks is determined by the size of the
input image, for example, we use six blocks for 128 × 128
images and nine blocks for 256 × 256 and high-resolution
training images. The discriminator network consistsof four convolutional layers with leaky rectiﬁed linear
units(ReLU). Researchers5),16) proposed a path-level discriminator architecture which has fewer parameters thana full image discriminator. And it can work on arbitrarily
sized images in a fully convolutional fashion. For discriminator networks, we use PatchGANs5),16),27),28), which aim
to classify whether 70 × 70 overlapping image patches are

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

real or fake. Besides that, we also use the instance nor5),29)

at most one everywhere, so they consider directly con-

for all layers of the generator andstraining the gradient norm of the critic’s output concern-

discriminator networks, with the exception of the input

ing its input. To circumvent the tractability issues, theyand output layer.

also enforced a version of the constraint with a penalty
on the gradient norm for random samples x̃g ∼ Pg .
We express the objective as:

malization scheme

3.1

GAN review

The objective function of GAN is a minimax game ofa two-player: the generator and the discriminator. We
deﬁne Pr to be the data distribution, Pg to be the model
distribution deﬁned by x̃g = G(z), where z is the input
to the generator. The discriminator network D adjusts
its weights to reliably distinguish real data samples xr ∼Pr from fake data samples x̃, randomly sampled fromsome distribution Pg , via the generator network. The
generator network G adjusts its weights to fool D. The
two networks are trained iteratively using a loss function
given by:
LGAN = min max Exr ∼ Pr [log(D(xr ))]+
G

D

(1)

Ex̃g ∼ Pg [log(1 − D(x̃g ))].
When the discriminator is trained to its optimal state
before each generator parameter is updated. Minimization of the function is equivalent to minimization of theJensen Shannon divergence between Pr and Pg . However,doing so often leads to vanishing gradients as the discriminator saturates. To address GAN’s problems, Arjovsky13)
illustrated the GAN problem mathematically caused by
the JS divergence approximation and proposed to use theEarth-Mover Distance. The WGAN value function constructed using Kantorovich-Rubinstein duality30), can be
simpliﬁed to calculate:
LW GAN = min max Exr ∼ Pr [D(xr )]+
G

D∈Z

(2)

Ex̃g ∼ Pg [1 − D(x̃g )],where Z is the set of 1-Lipschitz functions. To enforce
constraints, WGAN applies a simple clipping to restrict
the maximum value of the weight. The weights of thediscriminator must be within a speciﬁc range controlled
by the given hyperparameters. However, they found thatweight clipping in WGAN leads to optimization diﬃculties due to the interactions between the weight constraint
and the cost function, and even in the case where the optimization succeeds, the resulting critic may have a pathological value surface. Gulrajani13) proposed WGAN-GP
uses a gradient penalty instead of weight clipping to enforce the Lipschitz constraint. The diﬀerentiable functionis 1- Lipschitz if and only if it has gradients with the norm

113

LW −GP = LW GAN + λEx̃g ∼ Pg (||∇x̃g D(x̃g )||2 − 1)2 .
(3)In this paper, we set our paramether λ = 10 in theequation3.
3.2 Our loss functions
In comparison to the CycleGAN loss, we use theWasserstein GAN (WGAN-GP) rather than the sigmoid
cross-entropy loss that is used in the CycleGAN model.
Results show the superior performance to the original
generator architectures in terms of both the convergenceof the generator and the sample quality.
The goal of our task is to learn a mapping function thatthe distribution of images from Gx is indistinguishable
from the distribution of Y by using an adversarial loss.
Two generators mapping Gx : X → Y and Gy : Y →X. Two adversarial discriminators Dx and Dy proposed
to distinguish whether images translated from anotherdomain. So, we also use the cycle consistency loss deﬁned
by CycleGAN.
LCY C (Gx , Gy ) = Ey∼ Py [||Gx (Gy (y)) − y||1 ]+Ex∼ Px [||Gy (Gx (x)) − x||1 ](4)

For encouraging the transfer, we preserve the consistency of the facial feature information between the input
and the output. We added style loss and content loss
from Johnson et al.17) have shown impressive results forneural style transfer and super-resolution.
Applying the MSE loss function will cause the output
image to be loose, such as details or high-frequency parts
that are lost in the image. So appropriately selecting thefeatures of a layer output as the input of the perceptual
loss function can enhance the details. Our goal is to minimize the content loss between the content image and thegenerated one, while minimizing the style loss between
the style image and the generated one. So in this paper,
we use the ResNet50 network model26). Let φj (x) be theactivations at the j − th layer of the ResNet50 for the
input x , which is a feature map of shape (Cj , Hj , Wj ).
The formula of the Gram matrix can be expressed as:

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

1
φj (x)h,w,c φj (x)h,w,c .
C j Hj W j
w=1
Hj

Gφj =

Wj

(5)

h=1

The correlations between the diﬀerent ﬁlters applied at
a given layer. These feature correlations are captured the
texture patterns at a spatial scale, which correspond to
the physical appearance of textures, colors, patterns at
this scale. We pass the ﬁlter over the image to create thefeature maps at our layer, and then we compute the innerproduct between feature maps of a given layer. This innerproduct is called the Gram matrix.
The Style loss is the squared norm of the diﬀerence between the Gram matrices of the output and target image.
For optimization, style reconstruction is based on a setof layers of size J rather than a single layer. Incorporating feature correlations of multiple layers, we can obtainstationary, multi-scale representation of the input image.

VPLOH

Fig. 3VPLOH

FaceGen allows limited parametric control of facial
expressions

We deﬁne Lstyle to be the sum of losses for each layerj ∈ J. And it can be expressed as:Lstyle =(||Gφj (x) − Gφj (Gx (x))||2 +j(6)||Gφj (y) − Gφj (Gy (y))||2 ).
However, even though minimizing the style reconstruction loss preserves stylistic features from the target image it does not preserve its spatial structure. The content loss is the Euclidean distance between the output
image and the target image. Moreover, earlier layers
in our network are associated with more local information, whereas, in the higher layers, activations will contain more global information. Since the content is deﬁned
by the macrostructure of the image, the topmost layers
in our network will capture the content of the image17).
It enforces the generated image retaining the prescribed
structure, which is deﬁned as：
1
Lcontent =(||φj (x) − φj (Gx (x))||2 +
C
H
W
j
j
j
j
(7)
||φj (y) − φj (Gy (y))||2 ).
So, our total objective is:
Ltotal = LW −GP + βLCY C + Lstyle + Lcontent .

(8)

For training, we minimize the above objective function,
which consists of adversarial loss, cycle consistency loss,
style loss, and content loss. β is non-zero weights forbalancing loss functions.
3.3 Training details
We use Adam solver with a ﬁxed learning rate of
0.0002 and beta1=0.5. We set β = 10 in Eq.(8). The
114

Fig. 4

Samples of our training datasets

ResNet50 network is used to produce a feature descriptor for style transfer algorithms. The output of this network is the style and content features. We set content
layer: ‘res5b branch2a’, style layers: ‘res3a branch2a’,‘res4a branch2a’, ‘res5a branch2a’.

4.
4.1

Experiment Result

Datasets

Synthetic Images: Synthetic face images based on FaceGen4) model reconstruction which is shown in Fig.3.
The FaceGen system uses a “parameterized” approachto deﬁne the attributes that compose a face, and using
a set of ﬁxed parameters, a face model can be made tomorph and modify it independently. Once the texture of
the face is attached, the geometric details will be moreabundant in the face. You can see more information
about synthetic face images from the website31).
Real Images: We use the 300-W challenge32) real images in training. It includes AFLW33), AFW34), Helen35),
IBUG32), LFPW36). As shown in Fig.4, the upper part
represents real face images from the 300-W challenge.
The below part represents synthetic images from Face-IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
1HXWUDO$QJHU+DSS\)HDU'LVJXVW6DG6XUSULVH,QSXW

2XWSXW

Fig. 5)HPDOHUHDOV\QWKHWLF

Sample of images from FER2013 training database

Fig. 7

Other mapping results of our translation model,QSXW6LP*$1,QSXW

0DOHUHDOV\QWKHWLF:*$1*3&\FOH*$1

1R‫ܜܖ܍ܜܖܗ܋ۺ‬2XU

2XWSXW)HPDOHV\QWKHWLFUHDO

Fig. 60DOHV\QWKHWLFUHDO

One mapping results of our translation model

Gen 3D model reconstruction (from left to right, representing the facial expression attributes of neutral, anger,
happy, and fear respectively).
FER2013 Datasets: FER-201337) is from a Kaggle Facial Expression Recognition Challenge. It was collectedby Pierre and Aaron through the Google Image Search
Application Programming Interface (API). It is currently
the largest publicly available facial expression databasein the wild, making it possible for many researchers totrain machine learning methods where a large amount of
data38) is needed.
But this dataset is challenging: (1) Facial expressions
come from the natural environment, so low image resolution and diﬀerent angles will increase the diﬃculty, as
shown in the ﬁrst row in Fig.5. (2) There are various
watermark images(marked with a purple circle in Fig.5),
cartoon images(marked with a green circle), non-face images(marked with a blue circle) in the training datasets.
Especially，we need to focus on some facial expressions(marked with a red circle). The labeled category of thispicture is fear, but even for the human eye, it is diﬃcult to determine whether it is neutral or fear without
considering the actual environment.
4.2 Qualitative evaluation
In our result, we can get two models: S → R and
R → S(S: synthetic images R: real images). We canconvert the synthetic face images generated by the FaceGen 3D model into more realistic face images. Besides,
this process is reversible. Depending on diﬀerent tasks
and goals, you can choose from synthetic images to real
images or from real images to synthetic images. We giveour experiment results in Fig.6 and Fig.7.
Figure 6 shows female and male results of the model
S → R. We can convert the synthetic face image into a115

Fig. 8Diﬀerent models result for translation from real image to synthetic image

real face with multiple styles of expression. The advantage is that the generated face image has the high perceptual quality and can be used to improve the performanceof the facial expression classiﬁer.
Figure 7 shows female and male results of the model
R → S. We can see that our experiments perform wellwith large-scale expression poses. And the generated
faces maintain the characteristics of synthetic and real
images at the same time.
We use the same evaluation datasets and metrics andcompared our algorithm against several following latest
works: SimGAN15), WGAN-GP13), CycleGAN5). Amongthe four input images in Fig.8 is the normal image, animage with exaggerated expressions, a dark picture, andan image under intense light. From left to right: input, SimGAN15), WGAN-GP13), CycleGAN5), Ours without Lcontent , Ours. Nearly all images work well by our
method despite the diversity of their colors, expression
gesture, and brightness. The image generated by SimGAN is aﬀected by the background of the image, causing
the face connected to the background. WGAN-GP lacks
the details of the face. For the image under strong lightand dark light, some positions that were aﬀected by lightor darkness are missing from CycleGAN results. And if
we remove Lcontent from our method, it cannot learn the
background of synthetic data. The dashed box on the picture also shows the comparison between CycleGAN andour model.
We use an illustrative -Toy Experiment39),40) as ourIIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）%

Algorithm 1: Classiﬁer training using GAN data
augmentation$%

Data: Labelled FaceGen dataset Df acegen , real
dataset Dreal , Number of samples to$

2UL

Fig. 9

6LP*$1:*$1B*3&\FOH*$1

2XUV

Toy domain experiment result of diﬀerent GAN models

append each iteration Niter ;1

evaluation. It based on synthetic data in domains A and
B. Both samples are drawn from Gaussian mixture models. The task is to ﬁnd relations between domain A anddomain B and translate from domain A into domain B,
which has seven modes spread around the arc of a circle. In this experiment, we provide 100,000 datasets andrandomly divide the datasets into a training set and avalidation set at a ratio of 0.33, and test on 1000 samples. The training performed for 50,000 iterations, theresulting translated samples in Fig.9 show very diﬀerent
behavior depending on the model used.
In Fig.9, the target domain mark with black points.
Colored points on the domain B planes represent samples from domain A that mapped to domain B, and eachcolor denotes samples from each mode in domain A. The
left-most ﬁgure demonstrates the initial state of the toy
experiment, where all modes of domain A mapped to almost a single location because of the initialization of thegenerator. Results show many translated points of different colors localize around the same mode in domain
B. However, both SimGAN and WGAN-GP models fail
to cover all of the modes in domain B, because the mapping from domain A to domain B is injective. For example, some translated points of diﬀerent colors (likes
pink and gray) locate around the same B domain mode.
CycleGAN and Our (with a reconstruction loss) results
prevent mode-collapse by translating into distinct wellbounded regions that do not overlap. But some coloredpoints from domain A in CycleGAN mapped to fewer areas of the domain B than our results. Our discriminatorfor domain B perfectly fools by translated samples fromdomain A around the modes of domain B.
4.3 Quantitative evaluation
Our method can alleviate the class imbalance in the
training datasets by generating synthetic images. Thus,
we tried to use the augmented synthetic datasets as augmented data for training a classiﬁer to distinguish thefacial expressions from each other. The overall procedure
is described in Algorithm1.
(a) Metric
In Table 1, we summarize the calculation method of116

2Train our GAN using{Df acegen , Dreal };Generate sample set Dgen from GAN transfer;

3Form augumented dataset Daugmented⇐{Dgen , Dreal };4

Set Dtraining ⇐{};
Initialize CNN parameters;for Number of CNN training iterations do

5
6
7
8

for xn ∈Daugmented doAppend the samples xn with the Niter to
Dtraining ;9train CNN using Dtraining ;10

Evaluate CNN balanced accuracy using
testsets;

11
12

end
end
Table 1Deﬁnitions of the evaluation metrics

Performance Measure

Mathematical Formula

Precision

TP/(FP+TP)

F1

2TP/(2TP+FP+FN)

RecallTP/(FN+TP)

Accuracy(TP+TN)/(TP+TN+FN+FP)

TP:true positive, FP:false positive, TN:true negtive,
FN:false negative

F1-score, Accuracy, Precision, and Recall rate as our baseevaluation metric. We use this metric to compare thediﬀerence between the classiﬁer’s results and the groundtruth. The precision rate(the number of correct targets
divided by the number of all returned samples) shows
the positive predictive value. The recall (the number ofright targets divided by the number of targets that should
be returned) can reﬂect the sensitivity and real positive
rate of the model. The higher the F1-score(the harmonicmean of precision and recall), the better the performanceof the model.
(b) FER2013
The datasets for training the model contains 35,887
samples of 48x48 pixel grayscale images of faces. It includes 28708 training images, 3589 public test images,
and 3589 private test images. These images marked with
one of the 7 emotion categories: 4953 images express
anger, 547 images express disgust, 5121 images expressIIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

fear, 8989 images express happiness, 6077 images express

FER2013 and our synthetic samples. (I),(III) and (V)sadness, 4002 images express surprise and 6198 images

consisting of the same FER2013 images but with diﬀer-

are emotionally neutral. As shown in Fig.10.

ent synthetic images. Please note that the test datasets

We build a neural network classiﬁer that trained on:

and validation datasets of our model are still the original
test datasets and validation datasets of FER2013.
In Table 2, II, and III denote the model used diﬀerent synthetic training datasets respectively. The results
show that the model that used our synthetic datasets
as training outperforms the model that used the original synthetic FaceGen dataset. Comparing to I, we canalso see that there are improvements in terms of Precision, Recall and F1-score with V. In the experiments, themodel that trained using our generated images and real(I) only real samples taken from FER2013, (II) only original FaceGen synthetic samples. (III) the combination of
FER2013 and original FaceGen synthetic samples. (IV)
only our synthetic datasets(Using one of our mapping
models S → R to generate the synthesized image with
diﬀerent facial expression styles). (V) a combination of

images achieved the highest scores.
In Table 3, ‘R’ denotes ‘Real images’, ‘S’ denotes ‘Synthetic images’,‘SR’ denotes ‘Real images and Synthetic
images’. Our method based on the backbone of the VGG
network41),44). As a result, we achieve an accuracy rateof 72.5 on the Private Test dataset. Observe that thisperformance is a consequence of our adaptive data generation algorithm and the use of bidirectional mapping.
We presented the results on the Public Test datasets in

&RXQW







IHDU
KDSS\
VDG
VXUSULVH
QHXWUDODQJU\
GLVJXVWDQJU\GLVJXVWIHDUKDSS\VDGVXUSULVHQHXWUDO(PRWLRQ

Fig. 10

An overview of FER2013

Table 2Comparison of the evaluation metrics of the classiﬁcation model under diﬀerent training datasets
Evaluate
Top1-acc Top5-acc
Pre
Rec
F1
Methods
I

64.70

95.59

60.31

64.27

63.61

II

27.92

77.91

26.84

27.95

27.24

III

65.16

96.13

61.56

64.65

64.58

IV

38.28

81.76

32.37

37.91

32.52

V

66.54

96.29

63.02 66.14 66.07

Table 3Comparison of the diﬀerent classiﬁcation models
on FER2013 datasets
MethodTrained On Tested with Accuracy(%)

Human37)-R

65.0± 541)

GoogleNet

R

R

65.2

Xception42)

R

R

66.0

VGG

R

R

70.8

Ours

SR

R

72.5

43)

+DSS\

$QJU\

6XUSULVH)HDU

6XUSULVH

1HXWUDO

6DG

1HXWUDO

6DG

1HXWUDO

)HDU

$QJU\

Fig. 11

The predicted labels and corresponding probabilities of the sample face randomly selected in the test datasets

117

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

Fig.11. In the ﬁrst two rows of Fig.11, we have shown

the main feature of sad, so these two expressions will

the ground-truth labels and the corresponding predictedalso be confused during classiﬁcation. Fear is most eas-probabilities of our model. In the last rows of Fig.11, we

ily confused with sadness or anger. Because both facialshow a few examples with light blue labels to indicate
false predictions. These examples need to be considered
in a real-world environment and typically have a second
possible emotion in them. For example, for the ﬁrst sample of the last rows, the correct label is blue, the maximum probability predicted by our model is happy. Under
the natural condition (neutral expressions), the corners ofeyes and mouth corners will drop slightly, which is also

expressions have the same characteristics of tightening
eyebrows, straining the forehead, and opening lips.
Figure 12 and Fig.13 are the confusion matrix for themodel predictions on the test dataset. The matrix gives
the counts of emotion predictions and some insights into
the performance of the multi-class classiﬁcation model.
From the confusion matrix, we can see that the fear recognition is the most diﬃcult one, and happy and surpriseFig. 12

Normalized confusion matrix of our method on
FER2013 PublicTest datasets

Fig. 14

Fig. 13

Normalized confusion matrix of our method on
FER2013 PrivateTest datasets

The results of real-time facial application on 300-VW datasets45)

118

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）are easier to recognize than the other facial expressions.
The majority of the happy expressions have distinctive
features, such as the slightly open mouth, the lifted corners of the mouth. In contrast, disgust is easily confusedwith anger. Because the two expressions have the same
properties of the drooping eyebrow and the squinting.
An example of our complete application show in
Fig.14. Once the model train to its optimal state, we
can apply it to practical applications. Our Real-time facial applications include face detection, facial expression
classiﬁcation, and facial marker detection.

5.

Conclusion

We propose an Improved CycleGAN for bidirectional
mappings translation. Our network not only generates asynthetic image that incorporates real facial characteristics information but also real-world images with synthetic
style. The generated images demonstrate the superiority of our proposed method. Next, we make use of the
facial expression classiﬁcation model to assess the validity of the generated face images. On our datasets, therecognition rate of classiﬁcation is improved. We achieve
an accuracy rate of 72.5% on FER2013 PrivateTest andan accuracy rate of 71.2% on FER2013 PublicTest on
the task of facial expression attribute classiﬁcation. This
performance improvement attribute to our adaptive data
generation algorithm. In this experiment, we didn’t consider occlusion when creating synthetic datasets, such as
sunglasses, masks, and so on. But we know that real
datasets may have various possibilities. Therefore, further improvement to cope with such cases will be needed.

References
1) A. Todorov, N. N. Oosterhof: “Modeling Social Perceptionof Faces [Social Sciences]”, IEEE Signal Processing Magazine,
Vol. 28, No. 2, pp.117–122 (2011).
2) X. Zhang, Y. Fu, S. Jiang, X. Xue, Y. G. Jiang, G.
Agam: “Stacked Multichannel Autoencoder–an Eﬃcient Way
of Learning from Synthetic Data”, Multimedia Tools and Applications, Vol. 77, No. 20, pp. 26563–26580 (2018).
3) J. S. Supancic, G. Rogez, Y. Yang, J. Shotton, D. Ramanan:“Depth-Based Hand Pose Estimation: Data, Methods, andChallenges”, Proc. of the IEEE International Conference on
Computer Vision (ICCV), pp.1868–1876 (2015).
4) FaceGen, http://www.facegen.com (2020).)5) J. Y. Zhu, T. Park, P. Isola, A. A. Efros: “Unpaired Imageto-Image Translation Using Cycle-Consistent Adversarial Networks”, Proc. of the IEEE International Conference on Computer Vision (ICCV). pp.2223–2232 (2017).
6) L. Perez, J. Wang: “The Eﬀectiveness of Data Augmentationin Image Classiﬁcation Using Deep Learning”, Convolutional
Neural Networks Vis. Recognit, Vol.11 (2017).
7) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-

119

Farley, S. Ozair, A. Courville, Y. Bengio: “Generative Adversarial Nets”, Advances in Neural Information Processing Systems, pp.2672–2680 (2014).
8) L. A. Gatys, A. S. Ecker, M. Bethge: “Image Style Transfer Using Convolutional Neural Networks”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 2414–2423 (2016).
9) E. L. Denton, S. Chintala, R. Fergus: “Deep Generative Image
Models Using a Laplacian Pyramid of Adversarial Networks”,
Advances in Neural Information Processing Systems, pp.1486–
1494 (2015).
10) D. P. Kingma, M. Welling: “Auto-Encoding Variational
Bayes”, Proc. of Second International Conference on Learning
Representations (ICLR) (2014).
11) T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen: “Improved Techniques for Training GANs”,
Advances in Neural Information Processing Systems, pp.2234–
2242 (2016).
12) M. F. Mathieu, J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, Y. LeCun: “Disentangling Factors of Variation in Deep
Representation Using Adversarial Training”, Advances in Neural Information Processing Systems, pp.5040–5048 (2016).
13) I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. C.
Courville: “Improved Training of Wasserstein GANs”, Advances in Neural Information Processing Systems, Vol.30,
pp.5767–5777 (2017).
14) J. Gauthier: “Conditional Generative Adversarial Nets for
Convolutional Face Generation”, Class Project for Stanford
CS231N: Convolutional Neural Networks for Visual Recognition, Winter Semester, Vol.2014, No.5, pp. 2 (2014).
15) A. Shrivastava, T. Pﬁster, O. Tuzel, J. Susskind, W. Wang,
R. Webb: “Learning from Simulated and Unsupervised Images through Adversarial Training”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (ICPR),
pp.2242–2251 (2017).
16) P. Isola, J. Y. Zhu, T. Zhou, A. A. Efros: “Image-to-Image
Translation with Conditional Adversarial Networks”, Proc. of
the IEEE Conference on Computer Vision and Pattern Recognition (ICPR), pp.5967–5976 (2017).
17) J. Johnson, A. Alahi, L. Fei-Fei: “Perceptual Losses for RealTime Style Transfer and Super-Resolution”, Proc. of European
Conference on Computer Vision (ECCV), pp.694–711 (2016).
18) H. Q. Gao, K. Ogawara: “Generative Adversarial Network for
Bidirectional Mappings between Synthetic and Real Facial Image”, Proc. of Twelfth International Conference on Digital Image Processing, Vol. 11519, p. 115190J (2020).
19) Y. Wu, F. Yang, Y. Xu, Y. Ling: “Privacy-Protective-GAN for
Privacy Preserving Face De-Identiﬁcation”, Journal of Computer Science and Technology, Vol.34, No.1, pp.47–60 (2019).
20) C. Tian, Y. Xu, L. Fei, J. Wang, J. Wen, N. Luo: “Enhanced
CNN for Image Denoising”, CAAI Trans. on Intelligence Technology, Vol.4, No.1, pp.17–23 (2019).
21) L. Wan, J. Wan, Y. Jin, Z. Tan, S. Z. Li: “Fine-Grained MultiAttribute Adversarial Learning for Face Generation of Age,
Gender and Ethnicity”, Proc. of International Conference on
Biometrics, pp.98–103 (2018).
22) T. Karras, S. Laine, T. Aila: “A Style-Based Generator Architecture for Generative Adversarial Networks”, Proc. of theIEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp.4401–4410 (2019).
23) T. Karras, T. Aila, S. Laine, J. Lehtinen: “Progressive Growing of GANs for Improved Quality, Stability, and Variation”,
arXiv:1710.10196 (2017).

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）24) H. Chang, J. Lu, F. Yu, A. Finkelstein: “Pairedcyclegan: Asymmetric Style Transfer for Applying and Removing
Makeup”, Proc. of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), pp.40–48 (2018).

D. Chan, M. H. Mahoor: “Facial Expression Recognition fromWorld Wild Web”, Proc. of the IEEE Conference on Computer
Vision and Pattern Recognition Workshops (CVPR Workshops), pp.58–65 (2016).

25) Y. Shen, P. Luo, J. Yan, X. Wang, X. Tang: “FaceidGAN: Learning a Symmetry Three-Player GAN for IdentityPreserving Face Synthesis”, Proc. of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.821–
830 (2018).

39) T. Kim, M. Cha, H. Kim, J. K. Lee, J. Kim: “Learning to
Discover Cross-Domain Relations with Generative Adversarial Networks”, Proc. of the 34th International Conference on
Machine Learning, Vol.70, No.18, pp.57–1865 (2017).

26) K. He, X. Zhang, S. Ren, J. Sun: “Deep Residual Learning for
Image Recognition”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770–778(2016).
27) C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham,
A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, W. Shi:
“Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”, Proc. of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.4681–
4690 (2017).
28) C. Li, M. Wand: “Precomputed Real-Time Texture Synthesis
with Markovian Generative Adversarial Networks”, Proc. ofEuropean Conference on Computer Vision (ECCV), pp.702–
716 (2016).
29) D. Ulyanov, A. Vedaldi, V. Lempitsky: “Improved Texture
Networks: Maximizing Quality and Diversity in Feed-Forward
Stylization and Texture Synthesis”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 6924–6932 (2017).
30) L. Hörmander, The Analysis of Linear Partial Diﬀerential Operators I: Distribution Theory and Fourier Analysis, Springer(2015).
31) qiqi7788,(2020).

https://github.com/qiqi7788/dataset-and-result32) C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, M. Pantic: “300
Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge”, Proc. of the IEEE International Conference on Computer Vision Workshops (ICCV), pp.3–18 (2016).
33) M. Koestinger, P. Wohlhart, P. M. Roth, H. Bischof: “Annotated Facial Landmarks in the Wild: A Large-Scale, RealWorld Database for Facial Landmark Localization”, Proc.
of 2011 IEEE International Conference on Computer Vision
Workshops (ICCV Workshops), pp.2144–2151 (2011).
34) X. Zhu, D. Ramanan: “Face Detection, Pose Estimation, andLandmark Localization in the Wild”, Proc. of 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.2879–2886 (2012).
35) V. Le, J. Brandt, Z. Lin, L. Bourdev, T. S. Huang: “Interactive
Facial Feature Localization”, Proc. of European Conference on
Computer Vision (ECCV), pp.679–692 (2012).
36) P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman: “Localizing
Parts of Faces Using a Consensus of Exemplars”, IEEE Trans.
on Pattern Analysis and Machine Intelligence, Vol.35, No.12,
pp.2930–2940 (2013).
37) I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M.
Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D. Lee,Y. Zhou, C. Ramaiah, F. X. Feng, R. F. Li, X. J. Wang, D.
Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu,
M. Popescu, C. Grozea, J. Bergstra, J. J. Xie, L. Romaszko,
B. Xu, Z. Chuang, Y. Bengio: “Challenges in Representation
Learning: A Report on Three Machine Learning Contests”,
Proc. of International Conference on Neural Information Processing, pp.117–124 (2013).
38) A. Mollahosseini, B. Hasani, M. J. Salvador, H. Abdollahi,12040) Z. Yi, H. Zhang, P. Tan, M. Gong: “Dualgan: Unsupervised
Dual Learning for Image-to-Image Translation”, Proc. of theIEEE International Conference on Computer Vision (ICCV),
pp.2849–2857 (2017).
41) P. Giannopoulos, I. Perikos, I. Hatzilygeroudis: “Deep Learning Approaches for Facial Emotion Recognition: A Case Studyon FER-2013”, Advances in Hybridization of Intelligent Methods, pp. 1–16 (2017).
42) O. Arriaga, M. Valdenegro-Toro, P. Plöger: “Real-Time Convolutional Neural Networks for Emotion and Gender Classiﬁcation”, Proc. of 27th European Symposium on Artiﬁcial Neural
Networks, pp. 24–26 (2019).
43) K. Simonyan, A. Zisserman: “Very Deep Convolutional Networks for Large-Scale Image Recognition”, Proc. of 3rd International Conference on Learning Representations (ICLR)(2015).
44) S. Minaee, A. Abdolrashidi: “Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network”,
Computing Research Repository, Vol.abs/1902.01019 (2019).
45) J. Shen, S. Zafeiriou, G. G. Chrysos, J. Kossaiﬁ, G. Tzimiropoulos, M. Pantic: “The First Facial Landmark Tracking in-the-Wild Challenge: Benchmark and Results”, Proc. of
the IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 50–58 (2015).

(Received May 29, 2020)(Revised October 7, 2020)(Student Member)
She received the M.E. degree from Beihang
University of Software in 2016. From 2019,
she has been staying at Ph.D. course in System Engineering at Wakayama University.
Her research interests include face detection,
3D face reconstruction, medical image processing, and deep learning.

Haoqi GAO

Koichi OGAWARAHe received his Ph.D. degree in Information
and Communication Engineering from the
University of Tokyo, Japan, in 2002. From2006 to 2011, he was a guest Associate Professor at Kyusyu University. Currently, he is an
Associate Professor at Wakayama University.
His research interests include robotics, computer vision, and machine learning. He won
the Best Vision Paper Award at the IEEE International Conference on Robotics and Automation 2007.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
IIEEJ

Paper

Robust Sphere Detection in Unorganized 3D Point Clouds Using an Eﬃcient
Hough Voting Scheme Based on Sliding Voxels
Jaime SANDOVAL† (Student Member), Kazuma UENISHI† (Member),
Munetoshi IWAKIRI†† (Member), Kiyoshi TANAKA† (Fellow)† Shinshu University , †† National Defense Academy of Japan<Summary> Sphere detection in point clouds is an important task in 3D computer vision with various
applications such as reverse engineering, medical imaging, Terrestrial Laser Scans (TLS) alignment, and soon. So far, several approaches have been proposed to detect spheres in point clouds. However, conventional
methods are ineﬃcient and inaccurate because they depend on random sampling, point-wise voting or
normal vectors estimation to generate hypothetical spheres. To overcome these drawbacks, we propose anovel algorithm that employs sliding voxels and Hough voting to robustly and eﬃciently detect spheres inunorganized point clouds. The proposed method can analyze all the points contained in point clouds without
deteriorating its eﬃciency and accuracy in contrast to conventional methods. Through experiments, we foundthat the proposed method can drastically reduce the processing time and achieve more accurate and robust
performance in severer conditions than conventional methods.
Keywords: sphere detection, unorganized point clouds, sliding voxel, Hough voting, eﬃcient RANSAC

1.

Introduction

Point clouds are sets of points in an R3 space that resemble the surface of objects. They can be obtained bya wide variety of laser-based sensing techniques and photogrammetry. However, because of these diverse ways ofobtaining them, we face several problems such as hugevariations in points density, sensing patterns, sensor artifacts, and noise.
Point clouds can be categorized as organized and unorganized. Organized point clouds can be arranged into
an image-like 2D matrix, in which each pixel is associated with a 3D point. These point clouds come fromrange/depth sensors or stereovision. On the other hand,
unorganized point clouds have no speciﬁed order andare just a list of 3D points, which are a more generalform than the organized counterparts. These come fromsensors that change their coordinate system while scanning, like rotating-head 3D LIDAR sensors or dense point
clouds from photogrammetry. However, because unorganized point clouds have no order, basic techniques such asnearest neighbor searching become increasingly diﬃcultand time-consuming in unorganized point clouds. Therefore, in this paper, we focus on developing algorithms
for unorganized point clouds, which are in high demand121

because of their numerous applications.
Man-made objects can be approximated by geometric primitives such as spheres, planes, and cylinders 1),2).
These geometric primitives can act as proxy entities forother real-world objects, for instance, the trunk of a tree
and human arms and legs can be approximated by cylinders, also, the head of animals and humans can be approximated as spheres.
Sphere detection is an important technique in 3D computer vision with applications in broad areas such as materials engineering 3), measuring 4), medicine 5) among others. As opposed to other parametric shapes such as planes
or other quadric surfaces, spheres have a viewpointindependent geometry. This property allows us to calculate its coeﬃcients with partial views such as those obtained from 3D sensors. Therefore, they are preferred astargets in point clouds registration 6)–8) where it is crucial
to detect their coeﬃcients with the highest accuracy.
However,the conventional sphere detection1),9)–12)
fail to detect spheres when the inmethods
liers ratio becomes too small due to noise, and the rangeof the point cloud.
Moreover, the processing time can increase exponentially as some algorithms depend on point-wise normal
vectors, and their accuracy is highly dependent on theirIIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

estimation. Also, normal vectors are calculated via Principal Component Analysis (PCA), a method known to bevery susceptible to noise and outliers.
Therefore, in this paper, we propose a novel, highly accurate, robust, and drastically more eﬃcient (high speed)
method for sphere detection based on sliding voxels. Instead of random sampling, it uses an eﬃcient octree subdivision to detect robustly hypothetical spheres deterministically. Then, the most prominent spheres are detectedvia Hough voting 13)–15). Lastly, its coeﬃcients are reﬁned and pruned by their completeness. Experimentswith synthetic and real point cloud data from Terrestrial
Laser Scanning (TLS) conﬁrm the superior performanceof the proposed method.
This paper is organized as follows. In section 2, we
brieﬂy give a literature overview and explain the conventional method. Section 3 describes in detail the proposed
method. Section 4 is focused on describing the datasets,
the evaluation method, used metrics, and showing experiments results. In section 5, we summarized the results
and future works.

2.
2.1

Previous Work

Literature overview

Several approaches to detect spheres using spatial information have been developed in recent years. A survey
work 16) summarizes the conventional methods for geometric primitive detection from 3D data. Region growing
clusters similar regions of the point cloud at random locations iteratively. Although computationally expensive, it
can be used to later ﬁt geometric primitives in segmented
regions.
Hough Transform (HT) 13),14) based algorithms forparametric shape detection have been proposed in thepast. As spheres become perfect circles when projected
into a plane, the circular HT was applied to detect spheresas circles in images 17). Unfortunately, unorganized point
clouds do not have a deﬁned projection to an image array. Moreover, when using images to detect spheres, theradius of circles vary depending on the distance from thesensor.
The parameter space for the sphere has four dimensions. Therefore, applying the Standard Hough Transform (SHT) 14) is unfeasible both in computational complexity and in memory usage of the accumulator arrayused to store votes and detect shapes. Ogundana et al. 9)
proposed a fast HT detector by ﬁxing the radius, castinga single vote for each point, and detecting spheres using a

122sparse accumulator. Nonetheless, they require computationally expensive point-wise normals estimation. Also,
ﬁxing the radius limits its applications in real-world scenarios where it is usually unknown.
Abuzaina et al. 10) also proposed a HT for sphere detection using a sparse accumulator. Their approach usesa polar representation of the parameter space and a moreexhaustive per-point voting. To overcome this, they ﬁxed
the radius and reduced the number of input points basedon the points’ density of a Kinect sensor limited at a certain range.
In the past, strategies based on random sampling were
proposed to reduce the computational complexity of thevoting phase in Hough transform methods. The Probabilistic Hough Transform (PHT) 18) reduces the number of evaluated points by selecting a random subset.
The Combined Multi-Point Hough Transform (CMPHT)
11)
follows the PHT approach to reduce computational
complexity, and evaluates several Hough transforms forsphere detection. A single-point HT with coarse accumulator quantization serves as an eﬃcient coarse approximation that identiﬁes regions-of-interest (ROI) where
spheres are more likely to be found. Then, a 4-point
Hough transform is chosen as a detection reﬁnement
over the ROI. Experiments with real point clouds witha Kinect sensor showed that the computational complexity and the success rate of CMPHT are highly aﬀectedby the inliers ratio
Nin
φr =
,
(1)
N
where Nin is the number of inliers of the point cloud andN is the number of total points, which means that lower
φr contains more outliers induced by noise or other nonspherical surfaces.
Reducing the dimensions of the parameter space wasan unexplored approach for primitive detection. A recent
work 12) shows a study of a multi-shape and multi-model
detector based on Point-Pair Features (PPF). However, it
expects an input cloud with normal vectors, and the PPF
is highly dependent on the correct estimation of its normal orientation. Although it outperforms state-of-the-art
primitive detection in their experiments, the dataset theyused were CAD models and point clouds with removed
background and geometric primitives present in the foreground where points are denser and less noisy. Theoretically, if we remove the computational complexity of normal vectors estimation, its processing time is still heavily
dependent on the total number of points since it needs

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

to compute PPFs for every unique combination of pointpairs with normals. Therefore, the authors restricted thenumber of points to compute to 2048 random points,
which is unacceptable for long-range (long distance between the sensor and its farthest point) point clouds suchas those used in TLS or large-scale photogrammetry.
2.2

RANSAC-based methods

RANSAC tries to ﬁt a model into a point cloud byrandom sampling iteratively. The number of trials k isdeﬁned byk=log(1 − z),log(1 − b)(2)where z is the probability that at least one of the data
points is error-free, and b is the probability that any setof selected data points is within the error tolerance tothe model. Finally, it selects the model that has above adeﬁned number of inliers within a threshold distance. Asb can be calculated from the sampling, z is a parameterdeﬁned by the user.
RANSAC and its variations, such as MSAC 26) are implemented alongside geometric models of planes, spheres,and cylinders. However, the PCL implementation follows the original RANSAC 20), which is a single-instanceand single-model ﬁtting algorithm. When detecting morethan one sphere using this approach, we would need toiteratively detect and remove inliers each time we ﬁnd
a good model. Hence, decreasing φr in each successful
detection making it increasingly diﬃcult to detect moreshapes, and to decide ﬁnishing conditions.
Wang et al. combine a RANSAC-like sampling strategy with energy minimization to detect spheres in Kinect
point clouds 21). It starts by drawing a small set of hypo-and the eﬀect of this bias in the ﬁnal model coeﬃcients of
their method is unknown since it was out of the scope of
their study 22). Therefore, this method is not suitable forapplications where the accuracy of the resulting spherecoeﬃcients is crucial 6)–8). Furthermore, selecting an appropriate basis out of a long-range point cloud becomescomplex task and highly depends on the inliers ratio of
the point cloud (φr ) since it inherits RANSAC disadvantages.
2.3 Eﬃcient RANSAC
Eﬃcient RANSAC (EFRANSAC) 1) uses octrees fora more eﬃcient localized sampling. For spheres, it uses
two random sampled points with their normal vectors togenerate hypothetical spheres. Then, iteratively executes
RANSAC over disjoint random subsets of the point cloudto validate the generated model.
Each sphere is reﬁned by thresholding the expected curvature at each point Nth . Also, they map sphere inliers
within a threshold  to a low-distortion bitmap that resembles the surface of the sphere. This bitmap of bin
size C allows EFRANSAC to select the biggest connectedcomponent as the ﬁnal inliers of each hypothetical sphere.
The ﬁnal coeﬃcients are reﬁned using non-linear leastsquares 27).
EFRANSAC ﬁnishing condition takes into account theoctree level of the samples and is parameterized similarly
to RANSAC. At a lower z, these algorithms will increase
their determinism at the cost of increasing their iterations. According to the EFRANSAC implementation in
the CGAL library 25), z is thresholded against
stopp =|Lc |1−
4|P |Odepth

|C|
,

(3)

thetical sphere models from random samples and then use
energy minimization to label spherical points. Although
it is not dependent on distance thresholds, its accuracy

where Lc is the largest candidate size (in number of

depends on the initial sphere models and weighing termsof the energy minimization function, which have to be
guessed by the user depending on the outliers rate.
Spheres are a subtype of quadric surfaces (quadrics);
therefore, they have nine Degrees-of-Freedom (DoF) parameters. A study 22) provides a quadric detector al-

the depth of the octree, and |C| is the number of candidates drawn so far.

gorithm that overcomes current limitations by using
RANSAC to search for sets of points (basis) that are likely
to be on a quadric surface. These bases provide a coarse
identiﬁcation of a quadric surface to be further reﬁned.
Nonetheless, they proposed to ﬁt linearly, a non-linear(quadric) surface. Hence, their ﬁtting results are biased,

123

points), and |P | is the number of available points thatare not part of the selected shape candidates. Odepth is

2.4 Drawbacks of conventional methods
Table 1 lists the conventional methods, their approachfor sphere detection, eﬃciency strategies, and their drawbacks. Conventional methods are ineﬃcient because they
depend on point-wise voting or normal vectors estimation
to generate hypothetical spheres.
Methods based on the Hough transform (and Houghvoting), due to point-wise exhaustive search, they resort to ﬁx the radius of the spheres they can detect 9),

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
Table 1

Summary of conventional methods

MethodDetection approachEﬃciency strategy

Drawbacks

Fast HT 9)

Single-vote HT

Fixing radius, sparse accumulator

Fixed radius

Kinect HT 10)
CMPHT 11)

SHT 14)
PHT 18)

PPF Hough voting 12)

PPF 19)Background points removal
Point-wise search
Coarse-to-ﬁne voting, random subsampling Weak to low φrHigh combinatorial complexity,
Random subsampling (up to 2048 points)
weak to low φrWang et al. 21)

RANSAC 20),energy minimization

Random sampling

Weak to low φrBirdal et al. 22)

RANSAC 20),
quadric voting

Random sampling

Weak to low φr , biased

EFRANSAC 1)

RANSAC 20),energy minimization

Random sampling, octree

Weak to low φr , biased

              

  !"#(  &    

$    %
'

Fig. 1   

Overview of the proposed methodor to limit the points they process 10)–12); thus, diminishing their accuracy and narrowing their applicability.
To avoid doing exhaustive search, RANSAC-based methods 1),21),22) resort to random sampling; making themweak to the inliers ratio φr of point clouds. As noise oroutliers in point clouds increase, these methods fail to obtain valid hypothetical spheres; leading to misdetections.
Moreover, their accuracy highly depends on the correct
estimation of point-wise normal vectors.
Furthermore, 3D LIDAR point clouds possess a longrange, and the detection of relatively small and noninvasive sphere targets 6)–8) is an extremely diﬃcult task
for all the conventional methods proposed so far.
In addition, most of the conventional methods lack ofpublic implementations, only EFRANSAC 1) is implemented in the CGAL Library 25). Therefore, in this work,
we provide comparative assessments with EFRANSAC
and the proposed method.

124

3.

Proposed Method

3.1 Main features and superiority
In order to solve the aforementioned drawbacks of theconventional methods, in this paper, we propose an eﬃcient and deterministic method to detect spheres in unorganized point clouds. The proposed method has two mainfeatures; (i) it employs a 3D space subdivision called sliding voxels that generates hypothetical spheres for Houghvoting without discarding any point. In other words, theproposed method is capable of analyzing the whole point
cloud without resorting to naive random sampling for hypothesis generation. Therefore, the sliding voxel technique contributes to achieving superior accuracy and robustness in sphere detection even in point clouds with low
φr . (ii) Also, the proposed method transforms voxelized
regions of the point cloud into local planes, which eﬃciently reduces the number of computations for Houghvoting. That is, as opposed to conventional subsampling strategies prone to noise and outliers, the proposed
method can achieve highly eﬃcient Hough voting by em-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

3D
space

dow with a stride of 1. We call this structure a sliding
voxel, and it helps us to robustly and eﬃciently identify

Axis
AlignedBounding
Boxspherical-like regions in the point cloud. Then we takeadvantage of this structure to generate a hypotheticalsphere for every sliding voxel in a point cloud.
3.3 Hypothetical spheres generation
Several local sphere ﬁtting methods exists, among
them, algebraic ﬁtting uses linear least squares to ﬁt a

Octreenodes

sphere in a point set. It works by rearranging the sphereequation(x − Cx )2 + (y − Cy )2 + (z − Cz )2 = r2

Fig. 2(4)

to the following
3D space subdivision using an octree

2xCx + 2yCy + 2zCz + α = x2 + y 2 + z 2 ,ploying sliding voxels, which contributes to reducing theentire processing time drastically without deteriorating
its accuracy. Moreover, these superiorities allow us to extend the applicability of the proposed method to the caseof processing a huge amount of point clouds captured by
TLS in real-world situations.
3.2 Process ﬂow
Figure 1 shows a graphical overview of the proposed method. We proposed an eﬃcient octree-based
point cloud subdivision to robustly estimate hypothetical spheres with our novel sphere ﬁtting algorithm. To
globally detect those spheres, we performed Hough votingwith a memory-eﬃcient accumulator based on nested tree
structures. Finally, the spheres are pruned by a completeness score and reﬁtted using the connected componentsof the projecting bitmap.
The proposed method starts by dividing the point cloudspace in a 3D grid using an octree. Figure 2 shows
how the 3D space of the point cloud is enclosed by an
Axis Aligned Bounding Box (AABB) and subdivision occurs recursively until it complies with a voxel size Vs .
The bounding box is expanded accordingly such that theleaves of the octree match the desired voxel size.
In each node of the octree, we save information about
the local geometry of its points: centroid and normal
vector. The normal vector was computed using PCA andcorresponds to the eigenvector associated with the smallest eigenvalue. Therefore, the number of points to process becomes the number of occupied voxels with threeor more points.
For each leaf, we are going to select their 26-neighbors
leaves, such that contiguous cells overlap and share geometrical properties, i.e., a 3D (3 × 3 × 3) sliding win-

125

(5)

where {Cx , Cy , Cz , r} are the parameters of the sphereand α = r2 − Cx2 − Cy2 − Cz2 .
Then, we can obtain the sphere parameters using theleast squares normal equation of its matrix representation
AT Ax = AT b.

(6)

As AT A is symmetric and positive-deﬁnite, it can besolved with Cholesky factorization to get the sphere parameters from x.
A second alternative is to use a non-linear least-squares
approach. The Levenberg-Marquardt(LM) 28) method
uses gradient-based optimization to ﬁnd the sphere coeﬃcients that minimize the error between a sphere modeland a point set. In the proposed method we used theerror function
arg minC,rn

(Pi − C)T (Pi − C) − r2

(7)

i=1

where Pi is a point of a set of n number of points, C isthe center of the sphere and r is the radius. The Jacobian
J of Eq. (8) given a point Pi is
J = [−2(Pix − Cx ), −2(Piy − Cy ), −2(Piz − Cz ), −2r]T(8)where {Pix , Piy , Piz } are the point coordinates.
Since its correct convergence depends on an initial hypothesis, it is not clear which value is best for each case.
When a sliding voxel falls in a planar surface we would
expect to get a sphere with an arbitrarily far center anda big radius. However, both linear and non-linear leastsquares approaches produce unpredictable results on planar point sets and are greatly aﬀected by their numberof outliers.
Therefore, we introduce a novel local ﬁtting algorithm
that provides a robust estimate of the best ﬁt sphere of

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

 
    







Fig. 3 Fig. 4

Sphere ﬁtting with sliding voxels

a point set. Given a set of N points with normal vectors
Ψ of every sliding voxel, for all combinations of its items
N 
2 , we estimate a sphere using the model generation
method described by Schnabel, et al. 1) which uses twopoints and their normal vectors to estimate the parameters of one hypothetical sphere at a time. However, they
do not indicate the speciﬁc method they used to estimate
the sphere center. Therefore, we refer to the appendix
section of this work for details about the estimation of asphere using two oriented points.
By doing this for every pair of points and normals insideeach sliding voxel, we are obtaining a set of all possible
spheres S ∈ R4 . As the sliding voxel has a maximum of
27 occupied voxels, the maximum cardinality of S is 351.
Ideally, if the surface is perfectly spherical, all the spheres
estimated will be the same, and S would have 0-variancein all its four dimensions. If we face with noise or outliers,
its variance will ﬂuctuate and so its mean and median.
To robustly get the most probable sphere, we select themedian sphere Sm ∈ S such thatarg min Si − Sμ 2 , i ∈ [1, |S|](9)iwhere Si ∈ S and |S| is the cardinality of the set S,  · 2
denotes the Euclidean norm, and Sμ is the mean spherein S.
Given the mean sphere Sμ and Sm , we also calculate
the spherical likelihood
δμ = Sm − Sμ 2(10)

If the sliding voxel falls into a perfect sphere, δμ will tend
to 0 and will represent the deviation of the median withrespect to the mean of the set of all possible spheres.
Therefore, δμ is thresholded with the parameter T hδμ to
avoid regions that do not possess a spherical geometry.
Figure 3 shows how a portion of the point cloud in thesliding voxel can deﬁne the underlying spherical geometryby using our ﬁtting method.
126
 Spheres accumulator structure

3.4 Hypothesis veriﬁcation
Once all our hypothetical spheres are estimated, we
make them converge into a Hough accumulator, since
the spheres accumulator has four dimensions, we chosea memory-eﬃcient nested tree structure shown in Fig. 4.
It is common in the literature to ﬁnd the accumulator discretization deﬁned by the number of bins of each dimension. However, since all sphere dimensions are expressedusing the same metric, we can parameterize it using theaccumulator bin size Accres instead of the bins number.
After the voting ﬁnished, the most prominent spheresare extracted by accumulating the votes and performing
peak detection in a 3 × 3 × 3 × 3 sliding window over theaccumulator. A sphere will only be extracted if it has aminimum number of votes Vmin .
As a result of the discretization, noise and outliers, theextracted spheres are not the ﬁnal spheres but an approximation of what they should be. Therefore, we have to
reﬁt and prune them, as shown in Fig. 1. Voting took
place with sliding voxel centroids, but reﬁtting will takeplace with the actual points. Therefore, we start to search
for inliers and reﬁt iteratively until the sphere coeﬃcientschanges are negligible.
To avoid the negative inﬂuence of outliers, we map theinlier points of the reﬁtting into a planePz
1x = arccosπP − C2(11)
1
1
atan2(Py , Px ) +y=2π2such that it results in a squared points distribution. Thena bitmap is generated by deciding the number of the discretization bins Bbins . From this bitmap, we perform twothings:To select the biggest cluster using 8-neighbors clustering.
To estimate a completeness measure.
After getting the biggest cluster, completeness K is

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）    













 



(a) Point cloud render 
  



Fig. 5Synthetic dataset M

measured by the ratio

Fig. 6

SO
K=
,
ST

(12)

where SO is the number of occupied pixels, and ST is thetotal number of pixels. Although there is no mapping
between a sphere and a plane without deformations, we
ﬁnd this ratio to be a close approximation of the spherecompleteness. Particularly in the visible quadrants, deformations are negligible.
This process of reﬁtting and pruning is applied for every sphere extracted from the accumulator. After all theextracted spheres are processed, the algorithm ﬁnishes.

4.

Experiments and Discussion

4.1 Datasets
We evaluated EFRANSAC and the proposed methodwith experiments against synthetic and real data.
The synthetic dataset M, shown in Fig. 5, was generated using sensor simulation in Blensor 1.0.18-RC1029)
and a model with six spheres, among other objects
recreating realistic scenarios of occlusion and loss of thespheres’ surface. Although the simulation provides organized point clouds, we treat them as unorganized. In thelower part, we can observe a render of the model used
to simulate a realistic time-of-ﬂight (TOF) sensor point
clouds. The three spheres on the back have the same radius, but we varied their levels of potential visibility fromthe sensor: 100%, 50% and 25%. Furthermore, as we areaiming to simulate a real sensor as close as possible, we

127(b) Ground truth spheres
A point cloud example from the M dataset

set the parameters of the sensor simulation software tomatch those of the KinectV2.
The simulations occurred at the same sensor position but with variations in the level of noise, with
a 0-mean Gaussian noise, and its variance σ set to[0.004, 0.008, 0.012, . . . , 0.04]. These variations are used
to modify the coordinates of the sensed points. Since it
is a KinectV2 simulation its number of points is 217,088,
its Bounding Box Diagonal Distance (BBDD) is around9.00[m] and its resolution is about 0.014[m]. Where
BBDD is the vector length of the extreme points of thebounding box.
Figure 6 shows an example of a point cloud projectedfrom M that corresponds to the lowest level of Gaussian noise (σ = 0.004). Note that the size of objects appear to be diﬀerent from Fig. 5 because their projection
method and camera position are diﬀerent. Their index isshown in the numbers displayed above the spheres points
in Fig. 6(a), and the ground truth spheres are shown separately in green as 3D models in Fig. 6(b).
The NDAJ dataset N consists of 34 point clouds obtained from a FARO 3D LiDAR scanner, they arehighly dense and cover a broad area.
The scanning target is a building of the Engineering campus of the National Defense Academy of Japan(NDAJ). Figure 7 shows a render of the registration of
all the point clouds of the dataset. This render was gen-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）(a) Nearest sphere: 6.15[m], (b) Farthest
4,958 points

Fig. 9  31[m],

Example of sensed spheres and their respective
ground truth 3D models of the R4 (LIDAR) pointcloud
Table 2
Dataset

Fig. 7sphere:169 points

Rendering of all subsets of the N dataset

Datasets information

Points[#]BBDD[m]CR[cm]

ML

217,088

8.77

1.00

MM

217,088

8.88

1.24

MH

217,088

8.99

1.47

NG

25,543,762.73 ±3,625,816.05 299.70±60.27 0.44±0.07

NR

18,261,275.62 ±2,615,343.03 344.15±18.81 0.30±0.04

truth data spheres of the real dataset are highly precise
since it was used to register the point clouds by manuallylabeling the spheres in the FARO Scene point cloud
processing software.
The location of each categorized scan can be seen in
Fig. 8. 26 of the point clouds were taken from groundlevel, numbered from G1 to G26 , and marked with redboxes in Fig. 8. Eight point clouds were taken fromthe roof of the building, numbered from R1 to R8 , andFig. 8marked with blue boxes in Fig. 8.
The ground truth spheres of N have ﬁxed positions,but depending on the distance from the sensor, the density of their points and their noise can vary enormously.
Figure 9 shows the eﬀect of a sphere sensed from 6.15[m]and 31[m], there is a big diﬀerence in the density of
their points when they are near or far from the sensor.
Therefore, a challenging feature of these long-range point

Sensor locations of the N dataset

erated by processing all the registered point clouds witha voxel size of 0.05[m] and applying normals estimation
with a radius of 0.075[m], then we activated the EDL
shader from the qEDL plugin in Cloud Compare.
Figure 7 also shows a close up of a point cloud of N
(G2 ) and its location inside the rendering. Three spheretargets that were physically placed near the sensor are
shown alongside its ground truth spheres shown in green.
These are part of the ground truth spheres set. Additionally, sensor artifacts produced by moving objects inthe scene are shown.
The ground truth spheres were generated by using thescanner’s software in a user-guided process. The ground128

clouds, is their variations in density, which makes it difﬁcult to validate hypothetical spheres by only counting
their inliers.
In Table 2, we summarize statistical information ofboth datasets. The synthetic dataset consists of 10 point
clouds that resemble the ones gathered from low-cost sensors. The table shows the data from the synthetic clouds
with low (ML ), mid (MM ), and high (MH ) levels ofGaussian noise when μ = 0 and σ become [0.004, 0.02,
0.04], respectively. Also, we divided the real point clouds
N dataset into two parts, ground (NG ) and roof (NR ),which separates the point clouds that were scanned at

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

the ground level and over the building. CR stands for

4.3

Cloud Resolution, which is a measure of the average den-

Both algorithms implementations are in C++, com-

sity of the point cloud and is deﬁned as the mean distance

piled with gcc 7.5 and O3 optimizations. We used the

between each point and its nearest neighbor.

EFRANSAC implementation from the CGAL library 4.11
25)
, while the proposed method was implemented mostly
using routines from the PCL Library 1.9.1 24). Since
EFRANSAC is nondeterministic, we executed the experiments 50 times and measured both mean and standard
deviation.
We adjusted the parameters of the evaluated methods
based on preliminary experiments to get the most accurate results, following to adjustments in improving processing time without deteriorating their accuracy. Table 3 and Table 4 show EFRANSAC and the proposed method parameters, respectively. For each dataset,
we provide results of both visual and numerical assessment. For the synthetic datasets, we calculated η foreach level of noise when using EFRANSAC and the proposed method. Figure 10 (a) shows the accuracy of the

4.2 Experimental setup
To evaluate the accuracy, we performed matching of theground truth spheres Sgt and the detected spheres Sdet
coeﬃcients. A match occurs if a sphere in Sdet is foundin Sgt by thresholding its Euclidean distance in R4 ,
Sdeti − Sgtj  < 0.1[m].
(13)This threshold provides an average maximum error tolerance of 0.025[m] for the four parameters of a spheremodel. To decide this threshold, we ran EFRANSAC on
the noisiest point cloud of M and adjusted it such thata visible bias coming from noise, and aﬀecting the position and radius of spheres do not alter the true positives
count T P when most of its inliers are close to the sphere(shown later in visual assessment).
All the detected spheres that satisfy Eq. (13) are T P ,then we can deﬁne the following metrics from information
retrieval 30) precision γ, recall ζ, and F1 score η,
TP
#Sdet
TP
ζ=
#Sgt
2T P
η=
#Sdet + #Sgt

γ=

Experiments results and discussion

Table 3
Name



C

EFRANSAC parameters
Nth

Min.
supportz

Normals radius

ML

0.016 0.80 0.95 700

0.00010

0.04

(14)

MM

0.020 0.80 0.95 1100

0.00005

0.14

(15)

MH
NG
NR

0.040 0.80 0.95 1100
0.020 0.02 0.10 200
0.020 0.02 0.10 200

0.00005
0.00005
0.00005

0.20
0.05
0.05

(16)

where # denotes set cardinality. Precision will max at 1
when we detected only correct spheres, while recall will
max at 1 when all the ground truth spheres were detected.
η is the harmonic mean between γ and ζ.

Table 4

Proposed method parameters

Name

Vs

Vmin Accres



T hδμ

K

Bbins

ML

0.16

20

0.05

0.020

0.4

0.1

50

MM
MH
NG

0.16
0.18
0.05

20
30
50

0.05
0.05
0.01

0.020
0.020
0.002

0.4
0.4
0.3

0.1
0.1
0.1

50
50
25

NR

0.05

50

0.01

0.002

0.3

0.1

25

(a) η experiment results(b) Processing time[s] experiment results
Fig. 10 η and processing time experiment results of the M dataset

129

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
Table 5

Detailed results per ground truth sphere of experiments with synthetic data of M.

Mean error in R4 [m]Mean error in R4 [m]Detection rate[%]Detection rate[%](EFRANSAC)(proposed)(EFRANSAC)(proposed)

12
30.11050.01960.01550.03210.02330.0173

89.8
100.0
100.0

100.0
100.0
100.0

Small radius (0.2[m])NearestOccluded

4
50.02190.03890.01380.0172

99.8
98.0

100.0
100.0

100% Visibility
50% Visibility

60.08860.0292

73.2

100.0

25% Visibility

Sphereevaluated methods. A value of one means the proposed
method detected all the ground truth spheres without
false positives, η goes down when either false positives orfalse negatives drop. The EFRANSAC accuracy drops
as the noise worsen, even when we increased its iterations and the radius of normal vectors. On the other
hand, the proposed method detected successfully all thespheres with high accuracy. Figure 10 (b) is showing thatthe proposed method is drastically more eﬃcient than
EFRANSAC, which is requiring more processing power
as noise increases due to the necessity of expanding the
support radius of normal vectors estimation and its iterations.
Table 5 shows the results of per-ground truth spherein the dataset M. The index of the sphere is shown in theleft-most column. The two following columns show theirmean error with respect to their closest match of detected
spheres in R4 by matching the ground truth spheres withthe one that minimizes the left term of Eq. (13). Also, wehighlighted in bold the ones that showed the best results.
EFRANSAC has a slightly better detection error thanthe proposed method only when its detection rate is competitive. Hence, in this step, EFRANSAC is ﬁltering
points that do not agree with the normal vector of the
hypothetical model. However, the heavy computationalcosts associated with computing point-wise normal vectors (see Fig. 10(b)) are not worth a few millimeters ofbetter precision.
On the other hand, in the detection rate of Table 5,
we can notice that EFRANSAC fails to detect the spherewith the least visibility. This happens because the normal
vectors near the border of the plane wall and the sphere
are imprecise. Hence, further reducing the available surface to detect the sphere number six.
Figure 11 shows the η results of the evaluated methods for each of the point cloud groups of the N dataset.
The y-axis of the plot is divided into two because of thepoor results of EFRANSAC since it had drastically lower

130

Fig. 11Note

η results using the N dataset, higher is better

precision and negatively aﬀected its η. EFRANSAC validates a model by thresholding its inliers count without
considering their geometry and completeness. Therefore,
the inliers ratio φr of those point clouds is quite low, andnear the machine epsilon, it is obvious that a methodthat relies only on random sampling like EFRANSAC is
failing very badly in the results.
Moreover, a sphere of the dataset usually contains a few
hundred of points while the whole point cloud accounts
for dozens of millions. This situation forces EFRANSAC
to set a lower value of inliers count that supports asphere candidate, thus detecting numerous false positives
and diminishing its precision. The standard approach toaddress this issue is to increase RANSAC iterations bymodifying its probability parameter described in Eq. (2).
However, this value becomes dangerously near the machine epsilon for single ﬂoat precision and thus numerically unstable. Furthermore, its performance decreases
in several orders of magnitude.
On the other hand, the proposed method opts for asmarter strategy by robustly measuring the likelihood ofregions to be spherical, converging hypothetical spheresfrom highly spherical regions into an accumulator andﬁltering by their completeness. It demonstrated superior
accuracy even in massive point clouds with very low φr .

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

We roughly estimated that this ratio for the N dataset isthan EFRANSAC.

less than 0.00005.

Figure 13, Fig. 14, and Fig. 15 show the detailed

Figure 12 illustrates the processing time of the evalu-

results of EFRANSAC and the proposed method against

ation methods for each point cloud of the N dataset. The
y-axis is divided into two sections with diﬀerent scales tovisualize the processing time results. For better visualization, and unlike the experiments with synthetic data, weare not including the normals vectors estimation time inthese results. If we included them, it would be extremely
diﬃcult to visualize and compare the high eﬃciency ofthe proposed method, since it is astronomically better

the N dataset. Figure 13 shows the proposed method has
superior accuracy due to the low precision of EFRANSAC
shown in Fig. 14 because EFRANSAC detected numerous
false positives in the N dataset. Figure 15 is showing how
many of the ground truth spheres were detected without
considering false positives. The proposed method showeda superior recall in all scenarios. Noticeably, at the markA in Fig. 15, EFRANSAC recall was extremely low,
and the proposed method recall was remarkably high.
EFRANSAC failed because the ground truth spheres ofthe point cloud G11 are relatively far from the sensor andhas a lower density, all the spherical points account for
0.0024% of the points, and a ground truth sphere waspartially occluded.
At the mark B in Fig. 15, EFRANSAC recall was very
close to the proposed method. Figure 16(a) shows theR3 point cloud from N . The green triangle represents

Fig. 12

Processing time[s] evaluation results using the N
dataset, lower is betterthe sensor location, and the blue circles the ground truth
spheres. One of the ground truth spheres is extremely
far from the sensor, its distance is represented by the reddashed line and is about 38.33[m]. Both methods failed

F1 score (η, log scale)

EFRANSAC

Proposed

1.00
0.30
0.10
0.01

G1 G2 G3 G4 G5 G6 G7 G8 G9 G10G11G12G13G14G15G16G17G18G19G20G21G22G23G24G25G26 R1 R2 R3 R4 R5 R6 R7 R8Datasets

Fig. 13

F1 score (η) results using the N dataset, higher is better

Precision(γ, log scale)

EFRANSAC

Proposed

1.00
0.30
0.10
0.01

G1 G2 G3 G4 G5 G6 G7 G8 G9 G10G11G12G13G14G15G16G17G18G19G20G21G22G23G24G25G26 R1 R2 R3 R4 R5 R6 R7 R8Datasets

Fig. 14

Precision (γ) results using the N dataset, higher is better

131

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
EFRANSAC

Proposed

1.0

Recall(ζ)

0.8
0.6
0.4
0.2
0.0

G1 G2 G3 G4 G5 G6 G7 G8 G9 G10G11G12G13G14G15G16G17G18G19G20G21G22G23G24G25G26 R1 R2 R3 R4 R5 R6 R7 R8Datasets
A○

Recall (ζ) results using the N dataset, higher is better    

Fig. 15B○



R3 point cloud from the N dataset

to detect this sphere since it is barely recognizable due toits extremely low density.
We also show a visual assessment of the experiment results. In Fig. 17, we can observe the results of executing

    

Fig. 17Fig. 16

Graphical comparison: M dataset

0.0381[m], neither the proposed method nor EFRANSAC
were able to detect. Figure 18(a) shows the detected
spheres of both the proposed method and EFRANSAC.
Due to the numerous false positives and for better visualization, we omitted EFRANSAC spheres larger than
0.3[m]. Even though the extra ﬁltering we can notice
EFRANSAC failed to detect one target sphere and detected several false positives in the background.

5.

Conclusions and Future Worksthe evaluated methods on the synthetic point cloud withthe least (σ = 0.004) and most noise (σ = 0.04). We
assigned a diﬀerent color to each detected sphere in theorder they are given by the evaluated methods. The proposed method detected the spheres ﬂawlessly with highaccuracy. EFRANSAC results are very good with low
noise scenarios, but as we test for higher levels of noise thenumber of misdetections and false positives arise. Moreover, as we need to increase thresholds and the support

Sphere detection is a core technique in point cloud processing with applications in computer vision, reconstruction, modeling, among others. However, existing algorithms work with various drawbacks such as ﬁxed radius,
low eﬃciency, and poor accuracy in noisy data with a numerous amount of outliers. To solve these problems, inthis paper, we proposed a new sphere detection method
based on sliding voxels and Hough voting.
Through experiments we found that the proposed

radius for normals estimation, the smallest spheres tend
to be undetected, and bigger errors in the spheres coeﬃcients become visible.
Figure 18 shows the ground truth and results ofspheres detection of the point cloud G1 . Three groundtruth spheres are not visible in Fig. 18(a), which wereplaced in the central part of the building with a radius of

method achieved 50 times faster processing time and is
1.08 times more accurate than EFRANSAC for the synthetic dataset M. In the real 3D LIDAR dataset N ,the proposed method achieved 31 times faster processing time and is 183 times more accurate (F-score) than
EFRANSAC without including normals estimation processing time. This is because the proposed method can132

IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
thotropic Material Models for Pvc-Coated Polyester via Inverse
Bubble Inﬂation”, Structural and Multidisciplinary Optimization, Vol. 54, No. 4, pp. 927–935 (2016).
4) B. Muralikrishnan, P. Rachakonda, V. Lee, M. Shilling,
D. Sawyer, G. Cheok, L. Cournoyer: “Relative Range Error Evaluation of Terrestrial Laser Scanners Using a Plate, aSphere, and a Novel Dual-Sphere-Plate Target”, Measurement,
Vol. 111, pp. 60–68 (2017).
5) M. van der Glas, F.M. Vos, C.P. Botha, A.M. Vossepoel: “Determination of Position and Radius of Ball Joints”, Proc. ofMedical Imaging: Image Processing, pp. 1571–1577 (2002).

(a) Ground truth spheres render of G1  6) M. Franaszek, G. S. Cheok, C. Witzgall: “Fast Automatic
Registration of Range Images from 3D Imaging Systems Using
Sphere Targets”, Automation in Construction, Vol. 18, No. 3,
pp. 265–274 (2009).
7) Y. Wang, H. Shi, Y. Zhang, D. Zhang: “Automatic Registration of Laser Point Cloud Using Precisely Located Sphere Targets”, Journal of applied remote sensing, Vol. 8, No. 1, pp. 1–14
(2014).
 8) D. Yun, S. Kim, H. Heo, K.H. Ko: “Automated Registrationof Multi-View Point Clouds Using Sphere Targets”, Advanced
Engineering Informatics, Vol. 29, No. 4, pp. 930–939 (2015).

 

9) T. Ogundana, C.R. Coggrave, R. Burguete, J.M. Huntley:“Fast Hough Transform for Automated Detection of Spheresin Three-Dimensional Point Clouds”, Optical Engineering,Vol. 46, No. 5,pp. 1–11 (2007).
10) A. Abuzaina, M.S. Nixon, J.N. Carter: “Sphere Detection in
Kinect Point Clouds via the 3D Hough Transform”, International Conference on Computer Analysis of Images and Patterns, pp. 290–297 (2013).

 11) M. Camurri, R. Vezzani, R. Cucchiara: “3D Hough Transformfor Sphere Recognition on Point Clouds”, Machine Vision andApplications, Vol. 25, No. 7, pp. 1877–1891 (2014).

 

Fig. 18(b) Detection results
Graphical comparison: N dataset (G1 )analyze all the points by employing sliding voxels evenfor big point clouds with numerous amount of points.
As our future works, we should investigate adaptive settings of the voxel size for various density levels in massive
point clouds to further improve its accuracy. Also, we12) C. Sommer, Y. Sun, E. Bylow, D. Cremers: “PrimiTect:
Fast Continuous Hough Voting for Primitive Detection”, Proc.
of the International Conference of Robotics and Automation(ICRA), arXiv preprint arXiv:2005.07457 (2020).
13) P.VC. Hough: “Method and Means for Recognizing Complex
Patterns”, US Patent 3,069,654 (1962).
14) D.H. Ballard: “Generalizing the Hough Transform to Detect
Arbitrary Shapes”, Readings in Computer Vision, pp. 714–725(1987).

need more comprehensive experiments to analyze the relationship among processing time, total number of points
and size of the spheres to give us better insight on theperformance and scalability of the sphere detection methods. Furthermore, as promising extensions of this workfor real-world applications, we are considering to detect
nearly spherical shapes, such as human heads, bone junction, and so on.

15) J. Illingworth, J. Kittler: “A Survey of the Hough Transform”,
Computer Vision, Graphics, and Image Processing, Vol. 44,
No. 1, pp. 87–116 (1988).

References18) N. Kiryati, Y. Eldar, A.M. Bruckstein: “A Probabilistic Hough
Transform”, Pattern Recognition, Vol. 24, No. 4, pp. 303–316(1991).

1) R. Schnabel, R. Wahl, R. Klein: “Eﬃcient RANSAC for PointCloud Shape Detection”, Proc. of Computer Graphics Forum,Vol. 26, No. 2, pp. 214–226 (2007).
2) J. Chen, H. Lai, C. Lin: “Point Cloud Modeling Using Algebraic Template”, International Journal of Innovative Computing, Information and Control, Vol. 7, No. 4, pp. 1521–1532(2011).
3) C.F. Jekel, G. Venter, M.P.Venter: “Obtaining Non-Linear Or-

13316) A. Kaiser, J.A. Ybanez Zepeda, T. Boubekeur: “A Survey of
Simple Geometric Primitives Detection Methods for Captured
3D Data”, Computer Graphics Forum, Vol. 38, No. 1, pp. 167–
196 (2019).
17) M. Kharbat, N. Aouf, A. Tsourdos, B. White: “Sphere Detection and Tracking for a Space Capturing Operation”, Proc.
of the IEEE Conference on Advanced Video and Signal Based
Surveillance, pp. 182–187 (2007).

19) B. Drost, M. Ulrich, N. Navab, S. Ilic: “Model Globally, Match
Locally: Eﬃcient and Robust 3D Object Recognition”, Proc.
of the IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, pp. 998–1005 (2010).
20) M. Fischler, R. Bolles: “Random Sample Consensus: A
Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography”, Communications of theIIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）
ACM, Vol. 24, No. 6, pp. 381–395 (1981).
21) L. Wang, C. Shen, F. Duan, K. Lu: “Energy-Based Automatic Recognition of Multiple Spheres in Three-Dimensional
Point Cloud”, Pattern Recognition Letters, Vol. 83, pp. 287–
293 (2016).
22) T. Birdal, B. Busam, N. Navab, S. Ilic, P. Sturm: “A Minimalist Approach to Type-Agnostic Detection of Quadrics in Point
Clouds”, Proc. of the IEEE Conference on Computer Vision
and Pattern Recognition, pp. 3530–3540 (2018).
23) G. Roth, M.D. Levine: “Extracting Geometric Primitives”
CVGIP: Image Understanding, Vol. 58, No. 1, pp. 1–22 (1993).
24) R. Rusu, S. Cousins: “3D is Here: Point Cloud Library(PCL)”, Proc. of IEEE International Conference on Robotics
and Automation (ICRA), pp. 1–4 (2011).
25) CGAL: Computational Geometry Algorithms Library, https://www.cgal.org (2019).
26) P.H. Torr, A. Zisserman: “MLESAC: A New Robust Estimatorwith Application to Estimating Image Geometry”, Computer
Vision and Image Understanding, Vol. 78, No. 1, pp. 138–156(2000).
27) C.M. Shakarji: “Least-Squares Fitting Algorithms of the NIST
Algorithm Testing System”, Journal of Research of the National Institute of Standards and Technology, Vol. 103, No. 6,
pp. 633–641 (1998).
28) D. Marquardt: “An Algorithm for Least-Squares Estimationof Nonlinear Parameters”, Journal of the Society for Industrial
and Applied Mathematics, Vol. 11, No. 2, pp. 431–441 (1963).
29) M. Gschwandtner, R. Kwitt, A. Uhl, W. Pree: “BlenSor:
Blender Sensor Simulation Toolbox”, Proc. of the International
Symposium on Visual Computing, pp. 199–208 (2011).
30) C. Manning, P. Raghavan, H. Schütze, Introduction to Information Retrieval, Natural Language Engineering, Cambridge
University Press (2010).
31) Distance Between 3D Lines and Segments://geomalgorithms.com/a07- distance.html (2020).

Appendix A

http:Sphere estimation from twooriented points

It starts by detecting the shortest segment between twolines deﬁned by two oriented points P1 and P2 using Sunday’s approach 31). Figure A. 1 illustrates how the center C = {Cx , Cy , Cz } of the sphere can be calculated fromthe line segments L1 = P1 nˆ1 and L2 = P2 nˆ2 . C is the
midpoint between the shortest line segment L∗ = E1 E2
between L1 and L2 . L∗ is the shortest line segment if andonly if it is orthogonal to L1 and L2 simultaneously. To
estimate the endpoints E1 and E2 , we need to parameterize L1 and L2 as follows,
E1 (s) = P1 + sn̂1 , where s ∈ R,
E2 (t) = P2 + tn̂2 , where t ∈ R.

(A–1)

Then, we need to estimate s∗ and t∗ such that the resulting line segmentL∗ (s∗ , t∗ ) = E1 (s∗ ) − E2 (t∗ )L∗ (s∗ , t∗ ) = P1 − P2 + s∗ n̂1 − t∗ n̂2(A–2)

is parallel to both L1 and L2 .
The angle between L∗ and {L1 , L2 } can be derived fromthe equationsn̂1 · L∗ (s∗ , t∗ ) = n̂1 2 L∗ (s∗ , t∗ )2 cos θ andn̂2 · L∗ (s∗ , t∗ ) = n̂2 2 L∗ (s∗ , t∗ )2 cos φ.

(A–3)

Since we need to ﬁx θ = φ = 90[deg], then substitutingcos (90[deg]) = 0 in Eq. (A–3) results in the following
equationsn̂1 · L∗ (s∗ , t∗ ) = 0,

(A–4)

n̂2 · L∗ (s∗ , t∗ ) = 0.

(A–5)

In other words, given the parameterized points E1 (s) andE2 (t) from Eq. (A–1), we need to ﬁnd s and t such thatthe conditions of Eq. (A–4) and Eq. (A–5) apply simultaneously and will name them s∗ and t∗ .
By substituting Eq. (A–1) in Eq. (A–4) and Eq. (A–5),
we gets(n̂1 · n̂1 ) − t(n̂1 · n̂2 ) = −n̂1 (P1 − P2 ) and(A–6)

s(n̂1 · n̂2 ) − t(n̂2 · n̂2 ) = −n̂2 (P1 − P2 ).

(A–7)

For the sake of simplicity we are going to apply thefollowing substitutions to Eq. (A–6) and Eq. (A–7)
a = n̂1 · n̂1
b = n̂1 · n̂2c = n̂2 · n̂2(A–8)

d = n̂1 (P1 − P2 )e = n̂2 (P1 − P2 ).
Figure A. 1

Sphere estimated with two points and theirnormal vectors

134

Then Eq. (A–6) and Eq. (A–7) becomesa − tb = −d(A–9)

sb − tc = −e.

(A–10)IIEEJ Transactions on Image Electronics and Visual Computing Vol.8 No.2（2020）

To solve s and t we apply Cramer’s rule and obtain
cd − beb2 − acdb − aet= 2.
b − acs=(A–11)(A–12)

Noticeably, if n̂1 and n̂2 are nearly parallel, the denominator b2 − ac tends to 0, and the solution is undeﬁned.
In this case, we ignore it if b2 − ac is close to the machine
epsilon.
After we estimate appropriate values of s and t, thecenter of the sphere C isC=

1
(E1 + E2 ),2(A–13)and its radiusr=1(P1 − C2 + P2 − C2 ).
2

Munetoshi IWAKIRI (Member)He received his B. E. degree in Computer Science in 1993, and received his M. E. degree inMathematics and Computer Science from the
National Defense Academy of Japan in 1998.
In 1999, he joined the Department of Computer Science, National Defense Academy ofJapan, as a Research Associate. In 2002, hereceived Dr. Eng. degree from Keio University, Tokyo, Japan. In 2005 he became Lecturer and in 2015 he became Associate Professor in the same institution. He is pursuing research related to Multimedia Processing andInformation Security. He is a member of theInformation Processing Society of Japan.

(A–14)(Received May 8, 2020)(Revised June 20, 2020)

Jaime SANDOVAL(Student Member)He received his B.E. degree in Computer Systems Engineering from the Universidad delValle del Fuerte (Mexico) in 2009, and his
M.E. degree in Electrical and Electronic Engineering from Shinshu University in 2017.
Currently, he is a Ph.D. student in the Interdisciplinary Graduate School of Science andTechnology of Shinshu University with a major in Systems Development Engineering. His
research interests are 3D Point Clouds Processing, Computer Vision and Image Processing.

Kazuma UENISHI (Member)He received his B.E. degree in Computer Science in 2008, and his M.E. degree in Mathematics and Computer Science in 2014 fromthe National Defense Academy of Japan.
Currently, he is enrolled as a Ph.D. student
in the Interdisciplinary Graduate School of
Science and Technology of Shinshu University
with a major in Systems Development Engineering. He is pursuing research in 3D Point
Clouds Processing.

135

Kiyoshi TANAKA (Fellow)
He received his B.S and M.S. degrees in Electrical Engineering and Operations Researchfrom the National Defense Academy, Yokosuka, Japan, in 1984 and 1989, respectively.
In 1992, he received the Dr. Eng. degreefrom Keio University, Tokyo, Japan. In 1995,
he joined the Department of Electrical andElectronic Engineering, Faculty of Engineering, Shinshu University, Nagano, Japan, andcurrently, he is a full professor in the academic assembly (Institute of Engineering) of
Shinshu University. He is the Vice-Presidentof Shinshu University as well as the director of the Center for Global Education and
Collaboration of Shinshu University. His research interests include image and video processing, 3D point cloud processing, information hiding, human visual perception, evolutionary computation, multi-objective optimization, smart grid, and their applications.
Currently, he is the president of IIEEJ, a fellow of IIEEJ, a member of IEEE, IEICE,IPSJ, JSEC, and so on.

Call for Papers
Special Issue on
Image-related Technology for Realizing Immersive MediaIIEEJ Editorial Committee
Research on immersive media, which is an advanced system with such as all-sky video (360-degree video), VR,
AR, and MR, is being actively conducted. In addition, the development of a head-mounted display (HMD) and thestudy of specific user interfaces are underway to fully demonstrate the appeal and the benefit of immersive media.
In ISO / IEC, Coded Representation of Immersive Media (MPEG-I) has been studied for realizing immersive media
in MPEG team, and though its standardization is being completed, further technologies are expected to be
standardized. ITU-T SG16 is also promoting the recommendation of ultra-high presence live experience (ILE:
Immersive Live Experience), and the momentum for systematization is increasing.
In this special issue, we are looking for a wide range of papers on elemental technologies that realize immersive
media and system development papers on research to realize immersive high-presence systems by applying thesetechnologies.
1. Topics covered include but not limited toAll-sky video (360-degree video), Free viewpoint Image, Point cloud, Light field, Holography, Head-mounted
display (HMD), User interface, User experience, Usability, Interaction, Immersive content, VR, AR, MR, SR,
xR, CG, Image processing, Image coding, Computer vision, Deep learning, MPEG-I, ILE
2. Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary contributed
paper. If the number of accepted papers is less than the minimum number for the special issue, the acceptancepaper will be published as an ordinary contributed paper. We ask for your understanding and cooperation.
3. Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.9, No.2 (December 2021)
4. Submission Deadline
Monday, May 31, 2021
5. Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org
6. Online Submission URL: http://www.editorialmanager.com/iieej/

136

Call for Papers
Special Issue on CG & Image Processing Technologies
Supporting and Expanding Human Creativities
IIEEJ Editorial Committee
The AI technology is expected to become a key technology for solving social problems and SDGs
(Sustainable Development Goals), such as declining birthrate, aging population, shortage of labor,
depopulation area, and so on, to which mature society, especially Japan is facing. The application scope ofthe technology in the field of image processing is expanding beyond the image generation and object
recognition to include areas related to creativity, such as attribute conditioned image generation,
super-resolution, image colorization, and line art coloring.
On the other hand, deep learning techniques don’t always have good nature at interpretability andexplainability of results required for creativity processing. Also, there is still issues to be addressed in AI
technology, such as the ineffectiveness to the human-computer interface field by a simple application. So,
technologies to support and expand the human creativity are gathering much concern, and quite promising inwide variety of applications.
This special issue of the paper targets various image-related technologies that support and expand
creativities, and calls for papers and system development papers that cover not only applications of deeplearning but also other technologies.
1.

Topics covered include but not limited to
Image Processing, Image Recognition, Image Detection, Pattern Recognition,
Computer Graphics, Visualization, Binocular Vision, 3D image processing
Computer Vision, Big Data, Image Data Bases,
Machine Learning, Deep Learning, Understandability, Explainability,
Creativity, Usability, Interpretability, Human Interface and Interaction, User Experience, Ubiquitous,
Other related fundamental / application / systemized technologies.

2.

Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary
contributed paper. If the number of accepted papers is less than the minimum number for the special
issue, the acceptance paper will be published as an ordinary contributed paper. We ask for yourunderstanding and cooperation.

3.

Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.10, No.1 (June 2022)

4.

Submission Deadline:

Tuesday, November 30, 2021
5.

Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org

6.

Online Submission URL: http://www.editorialmanager.com/iieej/

137

First Call for Papers
The 7th IIEEJ International Conference on
Image Electronics and Visual Computing 2021
(IEVC2021)
Shiretoko (Shari), Hokkaido, Japan / Sept. 8-11, 2021

https://www.iieej.org/en/ievc2021/
Purpose:The International Conference on Image Electronics and Visual Computing
2021 (IEVC2021) will be held in Shiretoko (Shari), Hokkaido, Japan, on Sept.
8-11, 2021, as the 7th international academic event of the Institute of
Image Electronics Engineers of Japan (IIEEJ) based on the great success of
IEVC2007 held in Cairns, Australia, IEVC2010 held in Nice, France, IEVC2012
held in Kuching, Malaysia, IEVC2014 held in Koh Samui, Thailand, IEVC2017
in Danang, Vietnam, and IEVC2019 held in Bali, Indonesia. The conference
aims to bring together researchers, engineers, developers, and students
from various fields in both academia and industry for discussing the latest
researches, standards, developments, implementations and application
systems in all areas of image electronics and visual computing.

Topics:
The conference will cover a broad set of research topics including, but not limited to, the following: Image and video coding Mobile image communication
 Image analysis and recognition
 Hardware and software implementation
 Segmentation and classification
 Modeling Computer vision
 Rendering Image restoration
 Visualization
 3D image processing Animation
 Object detection
 Interaction
 Motion analysis
 Non-photorealistic rendering Data hiding
 Content production
 Bioinformatics and authentication
 Virtual reality, augmented reality, and mixed reality
 Image and video retrieval
 Digital museum, digital archiving
 Artificial intelligence and deep learning Content delivery network
 Security and privacy
 Image assessment
 Big data and Cloud computing
 Printing and display technologies
 International standards
 Visual communication138

Paper submission:

The official language is English, and authors should submit their papers as PDF through the online submission system,
which will be available around Jan. 2021 at the following IEVC2021 official website:
https://www.iieej.org/en/ievc2021/
The paper submission guide and IEVC formats (TeX format / MS Word format) will be also provided at this site. The
organizing committee particularly encourages graduate students to present their works in the special sessions thatare now planned by the committee of the conference.

General Papers:

The general papers category is divided into two types: journal track and conference track.

 Journal Track:

Journal track aims to publish the papers on the journal in addition to the publishing in the conference, with a quick
review process. This type of paper will appear in a special issue on “Journal Track Papers in IEVC2021” in the IIEEJTransactions on Image Electronics and Visual Computing, Vol. 9, No. 2 (December 2021), if accepted through thejournal review process. The authors have to prepare two types of papers different in the amount, the paper forthe conference and the paper for the journal. The latter one is the extended version of the former one. Note thatthe paper for the journal should follow the “guidance for paper submission” available from the website of IIEEJ,
to be finally published in the IIEEJ Transactions.
Important Dates- Pre-Entry Submission (title, authors, 100 words abstract):
April 2, Friday, 2021
- Paper Submission (2-4 pages, for the conference):
April 16, Friday, 2021
- Paper Submission (6-8 pages, for the journal):
May 28, Friday, 2021
- Notification of Conference Acceptance:
June 18, Friday, 2021
- Camera-Ready Paper (2-4 pages, for the conference):
July 9, Friday, 2021

 Conference Track:

Conference track aims to present the papers about recent results and preliminary work at IEVC2021. The authors
are required to submit a paper of which length is 2-4 pages. Accepted papers will be published both in the online
proceedings of IEVC2021 (indexed by J-stage) and in the USB proceedings. Rejected papers in the conference track
can be resubmitted as late breaking papers.
Important Dates- Pre-Entry Submission (title, authors, 100 words abstract):
April 2, Friday, 2021
- Paper Submission (2-4 pages):
April 16, Friday, 2021
- Notification of Acceptance:
June 18, Friday, 2021
- Camera-Ready Paper (2-4 pages):
July 9, Friday, 2021

Late Breaking Papers:

All suitably submitted papers for this category will be accepted for the conference. The authors must submit an
abstract of which length is 1-2 pages, and select one from the following two types: 1) Technical papers or 2)
Art/Demo papers. All the registered papers as late breaking papers will be published only in the USB proceedingsof IEVC2021.
Important Dates- Pre-Entry Submission (title, authors):
June 23, Wednesday, 2021
- Abstract Submission (1-2 pages):
June 28, Monday, 2021
- Notification of Acceptance:
July 5, Monday, 2021
- Camera-Ready Paper (1-2 pages):
July 9, Friday, 2021

Further information:

After the conference, the Trans. on IEVC of IIEEJ is planning a forthcoming special issue
on “Extended Papers Presented in IEVC2021”, which will be published in June 2022.
More detailed information will be notified on the IEVC2021 website and the Journal of
IIEEJ.

139

Revised: January 6, 2017
Revised: July 6, 2018

Guidance for Paper Submission
1.

Submission of Papers(1) Preparation before submission
・ The authors should download “Guidance for Paper Submission” and “Style Format” from the“Academic Journals”, “English Journals” section of the Society website and prepare the paperfor submission.
・ Two versions of “Style Format” are available, TeX and MS Word. To reduce publishing costs
and effort, use of TeX version is recommended.
・ There are four categories of manuscripts as follows: Ordinary paper: It should be a scholarly thesis on a unique study, development orinvestigation concerning image electronics engineering. This is an ordinary paper topropose new ideas and will be evaluated for novelty, utility, reliability andcomprehensibility. As a general rule, the authors are requested to summarize a paper
within eight pages.
 Short paper: It is not yet a completed full paper, but instead a quick report of the partial
result obtained at the preliminary stage as well as the knowledge obtained from the said
result. As a general rule, the authors are requested to summarize a paper within four
pages.
 System development paper: It is a paper that is a combination of existing technology or it
has its own novelty in addition to the novelty and utility of an ordinary paper, and thedevelopment results are superior to conventional methods or can be applied to othersystems and demonstrates new knowledge. As a general rule, the authors are requested
to summarize a paper within eight pages.
 Data Paper: A summary of data obtained in the process of a survey, product development,
test, application, and so on, which are the beneficial information for readers even though
its novelty is not high. As a general rule, the authors are requested to summarize a paper
within eight pages.
・ To submit the manuscript for ordinsry paper, short paper, system development paper, or data
paper, at least one of the authors must be a member or a student member of the society.
・ We prohibit the duplicate submission of a paper. If a full paper, short paper, systemdevelopment paper, or data paper with the same content has been published or submitted toother open publishing forums by the same author, or at least one of the co-authors, it shallnot be accepted as a rule. Open publishing forum implies internal or external books,
magazines, bulletins and newsletters from government offices, schools, company
organizations, etc. This regulation does not apply to a preliminary draft to be used at anannual meeting, seminar, symposium, conference, and lecture meeting of our society or othersocieties (including overseas societies). A paper that was once approved as a short paper andbeing submitted again as the full paper after completion is not regarded as a duplicate
submission.
(2) Submission stage of a paper・ Delete all author information at the time of submission. However, deletion of reference
information is the author’s discretion.
・ At first, please register your name on the paper submission page of the following URL, andthen log in again and fill in the necessary information. Use the “Style Format” to upload yourmanuscript. An applicant should use PDF format (converted from dvi of TeX or MS Word140

format) for the manuscript. As a rule, charts (figures and tables) shall be inserted into themanuscript to use the “Style Format”. (a different type of data file, such as audio and video,
can be uploaded at the same time for reference.)
http://www.editorialmanager.com/iieej/
・ If you have any questions regarding the submission, please consult the editor at our office.
Contact:
Person in charge of editing
The Institute of Image Electronics Engineers of Japan
3-35-4-101, Arakawa, Arakawa-Ku, Tokyo 116-0002, Japan
E-mail: hensyu@iieej.org
Tel: +81-3-5615-2893, Fax: +81-3-5615-2894

2.

Review of Papers and Procedures(1) Review of a paper・ A manuscript is reviewed by professional reviewers of the relevant field. The reviewer willdeem the paper “acceptance”, “conditionally acceptance” or “returned”. The applicant is
notified of the result of the review by E-mail.
・ Evaluation method
Ordinary papers are usually evaluated on the following criteria:
 Novelty: The contents of the paper are novel.
 Utility: The contents are useful for academic and industrial development.
 Reliability: The contents are considered trustworthy by the reviewer.
 Comprehensibility: The contents of the paper are clearly described and understood by
the reviewer without misunderstanding.
Apart from the novelty and utility of an ordinary paper, a short paper can be evaluated byhaving a quickness on the research content and evaluated to have new knowledge withresults even if that is partial or for specific use.
System development papers are evaluated based on the following criteria, apart from thenovelty and utility of an ordinary paper.
 Novelty of system development: Even when integrated with existing technologies, thenovelty of the combination, novelty of the system, novelty of knowledge obtained fromthe developed system, etc. are recognized as the novelty of the system.
 Utility of system development: It is comprehensively or partially superior compared tosimilar systems. Demonstrates a pioneering new application concept as a system. The
combination has appropriate optimality for practical use. Demonstrates performance
limitations and examples of performance of the system when put to practical use.
Apart from the novelty and utility of an ordinary paper, a data paper is considered novel if
new deliverables of test, application and manufacturing, the introduction of new technology
and proposals in the worksite have any priority, even though they are not necessarily
original. Also, if the new deliverables are superior compared to the existing technology andare useful for academic and industrial development, they should be evaluated.
(2) Procedure after a review
・ In case of acceptance, the author prepares a final manuscript (as mentioned in 3.).
・ In the case of acceptance with comments by the reviewer, the author may revise the paper inconsideration of the reviewer’s opinion and proceed to prepare the final manuscript (as141mentioned in 3.).
・ In case of conditional acceptance, the author shall modify a paper based on the reviewer’s
requirements by a specified date (within 60 days), and submit the modified paper for
approval. The corrected parts must be colored or underlined. A reply letter must be attachedthat carefully explains the corrections, assertions and future issues, etc., for all of theacceptance conditions.
・ In case a paper is returned, the author cannot proceed to the next step. Please look at thereasons the reviewer lists for the return. We expect an applicant to try again after reviewing
the content of the paper.
(3) Review request for a revised manuscript
・ If you want to submit your paper after conditional acceptance, please submit the reply letterto the comments of the reviewers, and the revised manuscript with revision history to thesubmission site. Please note the designated date for submission. Revised manuscripts
delayed more than the designated date be treated as new applications.
・ In principle, a revised manuscript willl be reviewed by the same reviewer. It is judged either
accceptance or returned.
・ After the judgment, please follow the same procedure as (2).

3.

Submission of final manuscript for publication(1) Submission of a final manuscript
・ An author, who has received the notice of “Acceptance”, will receive an email regarding thecreation of the final manuscript. The author shall prepare a complete set of the final
manuscript (electronic data) following the instructions given and send it to the office by thedesignated date.
・ The final manuscript shall contain a source file (TeX edition or MS Word version) and a PDF
file, eps files for all drawings (including bmp, jpg, png), an eps file for author’s photograph(eps or jpg file of more than 300 dpi with length and breadth ratio 3:2, upper part of the body)for authors’ introduction. Please submit these in a compressed format, such as a zip file.
・ In the final manuscript, write the name of the authors, name of an organizations,introduction of authors, and if necessary, an appreciation acknowledgment. (cancel macros in
the Style file)・ An author whose paper is accepted shall pay a page charge before publishing. It is theauthor’s decision to purchase offprints. (ref. page charge and offprint price information)(2) Galley print proof
・ The author is requested to check the galley (hard copy) a couple of weeks before the paper ispublished in the journal. Please check the galley by the designated date (within one week).
After making any corrections, scan the data and prepare a PDF file, and send it to our officeby email. At that time, fill in the Offprint Purchase Slip and Copyright Form and return thescanned data to our office in PDF file form.
・ In principle, the copyrights of all articles published in our journal, including electronic form,
belong to our society.
・ You can download the Offprint Purchase Slip and the Copyright Form from the journal on our
homepage. (ref. Attachment 2: Offprint Purchase Slip, Attachment 3: Copyright Form)(3) Publication
・ After final proofreading, a paper is published in the Academic journal or English transaction(both in electronic format) and will also be posted on our homepage.

142

IIEEJ Trans. on Image Electronics and Visual Computing©2020 IIEEJ

Vol. 8, No. 2 2020

Editor in Chief: Mei Kodama
The Institute of Image Electronics Engineers of Japan
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan