IIEEJ Transactions on
IIEEJ Transactions on
Image Electronics and Visual Computing

Image Electronics andVisual Computing

Special Issue on Journal Track Papers in IEVC2021 PartⅡ
Special Issue on CG & Image Processing Technologies
Supporting and Expanding Human Creativities

Vol. 10, No. 1 2022

I.I.E.E.J

Editorial Committee of IIEEJ
Editor in Chief
Mei KODAMA (Hiroshima University)

Vice Editors in Chief
Osamu UCHIDA (Tokai University)
Naoki KOBAYASHI (Saitama Medical University)
Yuriko TAKESHIMA (Tokyo University of Technology)

Advisory Board
Yasuhiko YASUDA (Waseda University Emeritus)
Hideyoshi TOMINAGA (Waseda University Emeritus)
Kazumi KOMIYA (Kanagawa Institute of Technology)
Fumitaka ONO (Tokyo Polytechnic University Emeritus)
Yoshinori HATORI (Tokyo Institute of Technology)
Mitsuji MATSUMOTO (Waseda University Emeritus)
Kiyoshi TANAKA (Shinshu University)
Shigeo KATO (Utsunomiya University Emeritus)

Editors
Yoshinori ARAI (Tokyo Polytechnic University)
Chee Seng CHAN (University of Malaya)
Naiwala P. CHANDRASIRI (Kogakuin University)
Chinthaka PREMACHANDRA (Shibaura Institute of Technology)
Makoto FUJISAWA(University of Tsukuba)
Issei FUJISHIRO (Keio University)
Kazuhiko HAMAMOTO (Tokai University)
Madoka HASEGAWA (Utsunomiya University)
Ryosuke HIGASHIKATA (FUJIFILM Business Innovation Corp.)
Yuki IGARASHI (Ochanomizu University)
Tomokazu ISHIKAWA（Toyo University）
Masahiro ISHIKAWA （Saitama Medical University）
Naoto KAWAMURA (Canon OB)
Shunichi KIMURA (FUJIFILM Business Innovation Corp.)
Shoji KURAKAKE (NTT DOCOMO)
Kazuto KAMIKURA (Tokyo Polytechnic University)
Takashi KANAI (The University of Tokyo)
Tetsuro KUGE (NHK Engineering System, Inc.)
Koji MAKITA (Canon Inc.)
Tomoaki MORIYA (Tokyo Denki University)
Paramesran RAVEENDRAN (University of Malaya)
Kaisei SAKURAI (DWANGO Co., Ltd.)
Koki SATO (Shonan Institute of Technology)
Syuhei SATO（University of Toyama）
Masanori SEKINO (FUJIFILM Business Innovation Corp.)
Kazuma SHINODA (Utsunomiya University)
Mikio SHINYA (Toho University)
Shinichi SHIRAKAWA (Aoyama Gakuin University)
Kenichi TANAKA (Nagasaki Institute of Applied Science)
Yukihiro TSUBOSHITA (Fuji Xerox Co., Ltd.)
Daisuke TSUDA (Shinshu University)
Masahiro TOYOURA (University of Yamanashi)
Kazutake UEHIRA (Kanagawa Institute of Technology)
Yuichiro YAMADA (Genesis Commerce Co.,Ltd.)
Norimasa YOSHIDA (Nihon University)
Toshihiko WAKAHARA (Fukuoka Institute of Technology OB)
Kok Sheik WONG (Monash University Malaysia)

Reviewer
Hernan AGUIRRE (Shinshu University)
Kenichi ARAKAWA (NTT Advanced Technology Corporation)
Shoichi ARAKI (Panasonic Corporation)
Tomohiko ARIKAWA (NTT Electronics Corporation)
Yue BAO (Tokyo City University)
Nordin BIN RAMLI (MIMOS Berhad)
Yoong Choon CHANG (Multimedia University)
Robin Bing-Yu CHEN (National Taiwan University)
Kiyonari FUKUE (Tokai University)
Mochamad HARIADI (Sepuluh Nopember Institute of Technology)
Masaki HAYASHI (UPPSALA University)
Takahiro HONGU (NEC Engineering Ltd.)

Yuukou HORITA (University of Toyama)
Takayuki ITO (Ochanomizu University)
Masahiro IWAHASHI (Nagaoka University of Technology)
Munetoshi IWAKIRI (National Defense Academy of Japan)
Yoshihiro KANAMORI (University of Tsukuba)
Shun-ichi KANEKO (Hokkaido University)
Yousun KANG (Tokyo Polytechnic University)
Pizzanu KANONGCHAIYOS (Chulalongkorn University)
Hidetoshi KATSUMA (Tama Art University OB)
Masaki KITAGO (Canon Inc.)
Akiyuki KODATE (Tsuda College)
Hideki KOMAGATA (Saitama Medical University)
Yushi KOMACHI (Kokushikan University)
Toshihiro KOMMA (Tokyo Metropolitan University)
Tsuneya KURIHARA (Hitachi, Ltd.)
Toshiharu KUROSAWA (Matsushita Electric Industrial Co., Ltd. OB)
Kazufumi KANEDA (Hiroshima University)
Itaru KANEKO (Tokyo Polytechnic University)
Teck Chaw LING (University of Malaya)
Chu Kiong LOO (University of Malaya) F
Xiaoyang MAO (University of Yamanashi)
Koichi MATSUDA (Iwate Prefectural University)
Makoto MATSUKI (NTT Quaris Corporation OB)
Takeshi MITA (Toshiba Corporation)
Hideki MITSUMINE (NHK Science & Technology Research Laboratories)
Shigeo MORISHIMA (Waseda University)
Kouichi MUTSUURA (Shinsyu University)
Yasuhiro NAKAMURA (National Defense Academy of Japan)
Kazuhiro NOTOMI (Kanagawa Institute of Technology)
Takao ONOYE (Osaka University)
Hidefumi OSAWA (Canon Inc.)
Keat Keong PHANG (University of Malaya)
Fumihiko SAITO (Gifu University)
Takafumi SAITO (Tokyo University of Agriculture and Technology)
Tsuyoshi SAITO (Tokyo Institute of Technology)
Machiko SATO (Tokyo Polytechnic University Emeritus)
Takayoshi SEMASA (Mitsubishi Electric Corp. OB)
Kaoru SEZAKI (The University of Tokyo)
Jun SHIMAMURA (NTT)Tomoyoshi SHIMOBABA (Chiba University)
Katsuyuki SHINOHARA (Kogakuin University)
Keiichiro SHIRAI (Shinshu University)
Eiji SUGISAKI (N-Design Inc. (Japan), DawnPurple Inc.(Philippines))
Kunihiko TAKANO (Tokyo Metropolitan College of Industrial Technology)
Yoshiki TANAKA (Chukyo Medical Corporation)
Youichi TAKASHIMA (NTT)Tokiichiro TAKAHASHI (Tokyo Denki University)
Yukinobu TANIGUCHI (NTT)Nobuji TETSUTANI (Tokyo Denki University)
Hiroyuki TSUJI (Kanagawa Institute of Technology)
Hiroko YABUSHITA (NTT)
Masahiro YANAGIHARA (KDDI R&D Laboratories)
Ryuji YAMAZAKI (Panasonic Corporation)IIEEJ OfficeOsamu UKIGAYA
Rieko FUKUSHIMA
Kyoko HONDA

Contact Information
The Institute of Image Electronics Engineers of Japan (IIEEJ）
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
Tel : +81-3-5615-2893 Fax : +81-3-5615-2894
E-mail : hensyu@iieej.org
http://www.iieej.org/ (in Japanese)
http://www.iieej.org/en/ (in English)
http://www.facebook.com/IIEEJ (in Japanese)
http://www.facebook.com/IIEEJ.E (in English)

IIEEJ Transactions on
Image Electronics and Visual Computing
Vol.10 No.1 June 2022
CONTENTS

Special Issue on Journal Track Papers in IEVC2021Part ＩＩ
1

Upon the Special Issue on Journal Track Papers in IEVC2021Part II

Osamu UCHIDA

Contributed Papers
2

Robust Plane Detection in Terrestrial Laser Scanning Point Clouds using
Efficient Multi-scale Sliding Voxels and Median Plane Weighted PCA

Jaime SANDOVAL, Kazuma UENISHI, Munetoshi IWAKIRI,
Kiyoshi TANAKA

11

Aggregative Input Convolution for Large-Scale Point Cloud Semantic
Segmentation

Kana KURATA, Yasuhiro YAO, Shingo ANDO, Naoki ITO,
Jun SHIMAMURA

19

Registration of Histopathological Heterogeneous Stained Images Utilizing
GAN Based Domain Adaptation TechniqueTanwi BISWAS, Hiroyuki SUZUKI, Masahiro ISHIKAWA,
Naoki KOBAYASHI, Takashi OBI

28

Spectral Super-Resolution Using CNN Decomposing a Color Image into
Luminance and Chrominance Components

Masahiro SAKAMOTO , Kazufumi KANEDA,
Bisser RAYTCHEV

36

Evaluating YOLOv3 for Identification and Classification of Functionaland Sclerosed Glomeruli

Thalita Munique COSTA, Lourenço BARBOSA, Yoko USAMI,
Mai IWAYA, Kiyoshi TANAKA, Fabio SCHNEIDER

47

Measurement of Eye Size Illusion Caused by Thickness of Eyeliner
on Double-Eyelid Eyes

Rena OKURI , Shuhei KODAMA, Tokiichiro TAKAHASHI

55

3D Distance Field-Based Apparel ModelingMasanori NAKAYAMA, Takami YAMAMOTO,
Issei FUJISHIRO

66

Invertible Fingerprint Replacement for Image Privacy Protection

Kazuya NAKAMURA, Shugo YAMAGUCHI,
Hideki TSUNASHIMA, Shigeo MORISHIMA

75

Game Development Using the Discrimination System to Improve Typing Skills Surena KAWAHARA, Teruaki HIRANO, Noriyoshi OKAMOTO,
Daisuke TAKAHASHI

Short Paper
82

High-Speed and Accurate Authenticity Judgment Using Physically Unclonable
Function of Inkjet Printed CodeKazuaki SUGAI, Kohei SHIRAI, Kitahiro KANEDA,
Keiichi IWAMURA

Special Issue on CG & Image Processing TechnologiesSupporting and Expanding Human Creativities
89

Upon the Special Issue on CG & Image Processing Technologies Supportingand Expanding Human Creativities

90

Face Reconstruction Algorithm based on Lightweight Convolutional Neural
Networks and Channel-wise Attention

99

Investigation of New Design Method Rooted in Local History and Culture
through Application of Photogrammetry: A Case Study on Application of
Maruko-bune, a Traditional Boat Unique to Lake Biwa in Japan, toArchitectural Design

Shinya KITAOKA

Contributed Papers
Haoqi GAO, Koichi OGAWARA

Short Paper
Akari YOSHIDA, Toshitomo SUZUKI, Hiroyuki TAGAWA

Regular Section
Contributed Papers
107

Generative Image Quality Improvement in Omnidirectional Free-Viewpoint
Images and Assessments

Qiaoge LI, Oto TAKEUCHI, Hidehiko SHISHIDO,
Yoshinari KAMEDA, Hansung KIM, Itaru KITAHARA

120

Computerized Classification Method for 1p/19q Codeletion in Low Grade
Gliomas from Brain MRI Images Using Three Dimensional Radiomics
Features

Daiki TANAKA, Akiyoshi HIZUKURI, Ryohei NAKAYAMA

127

Improvements over Coordinate Regression Approach for Large-Scale Face
Alignment

Haoqi GAO, Koichi OGAWARA

Announcements
136

Call for Papers : Special Issue on Data Interwork, Sharing, and Reusing
Technologies Supporting Digital Transformation Era

137

Call for Papers: Special Issue on Image-Related Technology for Digital
Fabrication

Guide for Authors
138

Guidance for Paper Submission

Published two times a year by the Institute of Image Electronics Engineers of Japan （IIEEJ）
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan
Tel：+81-5615-2893 Fax：+81-5615-2894 E-mail：hensyu@iieej.org http://www.iieej.org/

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Upon the Special Issue on
Journal Track Papers in IEVC2021 Part II
Editor: Osamu UCHIDA(Tokai University)
The 7th International Conference on Image Electronics and Visual Computing(IEVC2021) was held on September 8-11, 2021, as the international academic event of
Image Electronics Engineers of Japan (IIEEJ). It was based on the great success of theprevious six workshops in 2007 (Cairns, Australia), 2010 (Nice, France), 2012 (Kuching,
Malaysia), 2014 (Koh Samui, Thailand), 2017 (Da Nang, Vietnam), and in 2019 (Bali,
Indonesia). The conference aims to bring together researchers, engineers, developers, andstudents from various fields in both academia and industry for discussing the latest
studies, standards, developments, implementations, and application systems in all areasof image electronics and visual computing. The IEVC2021 Committee initially planned tohold IEVC2021 as a face-to-face conference in Shiretoko (Shari), Hokkaido, Japan, a World
Natural Heritage Site. However, due to the worsening of the COVID-19 infection
nationwide, the committee has decided to hold IEVC 2021 in a fully online format,
prioritizing the safety of the participants.
As well as in IEVC2019, IEVC2021 has two paper categories: general paper andlate-breaking paper (LBP). Moreover, the general paper category has two tracks: Journal
track (JT) and Conference track (CT). Journal track provides authors with the advantageof having their papers published in the special issue of IIEEJ Transactions on Image
Electronics and Visual Computing (IIEEJ Trans. IEVC) scheduled for the December 2021
issue (Part I), and this June 2022 issue (Part II).
This special issue Part II contains 9 papers that were accepted within the publication
schedule of June issue. As the special issue Part I, published in December 2021, contained
4 papers, the number of accepted papers in Special Issues of JT papers in IEVC 2021
reached to 13 in total.
Last but not least, I would like to thank all the reviewers and editors for theircontribution to improving the quality of papers. I would also like to express my deepest
gratitude to the members of the editorial committee of IIEEJ and the staff at the IIEEJ
office for various kinds of support.

1

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2021Part II -Robust Plane Detection in Terrestrial Laser Scanning Point Clouds using
Eﬃcient Multi-scale Sliding Voxels and Median Plane Weighted PCA
Jaime SANDOVAL† (Member), Kazuma UENISHI†† (Member),
Munetoshi IWAKIRI††† (Member), Kiyoshi TANAKA†† (Fellow)† AB.do Corp. , †† Shinshu University , ††† National Defense Academy of Japan<Summary> Planes are simple shapes often found in urban environments. Their detection is crucial
for several applications of 3D point clouds. Eﬀorts have focused on detecting planes in low-cost and lowrange scenarios. However, in Terrestrial Laser Scanning (TLS), the range is extended to a few hundred
meters. This long-range adds huge variations in points’ density that complicates deﬁning proper thresholds
for conventional methods. Therefore, we propose to employ the sliding voxel plane detector over multiple
voxel sizes to estimate hypothetical planes in a coarse-to-ﬁne way, followed by a merging Non-Maximum
Suppression to detect planes robustly. According to experimental results, the proposed method shows
superior precision, eﬃciency, and scalability in TLS point clouds.
Keywords: point couds, plane detection, ransac, hough transform, sliding voxel, PCA

algorithm for TLS point clouds that tackles the sparsity

1.

Introduction

problem by using multiple sliding voxel detectors over different scales, and maintains eﬃciency in large-scale TLS

3D point clouds are sets of unorganized points that resemble the surface of objects. Recently, with the advent

point clouds due to its coarse-to-ﬁne approach.
Experiments results show that the proposed methodof low-cost sensors and photogrammetry, point cloud processing has gained the attention of researchers and indus-

is superior in accuracy and eﬃciency when compared tothe state-of-the-art both in synthetic and real TLS point
clouds. Furthermore, it is scalable to point clouds withtry. Planes are the most common geometric primitive inurban environments and man-made objects. Therefore,
plane detection is an essential task in Simultaneous Localization And Mapping (SLAM)1)–3), reconstruction4) ,object detection5), keypoint detection6), among others.
Nonetheless, due to sensor artifacts, sensing patterns,

dozens of millions of points without drastically aﬀecting
its performance.
The organization of this paper is as follows. In section 2 we breﬂy summarize the conventional methods. In
section 3 we describe in detail the proposed method. Section 4 contains the experiments results and section 5 theconclusions.

noise, and other nuisances of point clouds, their detection
is non-trivial. Particularly, large range sensors, such as
those used in Terrestrial Laser Scanning (TLS), produce
point clouds with high variations in density over dozens oreven hundreds of meters. These nuisances complicate thedecision of important plane detection parameters such as2.
2.1

Conventional Methods

Eﬃcient RANSAC

Eﬃcient RANSAC7)(EFRANSAC) is a method thatsearch radius or voxel size.
In the past, several approaches7)–9) tried to solve plane
detection mostly for short range point clouds. However, they fail to detect most of the planes in TLS point
clouds because they expect a uniform density10). Another approach11) uses local robust statistics to detect

improves RANSAC12) eﬃciency in both hypothesis generation and model validation steps. To achieve higher
performance, it uses an octree for localized sampling andscoring over disjoint random subsets.
It delimits the search space for hypothesis generation
by selecting a random octree cell that contains a randomly drawn point. Then, it randomly selects the otherplanar patches, but expects accurate point-wise normal
vectors for it to succeed. Therefore, to overcome conventional methods limitations, we propose a plane detection

2 remaining samples from within that cell.

2

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

top approach, it is more prone to fall into local noise,

Once a hypothetical plane is drawn, it uses random disjoint subsets of the point cloud to extrapolate the model
validation score (number of inliers) of one subset into thewhere noise levels could exceed the breakpoint of the median in the worst scenarios.

set of all points. After validating a model, inliers are reﬁned by a connected component analysis in the projectionof the shape to a plane.
However, as its authors stated, it can tolerate up to a

2.4

The Sliding Voxel (SV) algorithm9) uses an octree tosubdivide the point cloud into a set of voxel centroids
and normal vectors V . A voxel and its 26 neighbors are

20% of outliers before returning false positives. Although
it has optimizations in the hypothesis generation it is still
expecting a degree of density and relatively high inliers
ratio of the point cloud. Moreover, it does not compute
point-wise normal vectors, which are frequently inaccurate and computationally expensive.
2.2

Sliding voxel

called a sliding voxel. For each sliding voxel, it eﬃciently
analyzes its neighborhood planarity and decides whether
to grab a hypothetical plane or not.
It validates hypothetical planes P using a 2-step approach: (1) it counts inliers voxels of all hypotheticalplanes in V using point-to-plane distance and normal vectors deviation, and (2) it sorts P by their inliers count indescending order and extracts the inliers from V . P are

Randomized Hough transform

The Randomized Hough Transform(RHT)8) for plane
detection is a method that employs a conditioned random
sampling to vote for hypothetical planes into an accumulator iteratively.

pruned by their thickness

λ3∈ R | λ1 ≥ λ2 ≥ λ3
T =λ2

A random sample is valid only if the maximum distance between their points falls inside a user deﬁned
range. Once a plane gets enough votes, their inliers are

estimated from PCA over the points distribution of each
inlier voxel of V . Finally, the planes are reﬁtted using thecentroid and normal vector from PCA.
This method is simple, robust, and highly eﬃcient inremoved from the point cloud and the accumulator is reseted. Then, it keeps iterating until there are not enoughpoints left or if the failure rate of drawing random samples exceeds a user deﬁned threshold.

short-range point clouds. But in TLS point clouds, thehigh variations in density and noise make it diﬃcult todeﬁne a suitable voxel size for every surface. At a larger

As a result of removing planar points, in each iteration,
the same-plane inliers ratio of the point cloud decreases,

voxel size, it can detect sparser planes but it can dismiss, merge or miscalculate smaller planes. On the other
hand, at a smaller voxel size it is more aﬀected by noise
and tends to dismiss sparser planes. Therefore, the user

making increasingly diﬃcult to draw good samples fromthe point cloud. Moreover, its random sampling mechanism expects some knowledge about the planes dimensions of the point cloud, and its ﬁnishing conditions have

has to decide the tradeoﬀ balance by setting the mostappropriate voxel size for each case.

to be carefully deﬁned. Otherwise, it will take increasingly longer times.
2.3

(1)

3.

Robust statistics plane detector

3.1
11)

Proposed Method

Overall routine

To solve previous methods issues in TLS point clouds,
we propose to employ SV not as a detector but as a hypothetical planes estimator. The key idea is to detect bigger
or sparser planes with a bigger voxel size and smaller ordenser planes with a small voxel size in a coarse-to-ﬁneway.

The Robust Statistics Plane Detector (RSPD) is abottom-top approach for plane detection. It trasverses
vertically an octree in the search for planar patches using
planarity tests based on robust statistics. Each planar
patch is grown over its coplanar neighbors and merged
until stable to deﬁne a plane.
For an easier parameterization, they thresholded theWhile the sliding voxel itself is a bottom-top approach,the proposed method is a top-bottom. Therefore, as opposed to RSPD, the proposed method is a combination

planarity tests in terms of angles and deﬁned them byrobust statistics metrics such as Median Absolute Deviation(MAD).

of the best of both approaches.
The proposed method ﬂow is depicted in Fig. 1. It
starts by deﬁning the voxel size of the octree levels deﬁned
by the user. Consequently, it employs a modiﬁed slid-

This method requires normal vectors and a computationally intensive graph built from the neighborhood ofeach point estimated in advance. Moreover, as a bottom-

3

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）  

 

 












 

Fig. 1 

 

Proposed method ﬂow

ing voxel algorithm to collect hypothetical planes. This
modiﬁcation lies on ﬁltering the points of V by thresholding their voxels thickness T . When T of Eq. (1) isa plane in the cluster with normal vector n̂c that satisfy

obtained from a point distribution sampled from a surface, it represents both its noise level and curvature as T

2arccos(|n̂p · n̂c |) ≤ θth
π(2)||dp | − |dc || ≤ th ,

(3)

and

increases proportionally to these characteristics. Therefore, the proposed method can omit regions with highnoise and outliers, especially when the voxel size is com-

where | | is the absolute value.
Then, a merging Non-Maximum Suppression (NMS)

posed of more than 1 surface where a smaller voxel size
is more suitable.
The parameters of the sliding voxel plane detector are
ﬁxed for all scales. The planarity threshold pth , inliers
threshold th , and angular threshold θth are shared between the proposed method and SV.

is applied iteratively until the resulting number of clusters is not reduced. The diﬀerence from a classic NMS

Figure 1(a) and (b) shows the planes detected at theinitial voxel size and ﬁnal voxel size, respectively. In Figure 1(a), sliding voxel inliers are shown as big and colored
dots, where each color represents a diﬀerent plane, andthe non-planar points are black. We can notice that only

and create a new cluster otherwise. Here, the voxel sizeis that, instead of only suppressing similar hypotheticalplanes, we merge those whose nearest points are up to
1 voxel size apart from the biggest plane in the cluster
considered is the smallest used by the detectors. Thismerging mechanism allows us to maximize the true positives without throwing away planes that are likely to bepart of a diﬀerent structure. Furthermore, when merging
clustered planes, we concatenate their sliding voxel inliers

the biggest planes were detected, while the small planes
were omitted by the detector. These non-planar points
will be the input of a detector with a smaller voxel size.
Figure 1(b) shows a hollowed cloud due to the eﬀectof removing planar points from previous scales. Here, we
can observe that more detailed and smaller planes wereand re-estimate each cluster plane.
Finally, the centroids and normals point clouds used ineach detector are combined for their use in pruning andvalidation of the resulting planes in the same way as SV.
3.2

detected. However, some sliding voxel inliers correspondto planes from previous scales. These inliers are near theborder of other surfaces and need to be merged to avoid

Median Plane Weighted PCA

During the global validation of the sliding voxel algorithm and the merging NMS of the proposed method, asmall percentage of errors can arise from merging sliding voxels with higher thresholds. Therefore, to extract
only highly reliable planes coeﬃcients, we apply a special
case of median ﬁlter on the plane coeﬃcients based on
the sliding voxel inliers.
To achieve this, we estimate weights for every sliding voxel using a Radial Basis Function (RBF) over the

duplicated detections.
Therefore, after ﬁnishing the detection of hypotheticalplanes, similar planes are clustered by their angle n̂ ={a, b, c} and oﬀset d components of its cannonical formax + by + cz + d = 0 using θth and th thresholds. A planewith normal vector n̂p belongs to a cluster when there is

4

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 1(a) PCA modelFig. 2

Point CloudS
S noisyB
B noisyN(b) MPWPCA model

signed distance to the median plane, then we apply aweighted PCA to extract the coeﬃcients, hence, we call
it Median Plane Weighted PCA (MPWPCA).
Figure 2 depicts a 2D example of this technique ap-

Ic =1
Σμ =  (C − μ)T diag(W )(C − μ)
wi

(9)

(10)

where diag(W ) is a diagonal matrix containing theweights and (C −μ) represents the centroids cloud matrix

distance to their median plane.
The proposed method performs MPWPCA as follows.

with points in their its rows and centered to the medianμ. Then, we estimate the robust plane from the smallest
eigenvector of Σμ and med(C) as in Eq. (6).

For every plane p from the sliding voxel plane detector
we have n pairs of centroids C ∈ Rn×3 and their corresponding normal vectors N ∈ Rn×3 . We ﬂip every sliding voxel normal if their deviation from p is more than90[deg]. Then, let4.

Experiments and Results

Table 1 shows the basic information of the point clouds
used for the experiments, where the range is expressed in{med(Nx ), med(Ny ), med(Nz )}(4)||{med(Nx ), med(Ny ), med(Nz )}||2

Bounding Box Diagonal Distance (BBDD). The S and S
noisy point clouds were obtained from a Velodyne HDL64E simulation in Blensor13). S noisy was generated by
setting Gaussian noise on with 0-mean and 0.01 variance.
The B and B noisy point clouds were obtained in thesame setting of S and S noisy , the model used for simulations comes from the Bacelona Robot Lab Dataset14) .
N is the G1 point cloud from a sphere detection study15) .

be the normalized median of the x,y, and z componentsof the normal vectors N , and(5)be the centroids median values the x, y, and z components of C, hence the median plane is deﬁned as

It was obtained by a high-precision FARO® LiDAR
scanner and roughly 80% of the points are planar.
For comparison experiments, we carefully generated
ground truth planes by manually selecting planar surface
points and estimated plane coeﬃcients with PCA using atool developed speciﬁcally for this purpose. EFRANSAC(6)

where med stands for “median” and || · ||2 is the L2 -norm.
We use pm to estimate sliding-voxel-wise weights wi ∈W based on the distance of every ci to pm as follows,∀ci ∈ C, wi = e(8)

and covariance matrix

distance to their median line. To understand the contribution of each point to the ﬁnal model, the blue points
were resized according to the weights estimated from theδ(ci ,pm )2thw i Ωi ,Consequently, we compute a weighted mean1 
w i ciμ= wi

plying a Weighted PCA with a strict threshold over the−nwhere Ωi is the number of points inside a sliding voxel
center. Planes that reduced their inliers by more thanthe median breakdown point (50%) are discarded.

lines deﬁne a distance threshold between the model andthe data points. Next, the median line model is estimatedfrom the points positions and normal vectors orientations.
Fig. 2(b) shows the resulting model in red after ap-

pm = {med(N ), −med(N ) · med(C)}Planes[#]
38
38
74
74
96

i=1

plied in a similar condition. Fig. 2(a) shows a set of points
and normal vectors, the blue line is the ground truth
and the red line model was obtained using PCA overthe points distribution. The upper and bottom dashedmed(C) = {med(Cx ), med(Cy ), med(Cz )}BBDD[m]200.9812
201.2574
160.6900
160.8506
347.0569

where th is both the inliers threshold and clustering oﬀset
threshold of the proposed method and δ is the point-toplane signed distance function. The inliers count Ic of
the plane is also weighted using W

Models estimated using PCA and MPWPCA

med(N ) =Point clouds information

Points[#]129,621
129,621
131,021
131,02133,789,031

and RHT are non-deterministic, therefore, we averaged
the data of 10 executions for all the evaluation metrics.

(7)

5

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Precision, higher is better(c) F1 -score, higher is better (d)(b) Recall, higher is better
Fig. 3

Processing time[s](logscale), lower is better

Numerical evaluation results

The source code of the proposed method and SV was
implemented in C++ using the PCL 1.11.1 routines on
Windows 10 compiled with MSVC 16.77 in release mode.

Fig. 3(a) shows the precision (correctness) results. The
proposed method is clearly more precise than the conventional methods, particularly, when median ﬁlter is turned
on. Fig. 3(b) shows the recall (completeness) results.
EFRANSAC has the highest recall in all the synthetic
cases, except for the B noisy point cloud. On the otherWe used the implementation of EFRANSAC from the
CGAL library16). For RHT, and RSPD we used the implementations made public by their authors. The computer for experiments has an Intel® Xeon® E-2224G
CPU at 3.5GHz with 64GB of RAM.

hand, the proposed method is always 2nd or 3rd place inall cases.
Fig. 3(c) shows the F1 -score results. Due to the superior precision and competitive recall, Proposed+ is clearly

EFRANSAC and RSPD require point-wise normal vectors. Therefore, to avoid bias in the detection results, weused the same input normal vectors for both RSPD andsuperior to the conventional methods.
In Fig. 3(d) we observe that the proposed method is

EFRANSAC estimated with the PCL command line tool
as follows. For the N point cloud we used a radius of
0.15[m], and for the synthetic datasets we used a neighborhood size (k) of 30.

one of the most eﬃcient since it removes planar points
adaptively with respect to the voxel size. Furthermore, it
escalates well with the size of the point cloud as observedin the N results.
On the other hand, EFRANSAC and RSPD are of theWe validated the performance of the proposed method
in terms of processing time and the well-known precision, recall, and F1 -score from the ﬁeld of information
retrieval17), against synthetic point clouds with realistic

most eﬃcient methods for the synthetic datasets but theirperformance do not escalate well in massive point clouds.
Although both require computationally expensive pointwise normal vectors, even if we subtract the estimation
time from EFRANSAC and RSPD, Proposed+ is still
2.48 times faster than EFRANSAC and 15.02 times fasternoise simulation. For this purpose, a detected plane wasmatched to a ground truth plane if their angle is less than
15[deg] and its oﬀset absolute diﬀerence is less than 0.2.
Furthermore, we manually adjusted the parameters of
the evaluated methods to maximize their F1 -score, andprovided a visual assessment of the detection results byextracting the inliers of their resulting plane coeﬃcientsin a consistent way for all the evaluated methods.
Figure 3 shows the numerical results of the experiments. Each bar represents the result of executing amethod over a dataset and all bars are grouped by theirthan RSPD in the N point cloud.
To assess the results visually we selected only themost accurate results. For each point cloud, we selected
the proposed method option with the highest F1 -score(Proposed+) and the next 2 highest from the conventional methods which turned out to be EFRANSAC andSV.

datasets. Among the evaluated methods, “Proposed”
stands for the proposed method and “Proposed+” stands
for the proposed method with its median ﬁlter (MPW-

To avoid bias in visual assessment, we segmented theplanes by counting their inliers in a consistent way foreach point cloud. The segmentation parameters are

PCA). For the non-deterministic methods, we show thevariations in their results using the standard deviationfrom their mean value.

shown in Tab. 2,  is the threshold for the absolute pointto-plane distance, and θ is the threshold for the maximum
angle deviation between the inlier and the plane normal

6

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Ground truth(b) EFRANSAC
Fig. 4

Table 2
Point CloudS
S noisyB
B noisyN(c) SV(d) Proposed+

Visual results for N

Segmentation parameters
Neighborhood
30[#]
30[#]
30[#]
30[#]
0.15[m]

[m]
0.050
0.070
0.050
0.100
0.025

θ[deg]
13.5
30.0
45.0
45.0
45.0

vector. The neighborhood size in number of points orin radius size was estimated as the minimum possible to(a) EFRANSAC
Fig. 5

estimate normal vectors with correct orientations.
Each plane was assigned with a diﬀerent color from(b) Proposed+

False positives comparison in the N point cloud

Figure 8 shows the visual results for the B point

the Glasbey LUT color scheme of the PCL library. They
were sorted sorted by their number of inliers in descendingorder for their extraction. Plane inliers are shown in theircloud. The selected methods performed well in the noiseless dataset with negligible errors. However, in Fig. 9
we can observe a lack of robustness of the selected methods. While the proposed method detected more accurately bigger planes, it had diﬃculties in detecting somerespective color and outliers in black. Noticeably, notall plane inliers can be selected due to noise and highdeviation of normal vectors from their estimation.
Figure 4 shows the visual results for the N point cloud.
The biggest planes were correctly detected by most methods. The high recall of EFRANSAC can also be con-of the smaller planes due to an aggressive ﬁltering. On theother hand, EFRANSAC and SV detected more smaller
planes at the cost of showing several false positives that
decreased drastically their precision.

ﬁrmed easily. However, its precision decreases when it
detects false positives on slightly curved regions such as
the ones shown in Fig. 5(a). On the other hand, both the5.

Conclusions

In this paper, we addressed the problem of detecting

proposed method and SV had a low false positives rateon these surfaces because they test for planarity before
accepting a hypothetical plane.
Figure 6 shows the visual results for the S point cloud.
The selected methods performed visually good and also
had similar segmentation patterns when compared withplanes in typical TLS point clouds obtained from the simulation of a cheaper and less precise rotating-head LiDAR
and a high-precision LiDAR scanner with a range of up
to roughly 300[m].
Conventional plane detection methods were designed toaddress the short-range scenarios and many of them suﬀer

their ground truth. On the other hand, in Fig. 7, theresults S noisy point cloud indicates a lack of robustnessof SV in broad planes and of EFRANSAC in less densefrom diﬃculties of setting appropriate thresholds for TLS
point clouds. Also, their eﬃciency do not escalate wellwith the number of points.

regions. This is due to insuﬃcient voxel size of SV andthe model validation of EFRANSAC which extrapolates
the inliers count for model validation resulting in a lackof robustness in less dense regions.

SV is the best performing on short-range point clouds,
but the sparsity of TLS increase the amount of computations it needs.
Our experiments indicate that7

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Ground truth(b) EFRANSAC
Fig. 6(a) Ground truth(c) SV(b) EFRANSAC(c) SV(d) Proposed+

Visual results for B(b) EFRANSAC
Fig. 9(d) Proposed+

Visual results for S noisy

Fig. 8(a) Ground truth(d) Proposed+

Visual results for S(b) EFRANSAC
Fig. 7(a) Ground truth(c) SV(c) SV

Visual results for B noisy

8(d) Proposed+

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

EFRANSAC can be drastically ineﬃcient in most TLS
point clouds. However, it is a good option for planar
approximation of slightly curved surfaces rather thantures and its Feature Descriptor”, The journal of the Instituteof Image Electronics Engineers of Japan: Visual Computing,
Devices & Communications, Vol. 46, No. 2, pp. 283–297 (InJapanese)(2017).

plane detection when performance is not important. We
found that both RSPD and EFRANSAC eﬃciency canbe competitive only when the point-wise normal vectors
are given or easy to compute. The latter being the case of7) R. Schnabel, R. Wahl, R. Klein: “Eﬃcient RANSAC for PointCloud Shape Detection”, Proc. of Computer Graphics Forum,Vol. 26, No. 2, pp. 214–226 (2007).
8) D. Borrmann, J. Elseberg, K. Lingemann, A. Nüchter: “The3D Hough Transform for Plane Detection in Point Clouds: A
Review and a New Accumulator Design”, 3D Research, Vol. 2,
Article 3 (2011).

sparse TLS point clouds but its performance drops drastically when density is increased.
Therefore, to overcome these limitations, we designed amore robust approach for detecting planes in TLS point
clouds by leveraging the best features of the sliding voxel
plane detector and use it as a hypothetical plane estima-

9) J. Sandoval, K. Uenishi, M. Iwakiri, K. Tanaka: “Robust, Eﬃcient and Deterministic Planes Detection in Unorganized Point
Clouds Based on Sliding Voxels”, IIEEJ Trans. on Image Electronics and Visual Computing, Vol. 7, No. 2, pp. 67–77 (2019).
10) S. Xia, D. Chen, R. Wang, J. Li, X. Zhang: “Geometric Primitives in LiDAR Point Clouds: A Review”, IEEE Journal of
Selected Topics in Applied Earth Observations and Remote
Sensing, Vol. 13, pp. 685–707 (2020).

tor for diﬀerent voxel sizes. The proposed method canalso can handle point clouds with dozens of millions ofpoints eﬃciently due to its coarse-to-ﬁne approach.

11) A.M.C. Araújo, M.M. Oliveira: “A Robust Statistics Approachfor Plane Detection in Unorganized Point Clouds”, Pattern
Recognition, Vol. 100, Article 107115, pp. 1–12 (2020).

According to the experiments results, the proposed
method with the median ﬁltering option is drasticallymore precise than the conventional methods and has thebest equilibrium between precision and recall. We demon-12) M. Fischler, R. Bolles: “Random Sample Consensus: aParadigm for Model Fitting with Applications to Image Analysis and Automated Cartography”, Communications of theACM, Vol. 24, No. 6, pp. 381–395 (1981).
13) M. Gschwandtner, R. Kwitt, A. Uhl, W. Pree: “BlenSor:
Blender Sensor Simulation Toolbox”, Proc. of the International
Symposium on Visual Computing, pp. 199–208 (2011).

strated that a hierarchical coarse-to-ﬁne method for plane
detection is highly eﬃcient in TLS point clouds and pro-

14) Barcelona Robot Lab Dataset, http://www.iri.upc.edu/
research/webprojects/pau/datasets/BRL/index.php (2021).

vides state-of-the-art precision in the evaluated datasets.
For future work, we are considering improving the recall of the proposed method eﬃciently by conducting ex-15) J. Sandoval, K. Uenishi, M. Iwakiri, K. Tanaka: “Robust
Sphere Detection in Unorganized 3D Point Clouds Using an Efﬁcient Hough Voting Scheme Based on Sliding Voxels”, IIEEJTrans. on Image Electronics and Visual Computing, Vol. 8,
No. 2, pp. 121–135 (2020).

periments on other planarity estimation metrics and abetter quantization method of the 3D space. Moreover,
we believe that the proposed method can be extended
to simultaneously detect other parametric shapes such as16) CGAL, Computational Geometry
https://www.cgal.org (2021).

spheres or cylinders.

17) C. Manning, P. Raghavan, H. Schütze: “Introduction to Information Retrieval”, Cambridge University Press (2009).

Algorithms

Library,(Received May 28, 2021)(Revised September 24, 2021)

References
1) K. Pathak, A. Birk, N. Vaskevicius, M. Pﬁngsthorn, S. Schwertfeger, J. Poppinga: “Online Three-Dimensional SLAM byRegistration of Large Planar Surface Segments and ClosedForm Pose-Graph Relaxation”, Journal of Field Robotics,
Vol. 27, No. 1, pp. 52–84 (2010).

Jaime SANDOVAL (Member)He received his B.E. degree in Computer Systems Engineering from the Universidad delValle del Fuerte (Mexico) in 2009, and his
M.E. degree in Electrical and Electronic Engineering from Shinshu University in 2017. In2020 he received the Dr. Eng. degree from
Shinshu University, and in the same year hejoined AB.do Corporation. His research interests are 3D Point Clouds Processing, Computer Vision and Image Processing.

2) K. Pathak, A. Birk, N. Vaskevicius, M. Pﬁngsthorn, J. Poppinga: “Fast Registration Based on Noisy Planes with Unknown Correspondences for 3-D Mapping”, IEEE Transactions
on Robotics, Vol. 26, No. 3, pp. 424-441 (2010).
3) K. Lenac, A. Kitanov, R. Cupec, I. Petrović: “Fast Planar
Surface 3D SLAM Using LIDAR”, Robotics and Autonomous
Systems, Vol. 92, pp. 197–220 (2017).
4) Y. Zhang, W. Xu, Y. Tong, K. Zhou: “Online Structure Analysis for Real-Time Indoor Scene Reconstruction”, ACM Transactions on Graphics (TOG), Vol. 34, No. 5, pp. 159 (2015).
5) D. Lin, S. Fidler, R. Urtasun: “Holistic Scene Understandingfor 3D Object Detection with RGBD Cameras”, Proc. of theIEEE International Conference on Computer Vision, pp. 1417–
1424 (2013).
6) K. Uenishi, J. Sandoval, I. Munetoshi, K. Tanaka: “VKOP:
3D Virtual Keypoint Detector Adapted to Geometric Struc-

9

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Kazuma UENISHI (Member)

Kiyoshi TANAKA (Fellow)

He received his B.E. degree in Computer Science in 2008, and his M.E. degree in Mathematics and Computer Science in 2014 fromthe National Defense Academy of Japan.
Currently, he is enrolled as a Ph.D. student
in the Interdisciplinary Graduate School of
Science and Technology of Shinshu University
with a major in Systems Development Engineering. He is pursuing research in 3D Point
Clouds Processing.

He received his B.S and M.S. degrees in Electrical Engineering and Operations Researchfrom National Defense Academy, Yokosuka,
Japan, in 1984 and 1989, respectively. In
1992, he received Dr. Eng. degree from
Keio University, Tokyo, Japan. In 1995, hejoined the Department of Electrical and Electronic Engineering, Faculty of Engineering,
Shinshu University, Nagano, Japan, and currently he is a full professor in the academic
assembly (Institute of Engineering) of Shinshu University. He is the Vice-President of
Shinshu University as well as the directorof Global Education Center (GEC) of Shinshu University. His research interests include image and video processing, 3D point
cloud processing, information hiding, humanvisual perception, evolutionary computation,
multi-objective optimization, smart grid, andtheir applications. He is a project leader of
JSPS Strategic Young Researcher Overseas
Visits Program for Accelerating Brain Circulation entitled Global Research on the Framework of Evolutionary Solution Search to Accelerate Innovation started from 2013. Hereceived IEVC2010 Best Paper Award fromIIEEJ, iFAN2010 Best Paper Award from
SICE, GECCO2011 Best Paper Award and
GECCO2015 Best Paper Award from ACMSIGEVO, ISPACS2011 Best Paper Award
from IEEE, Excellent Journal Paper Award
from IIEEJ two times, in 2012 and in 2014,
and Best Journal Paper Award from JSEC in
2012. He is a fellow of IIEEJ. He is a memberof IEEE, IEICE, IPSJ and JSEC. He is theformer editor in chief of Journal of the Institute of Image Electronics Engineers Japan
as well as IIEEJ Transactions on Image Electronics and Visual Computing.

Munetoshi IWAKIRI (Member)He received his B. E. degree in Computer Science in 1993, and received his M. E. degreein Mathematics and Computer Science from
National Defense Academy of Japan in 1998.
In 1999, he joined Department of Computer
Science, National Defense Academy of Japan,
as a Research Associate. In 2002, he received
Dr. Eng. degree from Keio University, Tokyo,
Japan. In 2005 he became Lecturer and in2015 he became Associate Professor in thesame institution. He is pursuing research related to Multimedia Processing and Information Security. He is a member of the Information Processing Society of Japan.

10

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）-- Special Issue on Journal Track Papers in IEVC2021Part II --Aggregative Input Convolution for Large-Scale Point Cloud Semantic Segmentation
Kana KURATA†, Yasuhiro YAO†, Shingo ANDO (Member) †, Naoki ITO†, Jun SHIMAMURA††NTT Human Informatics Laboratories<Summary> We propose an efficient semantic segmentation method for a large-scale point cloud. Previous point-based
semantic segmentation methods to large-scale point clouds have been difficult. This is because those methods infer semantic labels
to all the points used for feature extraction, and large-scale point clouds easily exceed their capacity. To solve this problem, we
propose a novel point-based approach that predicts class labels for a downsampled point cloud and expands the labels to the whole
point cloud by nearest-neighbor interpolation. The key idea of our approach is to give local features derived from the whole point
cloud to each sample point by the newly developed Aggregative Input Convolution (AIC) and convert those features into wider
context features by a point-based model for small-scale point clouds. AIC was experimentally confirmed to improve semantic
segmentation accuracy on a large-scale dataset.
Keywords: point cloud, deep neural network, semantic segmentation

useful

1. Introduction

featuresinhigh-resolution

point

cloudsthroughdownsampling. The local features extracted by the AIC module are

Recent advances in sensor and resolution enhancement

converted to wider context features by a conventional point-based

techniques1) have enabled easier acquisition of high-resolution and

DNN module for small-scale point clouds. Also, we evaluate thespatially large-scale point clouds (104-107 points per scene). Spatial

performance of nearest-neighbor interpolation under several

information for an object (e.g., location and size) can also be

sampling densities to predict semantic labels for an entire point

acquired by detecting the object from an obtained point cloud.

cloud.

One successful technique for extracting objects from point

In summary, our contributions are as follows:

clouds is semantic segmentation using a deep neural network

・We introduced AIC as the DNN module that passes useful features(DNN). DNNs for point clouds have been actively researched sincein high-resolution point clouds to small-scale convolutional DNNs.

Pointnet2), which predicts semantic labels without converting point

・We proposed a semantic segmentation method for large-scale

clouds to other structures (e. g., images or voxels). Methods thatpoint clouds by combination of AIC, Parametric Continuous

can directly process point clouds are called point-based methods.

Convolution

These give a class label for each target point depending on featuresnearest-neighbors.

calculated from coordinates. To classify each point based on its

・We experimentally evaluated the semantic segmentation method

local shape, methods based on the relative coordinates between

on S3DIS dataset.

each inference target point and its neighbor points have been

NeuralNetworks(PCCN)

andcolor-aware

2. Related Workdeveloped 3, 4, 5, 6, 7). However, these methods have high redundancy
and high computational cost for searching neighbor points andPoint cloud semantic segmentation is a computer vision task

calculating features using a DNN for all points in large-scale pointthat divides point clouds into subsets according to semantic

clouds. Thus, the targets of most point-based semantic

meanings. The output of most point cloud segmentation methods issegmentation models are limited to small-scale (less than 104

usually a semantic label for each point. Recent DNN-based

points) point clouds.

methods can be categorized as either methods that rely onintermediate representations or point-based methods.

To solve this problem, we propose a new point-based approachthat predicts class labels for downsampled point clouds and

The former methods include projection and voxel-based

expands the labels to whole point clouds by nearest-neighborarchitecture. Projection-based architectures8, 9, 10) are derived frominterpolation. This paper introduces a new point-based deep neuralmany works on 2D convolutional neural networks (CNNs).

network (DNN) module named Aggregative Input Convolution

Additionally, in voxel based methods, methods to make feature(AIC), which applies local features derived from a whole point

extraction process more efficient have been studied11, 12). OctNet11)

cloud to each inference target point. AIC was designed to preserve

reduced computational cost by efficiently handling empty areas

11

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

using hybrid grid-octree structure13). These methods with datahigh-resolution point clouds. Our method is composed of semantictransformation are based on previous research on graph neurallabel prediction by DNN that includes an Aggregative Input

networks (GNNs) and 2D CNNs. However, due to the

Convolution (AIC) network and PCCN6), followed by color-awaretransformation of the data structure, they cannot directly handle thenearest-neighbor interpolation. We introduce AIC, which is a DNN

detailed shapes represented by continuous coordinates, or

module, to aggregate information for detail point distribution astranslation and rotation in 3D space, etc.

features of a few inference target points. PCCN is a convolutional

The latter, point-based DNNs, have been developed to directly

neural network that requires higher computation cost than discrete

handle continuous coordinate point clouds without conversion to

convolution because it calculates convolution weights from

regular grid structures (e.g. 2D image and 3D voxels). Recent

neighbor point coordinates. Generally, the computation cost of

point-based methods tend to score highly in semantic segmentation

convolution operation in PCCN linearly decreases as the number oftasks. Because of the difference of data structures, convolution

input points decreases. In this study, the input points of PCCN are

operations used in standard convolutional neural networks (CNNs)

restricted to a small scale using AIC and downsampling to reduce

cannot be applied to point clouds. Therefore, various types ofcomputation cost while preserving high-resolution features, andfeature extraction operations for non-grid data have been

reducing classification errors around object boundaries bydeveloped.

color-aware nearest-neighbor interpolation. Figure 1 shows the

One way to process continuous coordinates is to use graphentire DNN architecture.

neural networks (GNNs). GNNs consume graph structured data

3.2 Aggregative Input Convolution (AIC) networkthat consists of inference target nodes and edges defining
relationships between the nodes14, 15, 16). For example, SPG15)

AIC is designed to utilize the rich shape information of aapplies gated recurrent units (GRU) 17) to graphs that consist of

high-resolution point cloud with minimum computational cost byclusters of primitive shapes and edges between clusters.

reducing the computationally expensive process of determining

PointNet2), the pioneer point-based method, directly encodes

neighbor points for all points of large-scale point cloud.

coordinates of each point by pointwise multi-layer perceptron

AIC is a DNN module that aggregates information for detailed(MLP). In PointNet++3), DNN architecture was improved topoint distribution as features of a few inference target points. It iscapture local shapes by encoding relative coordinates of neighborused in combination with a semantic segmentation DNN. AIC input

points around each reference point using pointwise MLP on relativeis the relative coordinates and colors of neighbor points of a few

coordinates to reference points. As well as PointNet++, KCNet4),inference target points.

PCNN5),

PCCN6)

A-CNN7)

also extract features based on

AIC searches the neighbor points for inference target points

relative coordinates between target points and its neighbor points.

from high-resolution point clouds before downsampling. In thisIn particular, PCNN5) and PCCN6) build CNN-like architectures forpaper, inference points are chosen by random sampling. In Fig. 1,

non-grid data by developing ways to give convolutional weights.

“sparse indexing” corresponds to this operation, and includes

PCNN5) learns convolution weights according to the order ofneighbor points search and indexing of coordinates and features ofand

PCCN6)

calculates the weights from relative

neighbor points for inference points. We incorporated dilation into

coordinates. On the other hand, A-CNN7) proposed annularsparse indexing, inspired by the dilated convolution method18). The

convolution operators to better capture local neighborhood

dilation factor is a coefficient that determines the distance between

geometry in point clouds.

points. For example, if the dilation factor is 3, neighboring points

distance, and thenIn conventional point-based methods such as those described

are selected from the closest point every three points. “Concatenate”

above, it is common to give predictions to all inference target points

is a function that combines the elements of multiple vectors in thebased on a directed graph between every target point and theirfeature channel axis. The number of neighbor points per target

neighbor points. Since searching for neighbor points and learning

point is set to be larger than a segmentation network, to cover thefeatures for a large number of points is computationally expensive,wide sphere of the entire point cloud. Figure 2 visualizes the AIC

downsampling is often used for preprocessing. However,

input. The blue points in Fig. 2 are inference target points which are

high-resolution shape features are lost by coarse downsampling.

given features by AIC, while the other points are their supportingneighbor points. Additionally, to adapt the method to various data,

3. Method

the number of AIC channels for shape and color features are

3.1 Overview

designed to be adjusted according to the characteristics of the target

Our model is designed to improve efficiency by reducing thedata. AIC extracts shape and color features separately and combinesthem into AIC output features as shown in Fig. 1.

number of inference points without losing useful features in

12

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 1 Proposed DNN architecture consists of AIC and semantic segmentation networks. N, Q, K mean the total number of input points, inference points
and neighbor points. Numbers in bracket are layer sizes. Labels' size "U" is the number of recognition target classes. White boxes mean input orintermediate data, blues are processing, and dotted lines show data flow only in the training phase.

The coordinates of a target point and its neighbor points, andthe RGB vector of each neighbor point are transformed bypointwise MLPs and are aggregated around each inference point by
a pooling layer. This operation encodes local distribution of points
by MLP as well as PointNet++3) but differs in that it is applied to ahigh-resolution point cloud before downsampling, and only
representative points with the derived features are given.
In our implementation, layer sizes were set as shown in Fig. 1,
and pooling layers were max pooling. Since indoor objects come ina variety of colors, the number of channels for shape and color
features was manually set to 224 and 32 respectively to cover shapevariations preferentially. Such separation of shape and color enables
adjustment of the number of channels for shape and color features.
The appropriate number of neighbor points and the dilation factor

Fig. 2are experimentally evaluated in Section 4.4.

AIC input at the number of inference target points 2048
and neighbor points 128

3.3 DNN architecture
AIC and PCCN are connected as shown in Fig. 1 and trainedtargeted in the nearest neighbor search and AIC feature extraction

together. PCCN, composed of three continuous convolution blocks

processes. This network is trained with cross-entropy loss between

with residual connections, receives AIC output feature and outputs

PCCN output and true label shown as “Labels” in Fig. 1 ofa class label prediction. Our method consumes lower computational

inference target points using an Adam optimizer.

cost than methods that process all points equally since only a small
subset (less than 104 points per scene) of the entire points are

13

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 3 Results for nearest-neighbor interpolation and color-aware nearest-neighbor interpolation under the 1/256 sampling rate.

3.4 Color-aware nearest-neighbor interpolation

Table 1 Accuracy of color-aware and standard nearest-neighborinterpolation.

In our method, semantic predictions by DNN of downsampled
point clouds are expanded by color-aware nearest-neighbor

Samplingrate

OA (standard)

OA (color-aware)computational cost to expand prediction from a few points and thus

1/512

91.89%

93.02%

is often used in point cloud processing. However, interpolation

1/256

94.39%

95.05%

errors around object boundaries become larger as the number of

1/128

96.14%

96.43%

interpolation.

Nearest-neighborinterpolation

requireslow

points having labels decreases. To avoid such errors, we adopt
color-aware

nearest-neighborinterpolation.

Thismethodof indoor objects. This paper identified these 13 classes and an

determines the closest point based on the Euclidean distance ofadditional category, "Other," to avoid exceptions when there areconcatenated vectors of coordinates and normalized RGB as

points with no labels. We used all scans of S3DIS for evaluation infollows.

Section 4.2 and assigned scans excluded Area 5 for training andᇱ    min ⁄max(1) ᇱ    min ⁄max

(2)

ᇱ    min ⁄max

(3)scans of Area-5 for the test in Sections 4.3 and 4.4, following Wanget al.6). As S3DIS contains various sizes of data (from 85 thousand
to 7 million points), we split scans of large rooms while limiting themax number of points to 524,288.

(4)   ଶ   ଶ  ଶ  ′ଶ  ′ଶ  ′ଶwhere Red, Green, and Blue are each channel of RGB, and x, y, z

4.2 Interpolation accuracy using color-aware
nearest-neighbor

are values of three-dimensional coordinates. The functions “max”
and “min” take the maximum and the minimum values from allFirst, we tested color-aware nearest-neighbor interpolation of

input points.

multiple sampling density in all scans of S3DIS dataset. This

4. Experimentsinterpolation is non-learning and it is independent from thesemantic segmentation by the DNN. In this experiment, the groundIn this Chapter, we report the results of three experiments:
accuracymeasurementofcolor-aware

truth labels are interpolated from randomly sampled points to thenearest-neighborwhole points by several methods including ours. And we evaluatedinterpolation, verification of accuracy improvement by AIC, andthe interpolation methods by the label accuracies. Table 1 showsparameter study of AIC. All experiments are conducted on an

the overall accuracy of standard and color-aware nearest-neighbor

NVIDIA RTX Titan GPU using an S3DIS dataset. In experimentsinterpolation. Sampling rate means a reduction ratio parameter ofdescribed in Sections 4.3 and 4.4, DNN models were trained for

AIC. Overall accuracy (OA) increased using the color-aware

300 epochs and the batch size for training was 4.

method instead of the standard method. Also, Fig. 3 shows that

4.1 Dataset

interpolation errors around object boundaries were reduced overall
by color-aware interpolation. In detail, interpolation accuracy

S3DIS is a large-scale dataset of indoor space that contains 3D

around the boundary between a wall (yellow-green) and a bookcase

scans of 271 rooms in 6 areas that contain multiple types of rooms(dark green) is significantly improved by the color-aware method.

(hallway, conference room, water closet, storage, and office). Each

However, interpolation errors still remain around some boundaries,

point has an RGB attribute and a semantic label from 13 categories

14

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

for example, between a wall and a window (gray). Thus, it isTable 2 Quantitative results of our model and baselines fordownsampled points

necessary to adjust the strength of color awareness by adjusting the
RGB value normalization range.

Method

OA

mAcc

mIoU

PCCN w/ pre-sampling

75.31%

61.41%

52.59%

We evaluated the effect of AIC by comparing the performance

Ours w/ pre-sampling

76.18%

69.81%

60.00%

of our architecture with two baselines. The first is PCCN with

Ours w/o pre-sampling

81.62%

71.14%

62.16%

4.3 Semantic segmentation accuracy

downsampling in preprocessing phase (pre-sampling), using a
DNN with the same structure as the semantic segmentation
network of our model (“PCCN w/ pre-sampling” in Table 2). The

Table 3

Accuracy for all points

second is our model with pre-sampling (“Ours w/ pre-sampling” inTable 2), the same structure as our model but with downsampled

PCCN6)

points input to AIC. This paper adopted AIC sampling rate of 1/256

Ours w/o pre-sampling (interpolated)

OA

mAcc

mIoU

-

67.01%

58.27%

79.75% 68.79% 59.09%

to balance efficiency and accuracy according to the results ofSection 4.2. Also, the pre-sampling rate was 1/256 to set the
number of PCCN input points in each model to the same value. We

Table 4 Evaluation results of AIC parameters and accuracy

employed this experimental setting to evaluate the effectiveness of

The number ofneighbor points
and dilation

extracting features from high-density points by AIC. In the case of
the two baselines, the number of all the input points in Fig. 1. “N”

OA

mAcc

mIoU

equals “Q.” Note that this evaluation is independent from the64, 280.84%

71.32%

61.93%

interpolation evaluation in Section 4.2.

64, 378.18%

67.04%

57.38%

Table 2 is the result for the downsampled points before

128, 180.29%

70.09%

60.48%

interpolation. In evaluation with the downsampled DNN inference

128, 274.78%

67.12%

57.59%

target points, our model without sampling before AIC (“Ours w/o

128, 381.62%

71.14%

62.16%

pre-sampling” in Table 2) achieved a higher score in overall
accuracy (OA), mean class IoU (mIoU) and mean class accuracy

points, our model has neighbor points for only a small number of(mAcc) than PCCN with downsampled points and Ours w/

inference target points. Therefore, our method reduces the

pre-sampling (Table 2).

maximum array size of internal data stored in RAM to 1/256 (AIC

The only difference between our model w/o pre-sampling anddownsampling rate) of that of directly applied original PCCN while

w/ pre-sampling is whether the neighbor points in AIC are selected

improving accuracy.

from a high-resolution point cloud or from a pre-sampled point

4.4 Parameter study of AIC

cloud. Therefore, the method of assigning features from thehigh-density point cloud is confirmed to improve the accuracy.

Next, we evaluated the approximate number of AIC neighbor

In the case of predicting labels to an entire point cloud, ourmethodwithcolor-aware

nearest-neighborpoints and dilation (Table 4). For a sampling rate of 1/256, the wallinterpolation

hanging paintings on right side of Fig. 4 (a) were identified by ouroutperformed the result of original PCCN in Wang et al.6) (Table 3).

method, although they were not identified on the left in Fig. 4 (a) or

Table 3 listed the score of the original PCCN paper that shows

the right in Fig. 4 (b). Particularly, some areas around the painting

mAcc and mIoU only. Unlike the experiment in Section 4.2, thison the right in Fig. 4 (a) were classified as a whiteboard. Theseexperiment operated interpolation based on inferred labels. In thisresults indicate that our architecture was capable of capturingexperiment, we followed the training and testing procedure used in

texture features, but texture variations were not sufficiently learned.

Wang et al.6). However, we used a different preprocessing method

Hence, it is necessary to study methods for learning enoughof data cutting. While Wang et al.6) split data into cubes of afeatures of objects with many texture variations and objects withspecific size in preprocessing, we split data by picking a fixedsmall amounts of data. Similar to the results described in Section

number of points in order of closeness from a randomly determined

4.2, some boundaries could not be separated correctly bypoint to avoid extra sampling in scenes with high-density points. In

color-aware nearest neighbor due to similarity in color between

contrast to original PCCN, which has neighbor points for all input

objects.

15

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 4

Interpolated semantic segmentation result at sampling rate 1/256, 128 AIC neighbor points and AIC dilation 32)

5. Conclusion

C. R. Qi, H. Su, K. Mo, L. J. Guibas: “PointNet: Deep Learning on
Point Sets for 3D Classification and Segmentation”, Proc. of the IEEE

We presented DNN architecture with a novel AIC module that

Conference on Computer Vision and Pattern Recognition (CVPR 2017),

aggregates information of detailed point distribution as features of app. 652–660 (2017).

few inference target points. Through the experiments on S3DIS

3)

C. R. Qi, L. Yi, H. Su, L. J. Guibas: “PointNet++: Deep Hierarchicaldataset, we find out that high resolution features inputs captured by

Feature Learning on Point Sets in a Metric Space”, Proc of Advances in

AIC improve semantic segmentation accuracy. In terms of the

Neural Information Processing Systems 30 (NIPS 2017) (2017).

overall method, our method reduces the maximum array size of

4)Yiru Shen, Chen Feng, Yaoqing Yang, Dong Tian: “Mining Pointinternal data to 1/256 (AIC downsampling rate) of that of directly

Cloud Local Structures by Kernel Correlation and Graph Pooling”,

applied PCCN6) while improving the accuracy. Experimentations

Proc. of the IEEE Conference on Computer Vision and Patternon other indoor/outdoor large-scale datasets and on the combination

Recognition (CVPR 2018), pp. 4548–4557 (2018).

of AIC with other DNNs instead of PCCN are future issues to

5)

Y. Li, R. Bu, M. Sun, W. Wu, X. Di, B. Chen: “PointCNN: Convolution

quantitatively evaluate the performance and efficiency of our model.

on X-transformed Points”, Proc. of Advances in Neural Information

Also, a future research topic is to develop methods for learning

Processing Systems 31 (NeurIPS 2018) (2018).

enough features of objects with many texture variations and objects

6)

with small amounts of data.

Parametric Continuous Convolutional Neural Networks”, Proc. of theIEEE Conference on Computer Vision and Pattern Recognition (CVPR

References
1)

S. Wang, S. Suo, W.-C. Ma, A. Pokrovsky, R. Urtasun: “Deep2018), pp. 2589–2597 (2018).

Y. Yao, M. Roxas, R. Ishikawa, S. Ando, J. Shimamura, T. Oishi:

7)“Discontinuous and Smooth Depth Completion with Binary

A. Komarichev, Z. Zhong, J. Hua: “A-CNN: Annularly Convolutional
Neural Networks on Point Clouds”, Proc. of the IEEE Conference on

Anisotropic Diffusion Tensor”, IEEE Robotics and Automation Letters,

Computer Vision and Pattern Recognition (CVPR 2019), pp. 7421–

Vol. 5, No. 4, pp. 5128–5135 (2020).

16

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

pp. 7440–7449 (2019).

7430 (2019).
8)

9)

F. J. Lawin, M. Danelljan, P. Tosteberg, G. Bhat, F. S. Khan, M.

17) K. Cho, B. van Merriënboer, Ç. Gülçehre, D. Bahdanau, F. Bougares,

Felsberg: “Deep Projective 3D Semantic Segmentation”, Proc. of 17th

H. Schwenk, Y. Bengio: “Learning Phrase Representations Using RNN

international Conference on Computer Analysis of Images and Patterns

Encoder–decoder for Statistical Machine Translation”, Proc. of the(CAIP) (2017).

2014 Conference on Empirical Methods in Natural Language
Processing (EMNLP 2014), pp. 1724–1734 (2014).

A. Boulch, B. Le Saux, N. Audebert: “Unstructured Point Cloud
Networks”,

18) F. Yu and V. Koltun: “Multi-Scale Context Aggregation by Dilated

Eurographics Workshop on 3D Object Retrieval (3DOR 2017) (2017).

Convolutions”, Proc. of International Conference on Learning

Semantic

Labeling

Using

Deep

Segmentation

Representations (ICLR) (2016).

10) M. Tatarchenko, J. Park, V. Koltun, Q.-Y. Zhou: “Tangent Convolutions
for Dense Prediction in 3D”, Proc. of the IEEE Conference on

AppendixComputer Vision and Pattern Recognition (CVPR 2018), pp. 3887–
3896 (2018).

APP1. DNN Structure of PCCN

11) L. Tchapmi, C. Choy, I. Armeni, J. Gwak, S. Savarese: “SEGCloud:PCCN6) is a convolutional neural network with residual

Semantic Segmentation of 3D Point Clouds”, Proc. of 2017

connection. Convolution weight is calculated as a continuous value

International Conference on 3D Vision (3DV), pp. 537–547 (2017).

from the relative coordinates of neighbor points.

12) G. Riegler, A. O. Ulusoy, A. Geiger: "OctNet: Learning Deep 3D
Representations at High Resolutions", Proc. of the IEEE Conference on

In our method, PCCN is used to give predictions to small-scaleComputer Vision and Pattern Recognition (CVPR 2017), pp. 3577–

point clouds that are downsampled in AIC rather than to the entire

3586 (2017).

point cloud.

13) A. Miller, V. Jain, J. L. Mundy: “Real-time Rendering and Dynamic

The original PCCN model used Wang et al.6) includes eight

Updating of 3-d Volumetric Data”, Proc. of the Workshop on Generalconvolution layers with residual connections and has a global

Purpose Processing on Graphics Processing Units (2011).

pooling layer connected after the last convolutional block (Fig.
APP1). Wang et al6) adopts separable convolution.

14) Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, J. M. Solomon:

DNN structure of PCCN used in this paper is composed of six“Dynamic Graph CNN for Learning on Point Clouds”, ACM Trans.

continuous convolution layers with dropout layer. The global

Graph Vol. 38, No. 5 (2019).
15) L. Landrieu, M. Simonovsky: “Large-scale Point Cloud Semantic

pooling layer was excluded from our model because of the effects

Segmentation with Superpoint Graphs”, Proc. of the IEEE Conferenceof accuracy improvement were not to be experimentally confirmed.
We set the number of neighbor points to 32 following theon Computer Vision and Pattern Recognition (CVPR 2018), pp. 4558–

original paper, feature channels to 256 based on the memory cap of4567 (2018).

VRAM, and dropout rate to 0.3 experimentally.

16) L. Landrieu, M. Boussaha: “Point Cloud Oversegmentation with
Graph-Structured Deep Metric Learning”, Proc. of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR 2019),

17

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. APP1 DNN architecture consists of PCCN.

Shingo ANDO (Member)He received the B.E. degree in electricalengineering from Keio University in 1998. Hereceived the Ph.D. degree in engineering from Keio
University in 2003. In 2003, he joined NTT. He has
been engaged in research and practical application
development in the fields of image processing,pattern recognition, and digital watermarks. He is amember of IEICE, ITE, and IIEEJ. He is currently
an associate professor at Shonan Institute ofTechnology.

（Received May 28, 2021）（Revised Mar. 3, 2022）

Kana KURATA
She received a B.S. in earth and planetarysciences from Nagoya University in 2016 and an
M.E. in environmental studies from Nagoya
University in 2018. She joined NTT in 2018,
where she has been engaged in research incomputer vision and pattern recognition.

Naoki ITOHe received the M.E. from Toyohashi Universityof Technology, Aichi, in 2001. He joined NTT in2001 and engaged in research on character
recognition. He moved to NTT EAST in 2004 andengaged in the development of security systems. He
moved to NTT Cyber Space Laboratories (now,NTT Media Intelligence Laboratories) in 2008. He
has been working on the development of real-world
digitalization technologies.

Yasuhiro YAOHe received the B.S. and M.E. degrees fromThe University of Tokyo, Japan, in 2007 and2010, respectively, where he is currently pursuing
the Ph.D. degree in information studies. In 2010,
he joined NTT as a Researcher. From 2013 to2016, he was a Cloud Solution Architect at
Dimension Data APAC, Singapore. He iscurrently a Senior Research Engineer at NTT
Human Informatics Laboratories. His researchinterests include computer vision and sensor
fusion.

Jun SHIMAMURAHe received a B.E. in engineering science from
Osaka University in 1998 and an M.E. and Ph.D.
from Nara Institute of Science and Technology in2000 and 2006. He joined NTT Cyber Space
Laboratories in 2000. He is currently senior researchengineer, supervisor of scene analysis technology at
NTT Human Informatics Laboratories, Japan. His
research interests include computer vision andmixed reality. He is a member of IEICE.

18

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）-- Special Issue on Journal Track Papers in IEVC2021Part II --Registration of Histopathological Heterogeneous Stained Images
Utilizing GAN Based Domain Adaptation TechniqueTanwi BISWAS†, Hiroyuki SUZUKI††, Masahiro ISHIKAWA††† (Member), Naoki KOBAYASHI††† (Fellow), Takashi OBI†
†Tokyo Institute of Technology, ††Gunma University, †††Saitama Medical University<Summary> Registration of histopathological images obtained from different staining techniques is very challenging becauseof much difference of their color information. In this study, we propose a promising image registration method that can overcome
the color difference of H&E and EVG stained images by means of GAN-based color conversion. Our proposed method consists of
two main parts: one is GAN based unsupervised domain adaptation network for converting H&E stained image to EVG stained
image which has similar distribution with the original EVG stained image and the other is SURF feature based registration
framework which provides the registered EVG stained image leveraging the generated EVG stained image obtained from thedomain adaptation network. The experimental result shows that our proposed method is able to provide better registration result
than the conventional method where domain adaptation technique is not incorporated.
Keywords: image registration, histopathological image processing, unsupervised domain adaptation, generative adversarial
network (GAN)

Previous studies have introduced different non-learningand learning based approaches for addressing the problem ofregistration of differently stained images1),3-11). The nonlearning based approaches can be further categorized asintensity-based3), feature-based1) and both feature andintensity-based4) registration methods. Comparing theperformances of different non-learning based methods, thehybrid method combining both intensity and features has been
observed to provide better result at the cost of large processing
time5). Also, there is still significant gap between the obtained
result and the result expected by the human annotator.
Recently M. Wodzinski and A. Skalski have proposed aneffective method for the registration of histology samples
obtained from multiple stain types6). Their proposed
methodology follows multiple steps including preprocessingof input images, feature-based initial alignment, exhaustive
rotation search, refinement of initial affine transformation andfinal calculation of deformation field. Though their proposed
method provides better registration result, the method is toocomplicated and time consuming to be applied in practicalapplications. Also, according to the authors, the key
component of their successful registration lies in the accuracyof initial alignment where our proposed method can be a great
substitute. Nowadays, different learning-based approaches
leveraging machine learning or deep learning, have also
proved their effectiveness to be applied for the purpose of
medical image registration7-11). But, most of these learning

1. Introduction
Histopathological staining is an indispensable
preprocessing step of preparing tissue slides for observing
under the microscope where coloring reagent is used tovisualize important features of tissue samples. Different typesof histological stains are used to particularly highlightdifferent tissue components. For the purpose of disease
analysis and diagnosis, it is required to compare these tissueslides with different stains visually. Though these differently
stainedhistopathologicalimageshave
identical
morphological information, they are neither aligned nor scale
adjusted because of being captured by different device indifferent time. Registration makes these similar anatomicallystructured images aligned together which corresponds to eachother and facilitates easy visual comparison of these tissueslides1). Also, for digital stain conversion or segmentation ofparticular tissue components with supervised learning,registration is considered to be a very important preprocessing
step that makes the image data ready for further processing2).
But, registration of these different types of stained images isvery challenging because of having possible deformations
during the preparation of tissue, repetitive pattern in texture,large image size and most importantly difference inappearances with diverse color information. This study aims
to provide an efficient method for the automatic registrationof these heterogeneous stained images.

19

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

based methods are highly dependent on having annotatedsegmentation data7-8) or prior spatially aligned corresponding
images10-11) as training data which is time-consuming,
expensive and labor-intensive to collect. In our approach, wehave also leveraged deep learning model for the adaptation of
the domain of differently stained images in an unsupervisedmanner to aid the registration.
In the field of machine learning or deep learning, most of
the analyses are performed using the dataset where training
and testing data are from same distribution having similarfeatures and are able to provide outstanding result. However,in real life applications the testing data may vary abruptly
from the training data distribution because of the lighting
condition, environmental issue etc. and in these cases thetrained deep learning models are not generalized well andseem to provide poor performance12-13). Unsupervised domain
adaptation technique can be a very useful solution in thisregard which tackles the domain shift problem between the
training (source domain) and testing (target domain) data
without requiring labeled data from the target domain and isable to bring significant improvement to the performance ofdeep learning models in real life applications. Numerous
researches have been performed to develop different methodsof domain adaptation which have proved to provide improved
results in various computer vision cases such asclassification14), object detection15) and segmentation16). Also,in medical image analysis, either supervised or unsupervised
domain adaptation technique seems to be very useful fordifferent objectives such as cancer classification17), tumor
segmentation18), skin disease classification19), retina
segmentation20) and so on. However, fewer works have been
observed where domain adaptation has been leveraged for the
purpose of medical image registration21). Specially, for theregistration of multiple stain types, to the best of ourknowledge, no prior work has been found where domain
adaptation has been utilized. Whereas we think of domain
adaptation as a promising technique of reducing the gap in theappearances of the differently stained images thus aiding tofind similar correspondences more easily. At the same time,
we also believe that, adapting the domain of the histologicalimages is a very challenging task due to the significant gap
between the color information of two different domains.
In this study, we have proposed an automatic image
registration method where GAN based unsupervised domain
adaptation model has been incorporated to address the domain
shift problem between two heterogeneous stained images andhand crafted feature detection technique like SURF feature22)
has been adopted to perform registration without requiringFig.1 Proposed registration methodprior spatially corresponding training data. The details of the
proposed method will be discussed in the next section and theother parts of this paper are organized as follows: section 3
describes the experimental details including data acquisitionand preprocessing, section 4 represents experimental result,section 5 represents discussion and finally section 6 concludes
our work.

2. Proposed Method
This study aims to provide a registration framework which
is able to overcome the problem of detecting and matching
features from images of different stains because of the huge
difference in the color information. For the implementation of
our proposed method, two different types of stained images
such as H&E (Hematoxylin and Eosin) stained and EVG(Verhoeff-Van Gieson) stained images are deployed in thisresearch. The proposed method is shown in Fig. 1. It contains
two stages including domain adaptation as a preprocessing
step for registration and the registration itself. The domain
adaptation part addresses the domain shift problem because of20

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig.2 Methodology of adopting the domain of H&E to the domain EVG

designed following the ResNet architecture24) and thediscriminators are based on PatchGAN architecture25). The
full objective function to train the model is described in Eq.(1).

color variation between differently stained images. This isdone by adapting the domain of H&E to its equivalent domain
EVG using GAN based translation. It reduces the distance ofthe distribution of color information between differently
stained images which enables our registration framework to
detect and match greater number of similar features between
them. The other part detects and matches local features such
as SURF between the original and translated EVG stained
image and produces the registered EVG stained image bytransforming the original EVG stained image.

ℒ

ℒ

.

𝜆ℒ

.

𝛾ℒ

.

(1)

represent
Where, ℒ . , ℒ. , ℒ . and ℒ
adversarial loss, cycle-consistency loss, identity loss and thetotal loss to train the model respectively. Also, 𝜆 and 𝛾 are
two scalar values which have been multiplied with cycleconsistency loss and identity loss respectively. Adversarial
loss contributes to generate realistic H&E and EVG stained
images which have similar distribution as the original dataset.
The cycle-consistency loss tries to ensure that the anatomical
structure of the input source domain and generated target
domain remains unchanged. The identity loss contributes toimitate the exact color of the target domain. The adversarial
loss has been calculated in terms of Mean Square Error (MSE)and the cycle-consistency and identity loss are calculated interms of Mean Absolute Error (MAE). CycleGAN has been
deployed in this study because of its ability to perform domain
adaptation in a fully unsupervised way where prior paired(aligned and scale adjusted) training data is not required. Also,
it facilitates the translation of color information from one
domain to another while remaining their structural
information unchanged.

2.1 Domain adaptation network
The objective of the domain adaptation network is toaddress the domain shift problem between H&E and EVG
stained images by converting the images from H&E domain
to its equivalent EVG domain so that the similarity in the color
information between the H&E and EVG stained images are
increased. This will contribute greatly to the detection andmatching of larger number of local features by the registration
framework in the later part. Our proposed domain adaptation
network is based on CycleGAN model23). The methodologyof adapting the domain of H&E stain to the domain of EVG
stain has been shown in Fig. 2. The network consists of two
generators and two discriminators. The ‘Generator HE-toEVG’ takes the H&E stained image as input and generates thecorresponding EVG stained image as output and the‘Generator EVG-to-HE’ takes the EVG stained image as
input and produces the equivalent H&E stained image as
output. The ‘Discriminator HE’ tries to identify if the input
image is coming from the original dataset of H&E domain orfrom the output of the ‘Generator EVG-to-HE’ and the‘Discriminator EVG’ tries to identify if the input image iscoming from the original dataset of EVG domain or from theoutput of the ‘Generator HE-to-EVG’. The generators are

2.2 Registration framework
Original EVG stained image cropped from the WSI, ISourand the converted EVG stained image, ITrans obtained from theoriginal H&E stained image, IRef has been used as the input of
the registration framework. For the registration purpose, styletransferred H&E stained image, ITrans has been considered astarget image according to which the registered image will beobtained. On the other hand, cropped EVG stained images

21

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

are hyperspectral images captured with hyperspectral cameraof NH3 EBA JAPAN CO. LTD with which an optical
microscope BX-53 by Olympus Corp. and a white LED are
attached. Hyperspectral H&E stained images are obtainedusing the transmittance information of the tissue
corresponding to the wavelength 350-1100nm with 5nm
interval (151 channels) having the image size 480×752. Afterremoving redundant information, we considered thehyperspectral image within the visible wavelength from 420
to 720 nm of 61 channels. EVG stained images are RGB
Whole Slide Image (WSI) obtained using Hamamatsu
photonics K.K. For the sake of retaining lower computational
complexity, H&E stained hyperspectral images have been
converted to sRGB image. The hyperspectral image is at first
converted to 𝑋, 𝑌 and 𝑍 tristimulus values using
illumination spectrum D65 which is a CIE standard illuminantcorresponding to daylight and 1931 CIE XYZ color matching
functions27). 𝑋, 𝑌 and 𝑍 tristimulus values are thenconverted to sRGB image using the XYZ to sRGB
transformation matrix.
H&E and EVG stained images have a consistentanatomical structure because EVG staining has been
performed after bleaching the tissue where H&E staining was
imposed prior. Same area as H&E stained image has been
cropped manually from EVG stained WSI to prepare thedataset with a set of H&E and EVG image which are
unaligned and not scale adjusted (unpaired). Then, EVG
stained image is resized so as to be 480×752. An example ofpreparing a set of H&E and EVG stained images is shown in
Fig. 3.

have been considered as the source image, ISour which isneeded to be transformed. To provide lower computational
complexity and higher effectiveness, hand crafted featuresuch as SURF feature22) has been leveraged in our registration
framework. Among different types of hand crafted features,
we have utilized SURF feature because of its faster
performance, robustness and scale and rotation invariant
property. SURF feature is detected and extracted both from
ITrans and ISour. The features obtained from these images arethen matched by calculating the distance between the featuresets. Outliers in the matched features are excluded andgeometric transformation of type affine is calculated where amatrix contains the information of translation (horizontal orvertical shifting), rotation, scaling and shearing for
determining the mapping of the position of each pixel fromthe source image to the corresponding position of that pixel inthe target image. This is done using the M-estimator SAmple
Consensus (MSAC) algorithm26) which is a variant of popular
RANSAC (Random Sample Consensus) algorithm and able
to provide improved result than RANSAC by incorporating abetter cost function. Though sometimes MLESAC(Maximum Likelihood Estimation Sample Consensus)
provides better accuracy than MSAC, it takes morecomputational complexity with longer processing time.
Considering the balance between processing time andperformance, we chose to use MSAC. Finally, the original
EVG stained image is warped according to the information ofgeometric transformation and the registered EVG stained
image is obtained.

3. Experiment

3.2 Experimental details

3.1 Experimental data

For the implementation of the first stage of our proposed
methodology, the domain adaptation CycleGAN network has
been trained with images from two different domains: H&E
and EVG. These training images have been prepared bycropping the overlapping area of size 256×256 from the set of
H&E and EVG stained images which have been preparedpreviously. After applying the image augmentation techniquesof rotation, flipping and brightness change on these images, a
total of 9952 images in each domain has been finalized to train
the model. Among the training images, H&E and EVG
stained images may have similar biological structures but they
do not spatially correspond to each other (unpaired). These
training data for the domain adaptation network have beencollected from two different tissue samples. While training thedomain adaptation network, empirically it has been found that

This study involves two different types of stained images:
H&E stained image and EVG stained image. Though H&E
stain is the most commonly used staining technique, elastic
and collagen fiber cannot be differentiated using it because
both of them are shown in pink color. On the other hand, EVG
shows elastic in deep blue color and collagen in orchid color
thus enables the easy discrimination of these two tissue
components. For visual comparison of these two types ofstains images or for digital conversion of these stain types andsegmentation of particular tissue component, proper
registration of these images plays an important role for further
analysis or processing2).
The H&E and EVG stained images used in thisexperiment are of human pancreatic tissue specimens whichhave been obtained from Biomax Inc. H&E stained images

22

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

adaptation part of this work has been implemented in Python
using Pytorch toolkit and registration part has been
implemented in Matlab using Computer Vision and Image
Processing toolbox.

4. Result
Our proposed domain adaptation network doesn’t change
the morphological information of the tissue. It only changes
the color information of the tissue components to adapt thetarget domain (EVG stained) from its source domain (H&E
stained). Figure 4 represents the result of our domain
adaptation network from which we can see that the converted
EVG stained image has been adapted well to the original EVG
stained image domain in terms of color information while
retaining the morphological information same as the input
H&E stained image. Now, the color information of theconverted EVG stained image and the original EVG stained
image has more similarity than that of original H&E andoriginal EVG which has enabled the registration framework
to obtain greater number of correctly matched features fromthe first pair than the later one. This is shown in Fig. 5 where(a) represents the correctly matched features between theoriginal EVG and original H&E without domain adaptationand (b) represents the correctly matched features between theoriginal EVG and converted EVG using the domain
adaptation network. By comparing these two, one can observethat larger number of matched features have been found forthe pair (b) than that of (a). For the visual evaluation of theregistration result, overlapping view of H&E stained image
and EVG stained image before and after registration with andwithout domain adaptation network has been represented inFig. 6. The magenta color represents H&E image and thegreen color represents EVG image. From Fig. 6 (a), we cansee that, though H&E stained and EVG stained image have
similar biological structure, they are not spatially aligned. Fig.
6 (b) and (c) represents the superimposed view of original
H&E stained image and registered EVG stained image
without and with domain adaptation network respectively. In
the superimposed image, where H&E and EVG stained
images are not aligned properly are looking blurred withdominating green color. From the zoomed view, it can be said
that H&E and EVG stained images have been aligned(registered) more accurately with our proposed domain
adaptation based registration method than the method where
domain adaptation is not applied.
Apart from visual evaluation, we have also determined theregistration result in terms of structural similarity (SSIM) and
Mutual Information (MI). As the name indicates, SSIM

(a)

(b)

(c)
Fig.3 Preparing dataset;(a) H&E stained image, (b) EVG Whole

Slide Images (WSIs), and (c) EVG stained image with same
biological structure as H&E is cropped from WSI

the model is generalized better for setting the value of λand γ of Eq. 1 to 5 and 0.5 respectively. The CycleGAN
domain adaptation model is trained on the GPU Quadro RTX
6000 for 81 epochs with training image batch size 1.
For the registration purpose, 85 image pairs of H&E
stained and EVG stained (cropped from WSI) from four
different tissue samples have been prepared. H&E stained
image, IRef of size 480×752 has been converted to its
corresponding EVG stained image, ITrans using the trained
domain adaptation CycleGAN model. This domain adapted
H&E stained image, ITrans and original EVG stained image,
ISour of size 480×752 has been used as the input of registration
framework. According to the information of calculated
geometrical transformation, ISour has been transformed toobtain the registered EVG stained image. The domain

23

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

(a)

(b)

Fig.4 Result of domain adaptation network;(c)(a) Input H&E, (b) Converted EVG, and (c) Original EVG image

Number of matched features = 53

Number of matched features = 71

Number of matched features = 50

Number of matched features = 96

(a)

(b)

Fig.5 Matched feature between (a) Original H&E and original EVG, and (b) Converted EVG and original EVG

registration result for all the test images of four different tissue
samples (T.S.1-4) have been recorded separately. Table 1
summarizes this registration result in terms of structuralsimilarity (SSIM) and Table 2 summarizes the registration
result in terms of MI comparing our proposed method (Prop.
Meth.) with the baseline SURF feature-based registration
without domain adaptation network (W/O DA) i.e.,
registration without stain conversion. In each cell of the table,
upper and lower values represent averages and standard
deviations respectively. For both cases of SSIM and MI,
higher value indicates better registration. From Table 1 andTable 2, we can see that for every tissue sample, our proposed
method has provided better registration result than thebaseline method.

determines the similarity in the structural information
between two images whereas MI indicates how well signal
intensity of one image can be predicted when another isknown. Both SSIM and MI are considered as good matching
metrics of images which measures the spatial alignment
between two images. Because of the non-deterministic
characteristics of MSAC algorithm, there are a little variationin the result among different runs of the registration
framework. To address this problem, we run each program forthe registration of a single pair of images 3 times. In order toperform a fair comparison between our proposed and baseline
method by considering the best possible performanceprovided by each method, among these three consecutive runs
we have recorded the maximum resultant value for each pairof images. Then the average and standard deviation of the24IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

This has enabled our domain adaptation network to produce
an intermediate domain image which has exact samestructural information as original H&E stained image andsame color information as original EVG stained image.
Because of the adversarial training nature of GAN, sometimes
it may generate very smooth noiseless image but which hasless similarity with the target domain color. On the other hand,
it may generate image with very close adaptation of target
domain color but introducing noises into the image. So,
careful consideration is required while choosing the trained
domain adaptation model for conversion purpose. Here, wehave concentrated on two main factors at the same time whileconverting H&E stained image to EVG stained image: one is
the generation of noiseless image and another is the perfectimitation to the color of original EVG stained image. Throughthe experiment, we have found that for ensuring larger
number of matched features it is very important for theconverted EVG stained image to be noiseless and perfectly
adapted with target domain. As elastic and collagen fiber havevery similar features in the H&E stained image and very
different feature in the EVG stained image, the differentiationof the elastic and collagen fiber is not so perfect in theconverted EVG stained image. But this didn’t hamper the
extraction and matching features because all other tissue
components have been adapted properly to the color of
original EVG stained image and also the area where elastic
and collagen is not identified properly is less. Moreover, many
CycleGAN based studies haven’t considered identity loss
during training21),28), but in our case considering identity loss
has been very effective to imitate the exact color of EVG
stained image and to differentiate the elastic and collagen fiberin the converted EVG stained image. The generation of
perfectly adapted and noiseless EVG stained image has
enabled the matching of larger number of features with thesource image distributed all over the image and notconcentrated on a single area of image which has contributed
greatly to the better estimation of geometric transformation
information and finally the improved registration result. It also
takes very low processing time to convert an image from H&E
to EVG. Once the domain adaptation model is trained it takes
around 3s to convert the domain. The registration frameworktakes less than 1s (around 0.7 – 0.9s) for the registration of asingle pair of images. So, the overall processing time of our
proposed method is very low which is around 4s only.

(a)

SSIM: 0.6316

SSIM: 0.6206(b)

SSIM: 0.6863

SSIM: 0.6603
(c)

Fig.6

Visual evaluation of registration result: Overlapping view of
H&E and EVG image; (a) before registration, (b) after
registration without domain adaptation, and (c) after registration
with proposed method

Table 1

Registration quality determination using SSIM (upper andlower values represent averages and standard deviations
respectively)

MethodW/O DA

T.S.1

T.S.2

T.S.3

T.S.4

Avg.

0.70530.63220.65810.67850.66850.04460.04240.06060.04520.04820.73150.66550.69190.71780.70170.04150.0456

0.05780.0413 0.0465

Prop. Meth.

Table 2 Registration quality determination using MI (upper and lower
values represent averages and standard deviations respectively)
MethodW/O DA

Prop. Meth.

T.S.1

T.S.2

T.S.3

T.S.4

Avg.

0.86800.81010.80190.86210.83550.16460.14370.14260.23130.17060.92530.87140.86930.94480.90270.16370.15650.14810.21820.1716

5. Discussion
In this work, the domain adaptation network is designed
to change only the color information of the source domain(H&E) to the target domain (EVG) while remaining themorphological information of the source domain unchanged.

25

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

6. Conclusion

pp. 3368–3372 (2018).
6)

In this research, we have proposed a promising method
for the registration of histopathological images obtained fromheterogeneous staining techniques such as H&E and EVG.
For reducing the gap of color information between these two
types of stained images, we have proposed to use a
CycleGAN based domain adaptation network which has
adapted the source domain images of H&E stained to thetarget domain images of EVG stained so that the registration
framework is able to detect and match more features which
has contributed greatly to the significant improvement ofregistered EVG stained image. As our proposed method
leverages an unsupervised domain adaptation technique like
CycleGAN and also uses the handcrafted feature such as
SURF feature for obtaining the registered image, the majoradvantage is that without requiring prior aligned training data,
the method is able to provide promising registration result
while retaining briefness, simplicity and lower computational
complexity in its implementation.

M. Wodzinski, A. Skalski: "Multistep, Automatic and Nonrigid Image
Registration Method for Histology Samples Acquired Using Multiple
Stains", Physics in Medicine & Biology, Vol. 66, No. 2 (025006), (2021).

7)

J. Kybic, J. Borovec, "Automatic Simultaneous Segmentation and Fast
Registration of Histological Images", Proc. of IEEE 11th International
Symposium on Biomedical Imaging (ISBI), pp. 774–777 (2014).

8)

D. Mahapatra, Y. Sun: “Integrating Segmentation Information for
Improved MRF-Based Elastic Image Registration”, IEEE Trans. on
Image Processing, Vol.21, No.1, pp. 170–183 (2012).

9)

D. Mahapatra, Z. Ge: "Training Data Independent Image Registration
with Gans Using Transfer Learning and Segmentation Information", Proc.
of IEEE 16th International Symposium on Biomedical Imaging, pp. 709–
713 (2019).

10) H. Sokooti, B. de Vos, F. Berendsen, B. P. F. Lelieveldt, I. Isgum, M.
Staring: "Non-Rigid Image Registration Using Multi-scale 3D
Convolutional Neural Networks", Proc. of Medical Image Computing
and Computer Assisted Intervention (MICCAI), pp. 232–239 (2017).
11) S. Miao, Z.J. Wang, R. Liao: "A CNN Regression Approach for Realtime 2D/3D Registration", IEEE Trans. on Medical Imaging, Vol.35,

Acknowledgement

No.5, pp. 1352–1363 (2016).
12) A. Torralba, A. A. Efros: “Unbiased Look at Dataset Bias”, Proc. of IEEE

The authors thank to Dr. Takaya Ichimura from Saitama
Medical University for his valuable comment on this work.
The authors are also grateful to Otsuka Toshimi Foundation
for the financial support.

Conference on Computer Vision and Pattern Recognition (CVPR), pp.
1521–1528 (2011).
13) E. A. Albadawy, A. Saha, M. A. Mazurowski: “Deep Learning for
Segmentation of Brain Tumors: Impact of Cross-Institutional Training

1)

References

and Testing”, Medical Physics, Vol. 45, No. 3, pp. 1150–1158 (2018).

L. Cooper,O. Sertel, J. Kong, G. Lozanski, K. Huang, M. Gurcana:14) M. Long, H. Zhu, J. Wang, M. I. Jordan: “Deep Transfer Learning with

“Feature-based Registration of Histopathology Images with Different

Joint Adaptation Networks”, Proc. of the 34th International Conference
on Machine Learning (ICML 2017), pp. 2208–2217 (2017).

Stains: an Application for Computerized Follicular Lymphoma
15)

Prognosis”, Computer Methods and Programs in Biomedicine, Vol. 96,

2)

No. 3, pp. 182–92 (2009).

Faster R-CNN for Object Detection in the Wild”, Proc. of Conference on

L. Septiana, H. Suzuki, M. Ishikawa, T. Obi, N. Kobayashi, N. Ohyama,Computer Vision and Pattern Recognition (CVPR), pp. 3339–3348

T. Ichimura, A. Sasaki, E. Wihardjo, D. Andiani: “Elastic and Collagen(2018).
16) Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, M.

Fibers Discriminant Analysis Using H&E Stained Hyperspectral

3)

Images”, Optical Review, Vol.26, pp. 369–379 (2019).

Chandraker: “Learning to Adapt Structured Output Space for Semantic

I. Arganda-Carreras, C. Sorzano, R. Marabini, J. M. Carazo1, C. Ortiz-

Segmentation”, Proc. of Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 7472–7481 (2018).

de-Solorzano, J. Kybic: “Consistent and Elastic Registration of

4)

5)

Y. Chen, W. Li, C. Sakaridis, D. Dai, L. V. Gool: “Domain Adaptive

Histological Sections Using Vector-Spline Regularization”, Computer17) T. Wollmann, C. S. Eijkman, K. Rohr: “Adversarial Domain Adaptation

Vision Approaches to Medical Image Analysis, Vol. 4241, pp. 85–95

to Improve Automatic Breast Cancer Grading in Lymph Nodes”, Proc. of(2006).

IEEE 15th International Symposium on Biomedical Imaging (ISBI), pp.

J. Borovec, J. Kybic, M. Bušta, C. Ortiz-de-Solórzano, A. Muñoz-

582–585 (2018).

Barrutia: “Registration of Multiple Stained Histological Sections”, Proc.

18) A. Gholami, Subramanian, V. S., Shenoy, N. Himthani, X. Yue, S. Zhao,

of IEEE International Symposium on Biomedical Imaging, pp. 1034–

P. Jin, G. Biros, K. Keutzer: “A Novel Domain Adaptation Framework

1037 (2013).

for Medical Image Segmentation”, Proc. of International Medical Image

J. Borovec, A. Munoz-Barrutia, J. Kybic: "Benchmarking of Image

Computing and Computer Assisted Intervention (MICCAI) Brainlesion
Workshop, pp. 289–298 (2018).

Registration Methods for Differently Stained Histological Slides", Proc.

19) E. Ahn, A. Kumar, M. Fulham, D. Feng, J. Kim: “Unsupervised Domainof 25th IEEE International Conference on Image Processing (ICIP 2018),

26

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Adaptation to Classify Medical Images Using Zero-Bias Convolutional

Tanwi BISWAS

Auto-Encoders and Context-Based Feature Augmentation”, IEEE Trans.

She received B.Eng. degree on Electronics andCommunication Engineering from Khulna Universityof Engineering and Technology, Bangladesh in 2016.
She completed M.Eng. on Information andCommunications in 2018 and currently pursuing PhD
on Human Centered Science and Biomedical
Engineering major in Tokyo Institute of Technology,
Japan. She is working on medical image processing
using deep learning, hyperspectral imaging, domain
adaptation and generative adversarial network (GAN).

on Medical Imaging, Vol.39, No.7, pp. 2385–2394 (2020).
20) M. Javanmardi, T. Tasdizen: “Domain Adaptation for Biomedical Image
Segmentation using Adversarial Training”, Proc. of 15th IEEE
International Symposium on Biomedical Imaging (ISBI), pp. 554–558(2018).
21) D. Mahapatra, Z. Ge: “Training Data Independent Image Registration
using Generative Adversarial Networks and Domain Adaptation”,
Pattern Recognition, Vol.100 (2020).

Hiroyuki SUZUKI
He received B.E., M.E. and Ph.D. degrees from Tokyo
Institute of Technology in 1998, 2000 and 2006,
respectively. From 2003 to 2004, he was a Researcher
with Tokyo Institute of Technology. From 2004 to 2020,
he was an Assistant Professor with Tokyo Institute ofTechnology. Since 2020, he has been working as an
Associate Professor with Center for Mathematics andData Science, Gunma University. His research interests
include optical information security, holograms,
biometric authentication, medical and healthcare
information systems, and medical image processing. He
is a member of IEICE, JSAP, and OSJ.

22) H. Bay, A. Ess, T. Tuytelaars, L. Van Gool: “SURF: Speeded Up Robust
Features”, Computer Vision and Image Understanding (CVIU), Vol.110,
No.3, pp. 1–10 (2008).
23) J. Zhu, T. Park, P. Isola, A. A. Efros: “Unpaired Image-to-Image
Translation Using Cycle-Consistent Adversarial Networks”, Proc. ofIEEE International Conference on Computer Vision (ICCV), pp. 2242–
2251 (2017).
24) K. He, X. Zhang, S. Ren, J. Sun: “Deep Residual Learning for Image
Recognition”, In Proc. of IEEE conference on Computer Vision and
Pattern Recognition (CVPR), pp. 770–778 (2016).

Masahiro ISHIKAWA

25) P. Isola, J. Zhu, T. Zhou, A. A. Efros: “Image-to-Image Translation with

(Member)

He is an assistant professor at the Faculty of Healthand Medical Care, Saitama Medical University. He
received his MS and PhD degrees in engineering from
Niigata University, Niigata Japan, in 2004 and 2007,
respectively. His research interests are medical image
processing, pathology image recognition, andcomputer-aided diagnosis.

Conditional Adversarial Networks”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 15967–5976(2017).
26) P.H.S. Torr, A. Zisserman: “MLESAC: A New Robust Estimator with
Application to Estimating Image Geometry”, Computer Vision andImage Understanding, Vol.78, No.1, pp. 138–156 (2000).

Naoki KOBAYASHI27) N. P. Jacobson, M. R. Gupta: "Design Goals and Solutions for Display of(Fellow)

He received his B.Sc. and M.E. degree from Tokyo
Institute of Technology, Tokyo, Japan, in 1979 and
1981, respectively, and his Ph.D. from Niigata
University, Niigata, Japan, in 2000. He worked forLaboratories of Nippon Telegraph and Telephone Corp.
from 1981 and 2008. He has been a professor at the
School of Biomedical Engineering, Faculty of Healthand Medical Care of Saitama Medical University since2008. His research interests include medical image
proceeding, image compression and biosignal
processing. He is a member of IEICE, IIEEJ, JSMBE
and IEEE.

Hyperspectral Images", IEEE Trans. on Geosci. And Remote Sensing
Vol.43, No.11, pp. 2684–2692 (2005).
28) M. T. Shaban, C. Baur, N. Navab, S. Albarqouni: "Staingan: Stain Style
Transfer for Digital Histological Images", Proc. of IEEE 16th
International Symposium on Biomedical Imaging (ISBI), pp. 953–956(2019).

(Received May 28, 2021)(Revised December 12, 2021)
Takashi OBI
He earned B.S. degree in Physics, and M.S. and Ph.D
degree in Information Physics from Tokyo Inst. of Tech,Japan in 1990, 1992 and 1996, respectively. Currently,
he is an Associate Professor of Laboratory for FutureInterdisciplinary Research of Science and Technology
in Institute of Innovative Research, Tokyo Inst. of Tech.
His research focuses on medical image processing,
medical informatics, information system and security,
etc. He is a member of IEICE, JAMIT, JSAP, JSNM,
JSMP and IEEE.

27

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2021Part II --Spectral Super-Resolution Using CNN Decomposing a Color
Image into Luminance and Chrominance Components
Masahiro SAKAMOTO† , Kazufumi KANEDA† (Member) , Bisser RAYTCHEV†† Hiroshima University, Graduate School of Advanced Science and Engineering<Summary> Hyper-spectral images are used in a wide range of ﬁelds such as industry, medicine, remotesensing, and so on. They are also used in computer graphics as light probe images and textures in spectral
rendering. The acquisition of spectral images is, however, costly in terms of equipment and time, which hinders its acquisition and use. Conventional spectral super-resolutions using deep learning have been adopting
a direct end-to-end learning method to RGB and hyper-spectral images. In contrast, we focus on the factthat hyper-spectral images are decomposed into luminance and chrominance components, and we propose
a novel spectral super-resolution using a deep learning to estimate each component separately. Finally, inthe proposed method, a hyper-spectral image is reconstructed by combining the estimated luminance andchrominance components.
Keywords: spectral super-resolution, hyper-spectral image, convolutional neural network, deep learning,
luminance and chrominance

RGB images use red, green, and blue to represent colors
based on human visual perception, while hyper-spectral

As a result, it becomes diﬃcult to maintain the signalto-noise ratio without reducing the resolution of the image3). Therefore, the trade-oﬀ between spectral and spatial resolutions is an important issue for the acquisition

images are composed of many wavelength bands withdozens or more bands and represent physical quantities.

of hyper-spectral images. In addition, the high cost of
hyper-spectral cameras is also a problem because of theSince materials in nature have unique spectral reﬂectance,
hyper-spectral images show the characteristics of materials in detail. Hyper-spectral images are used in a wide
range of ﬁelds such as industry, medicine, agriculture, andremote sensing1)2)because their abundant spectral information enables sophisticated discrimination that is diﬃcult to achieve with RGB images. Hyper-spectral images
are also used in computer graphics as light probe images

special sensors and complex processing.
For this reason, research on super-resolution for hyperspectral images has been conducted. Super-resolution for
hyper-spectral images can be classiﬁed into two types:
spatial super-resolution and spectral super-resolution.

1.

Introduction

We focus on the spectral super-resolution from RGB
images, which deals with the reconstruction of hyperspectral images with many bands and with the same
spatial resolution as the input RGB image. Since thissuper-resolution can use RGB images captured by general cameras, the acquisition of spectral images is a lower

and textures in spectral rendering. Physically-based rendering requires the use of hyper-spectral images to simulate physically accurate light transport for generating

cost than using expensive hyper-spectral cameras.
Conventional spectral super-resolution using deeplearning generates a hyper-spectral image by directly es-

images.
Commonly used cameras capture RGB images where
each light receiving element of the cameras is aﬀected bya wide range of wavelength, while hyper-spectral cameras can identify wavelengths in a narrow range and acquire wavelength information from dozens or hundreds

timating the spectral intensities. Taking advantage ofthe fact that hyper-spectral images are decomposed into
luminance and chrominance components, we propose anovel spectral super-resolution using a deep learning toestimate each component separately. Rather than learning spectral intensities directly end-to-end, a convolu-of bands. Capturing images in narrower bandwidth provide more spectral information, but it proportionally reduces the number of photons reaching a sensor element.

28

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

tional neural network (CNN) can estimate the nonlin-

between the images, but it is not easy to capture two

ear relationship between RGB and hyper-spectral images

diﬀerent images in the same position. To address the
problem, some methods proposed the correction for spa-

with better accuracy when learning each component sepa-

tial mis-alignment by using the spectral response of RGB
camera6).

rately. Finally, in the proposed method, a hyper-spectral
image is reconstructed by combining the estimated luminance and chrominance components.
The hyper-spectral images reconstructed by the proposed method can be used for spectral rendering in computer graphics. Although the proposed method cannot
solve the metamerism problem, which is an important issue of estimating spectrum, it may provide some solution
to the problem, because the spectral super-resolution using CNN estimates the spectral intensity not only fromSpectral super-resolution, which creates a hyperspectral image from an RGB image without using
any auxiliary low-resolution hyper-spectral image, ismost related to our research. In contrast to spatial
super-resolution, spectral super-resolution recovers highfrequency components in spectral domain without reducing spatial resolution. The automatic colorization of agrayscale image to an RGB color image also belongs to akind of spectral super-resolution. Recently a colorization
method7)has been improved by separating the image into

the tristimulus values at the pixel, but also from those of
the surrounding pixels. This may lead to some avoidanceof metamerism, and a spectrum close to the real could be
estimated.

2.
2.1

luminance and chrominance components. The method
estimates only the chrominance component using CNN.
In reconstructing a hyper-spectral image from an RGB
image, several methods have been proposed, such as amethod using K-SVD to learn a sparse dictionary of
hyper-spectral images8)and a method using an RBF (Radial Basis Function) network to learn a nonlinear mapping function between images9). Recently, deep learning

Related Work

Upsampling methods for rendering

Although three stimulus value representation of an
RGB image may be suﬃcient for a color display, it is notexactly enough for an accurate simulation of light transport. Physically-based rendering that allows for realistic representations requires densely sampled wavelengths
over the visible light region. However, we don’t have
many hyper-spectral images that can be used as textures
or light probes. To solve the problem, research on obtaining spectral distributions from RGB tristimulus values
has been developed.
Spectral rendering can generate physically accurate images, but it has the problem that the processing timeand memory usage becomes large because the number ofmethods based on CNN have been used for spectral superresolution. Inspired by a network model used for semantic segmentation10), Galliani et al.11)applied the model tospectral super-resolution. He et al.12)proposed a method
optimizing the network model by utilizing an auxiliary
spectral response function. They employed the loss function that combines L1 loss and a spectral angle mapper toachieve quick convergence. In the same way, Li et al.13)
proposed a novel adaptive weighted attention network using a spectral response function. They incorporated thediscrepancies between the original RGB image and therecovered RGB image from the hyper-spectral image asa ﬁner constraint for accurate reconstruction.

sampled wavelengths increases. It is desirable to express
the spectral distribution in a compact manner. In addition, spectral distribution normally tends to be smooth.
Considering these things, methods for reconstructing thespectral distribution using basis functions and few coeﬃcients have been proposed4),5).
2.2

3.

Proposed Method

The overview of the proposed method is shown in
Fig. 1. The proposed method learns luminance andchrominance components of hyper-spectral images separately, instead of directly learning spectral intensities.
Luminance and chrominance components are estimatedby the respective CNNs. The idea of the proposed
method is based on the fact that the shape of the spectral distributions is not aﬀected by the intensity of theincident light. Since the proposed method does not require one network to learn extra information, it can be

Spectral super-resolution

Most of the existing spectral super-resolution methods
recover spatial resolution, while few methods attempt torecover spectral information.
To improve the spatial resolution of hyper-spectral image, methods for combining a high resolution RGB image and a low resolution hyper-spectral image have beendeveloped. These methods require accurate alignment

29

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
Luminance network（Loss: MSE）

  Luminance
conversion

RGB

Chrominance network（Loss: Cos, SAM）

Hyper-Spectral






: Fully Convolutional DenseNets [10]: linear transformation

Fig. 1 Overview of the proposed method

ch:3






ch:31

















Convolution
Dense Block
Transition Down













Transition Up
Concatenation
Skip Connection

Fig. 3 The CIE XYZ color matching functions



the incident light. Section 3.3 describes the method for
calculating the spectral magnitude in detail.
The conversion from a hyper-spectral image S(λ) to(X, Y, Z) values is expressed by the following equations:1X=
S(λ)x̄(λ)dλ,
(1)
k Λ

1
Y=
S(λ)ȳ(λ)dλ,
(2)
k Λ

1
Z=
S(λ)z̄(λ)dλ,
(3)
k
 Λ
k= ȳ(λ)dλ,(4)Fig. 2 Diagram of Fully Convolutional DenseNets

expected to reduce the error of each component. Luminance has intensity information, and chrominance is represented by a nomalized spectral distribution. Using theluminance and chrominance output from the respective
CNNs, a hyper-spectral image is reconstructed by the luminance conversion in which the normalized chrominancecomponents are scaled to match the estimated luminancecomponents.
In both networks for luminance and chrominance components, we employed Fully Convolutional DenseNets10)(FC-DenseNets) shown in Fig. 2. We train the networks
for luminance and chrominance components separately.
Galliani et al.11)applied the FC-DenseNets to spectralΛwhere λ is the wavelength, Λ is wavelength range, x̄(λ),ȳ(λ), and z̄(λ) are the color matching functions shownin Fig. 3. We use MSE Loss as a loss function for the
network to estimate the spectral magnitude, and the network learns the relationship between (R, G, B) values ofan RGB image and (X, Y, Z) values of a hyper-spectral
image.
This allows the luminance network to learn the non-super-resolution, since the network for semantic segmentation outputs an image of the same size as the input
image. The FC-DenseNets has been frequently followed
in related works, and it is the standard method for spectral super-resolution using deep learning. Thus, we use
the network in our experiments as well.

linear relationship of tristimulus values between RGB andhyper-spectral images. Because of the nonlinear characteristics of the camera response function, RGB values andthe spectral intensities normally have a nonlinear relationship. Although the camera response function is speciﬁc to each camera, it can be approximated with gamma
correction for γ = 2.2 in most cases. Therefore, in our
experiments, we use gamma correction γ = 2.2 to re-

3.1 Luminance components
In the proposed method, the spectral magnitude of a
hyper-spectral image is determined based on X, Y , andZ values in CIE XYZ color space. Here, we consider“spectral magnitude” to correspond to the intensity of30

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

generate RGB images from hyper-spectral images as a

calculated from the (Xc , Yc , Zc ) values of the estimatedrepresentation of a standard camera response function.

chrominance components, and δ is a small value to avoid
the singularity where Lc is zero.

3.2 Chrominance componets
Suppose an N -bands hyper-spectral image, spectral
values for each pixel of the image can be considered
as an N -dimensional vector. For examples, just as(X, Y, Z) values in the color space can be considered asa three-dimensional vector. The direction of the vector corresponds to normalized chrominance components,
and the magnitude of the vector corresponds to luminance. In other words, chrominance components of N -Generally, Y in the CIE XYZ color space is assumed
to be the luminance. However, the magnitude of spectral
intensities is not only measured in Y . In our experiment,
we also use X and Z in addition to Y . This allows S toaccurately represent the features of ground truth, even if
the spectral distribution of Sc deviates signiﬁcantly fromthe wavelength range covered in Y . Ll and Lc denote thefollowing equations.
X Sum of tristimulus values:

spectral bands can be expressed as the direction of an
N -dimensional vector.
Cosine similarity is used to measure how close the directions of two vectors are. We apply a cosine similarity
shown in the following equation to the loss function of
the CNN to estimate the chrominance. The chrominancenetwork learns the similarity of spectral distributions to
the ground truth and estimates chrominance componentsof hyper-spectral images.
1
n i=1
n

Cos = 1 −Sc (i) · S(i)Sc (i)2 S(i)2

Ll = Xl + Yl + Zl ,X Maximum of tristimulus values:
Ll = max(Xl , Yl , Zl ),

(5)

1
cos−1
n i=1
n



Sc (i) · S(i)Sc (i)2 S(i)2


.

4.

Experimental Results

4.1 Datasets
We used two datasets, the CAVE dataset14)and the
ICVL dataset8). Both datasets provide hyper-spectral

The input to the chrominance network is an RGB image, and the output is an N -bands hyper-spectral image
without luminance information.

images as well as corresponding RGB images. To ensure accurate quantitative evaluation, input RGB images
were synthetically re-generated in our experiment. We
used the sRGB color space and approximated the nonlinear characteristics of the camera with gamma correction(γ = 2.2).
4.1.1 CAVE dataset
CAVE dataset is one of the most widely used hyper-

3.3 Luminance conversion
The ﬁnal hyper-spectral image S is reconstructed using
the estimated (Xl , Yl , Zl ) values. Pixel values of the normalized hyper-spectral image Sc is scaled by the following
linear transform:
Ll
Sc ,Lc + δ

(9)the maximum value of (Xc , Yc , Zc ). The results may
be diﬀerent depending on which maximum value ischosen as the criterion. We examine the both criteria
in our experiments.

(6)

SAM represents the angle between two vectors; thesmaller the spectral angle, the higher the similarity. In
our experiments, we examine Cos and SAM as a loss function to learn chrominance components.

S =Lc = max(Xc , Yc , Zc ).

The tristimulus values are compared for each pixel,
and the maximum value is used as the spectral magnitude L. The maximum value is selected, since itis representative of the most inﬂuential wavelength
region. We select the same value for Ll and Lc
from (X, Y, Z) values. For example, when Ll =max(Xl , Yl , Zl ) = Xl , then Lc = Xc regardless of

where n is the total number of pixels in a image, Sc (i)is reconstructed spectral values in i-th pixel, S(i) is theground truth, · is the inner product of vectors, and || ||2
is the Euclidean norm.
Spectral angle mapper (SAM) is also used to measure
the similarity of spectral values, which is shown as follows:
SAM =

(8)

The spectral magnitude of a hyper-spectral image isconverted considering the entire wavelength range,
by adopting the sum of the tristimulus values for the
spectral magnitude L.

,Lc = Xc + Yc + Zc .

(7)

where the spectral magnitude Ll is calculated from theestimated (Xl , Yl , Zl ) values, spectral magnitude Lc isspectral datasets. The dataset consists of 32 images witha size of 512 × 512. Each hyper-spectral image is sampled
31

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

in the wavelength range from 400 nm to 700 nm with a
RM SE =

bandwidth of 10 nm and contains 31 bands. The images
are divided into ﬁve sections, containing various objects

4.3

4.3.1

SAM ↓
16.089 ± 5.736
9.065 ± 2.915
9.351 ± 2.880

Proposed

Loss
MSE
Cos
SAM

Similarity ↑0.999 ± 0.001
0.999 ± 0.001
0.999 ± 0.001(10) · S(i)S(i)

S(i)2 S(i)2

.

(11)

Quantitative evaluation

Each component

we can improve the accuracy of chrominance components
on ICVL dataset compared to the end-to-end method.

Table 2 Quantitative evaluation of chrominance components on ICVL dataset
MethodEnd-to-End.

Quantitative evaluations for each dataset is shown in
Tables 1, 2, 3, and 4 for luminance and chrominancecomponents. The tables show the mean and standard
deviation (mean ± std).
Tables 3 and 4 show that the proposed method, which
focuses on (X, Y, Z) values of hyper-spectral images, improves the accuracy of the spectral magnitude comparedto the end-to-end method, which learns whole hyperspectral images with MSE Loss for each dataset.
In Table 1, the accuracy of the chrominance components of our method is improved by using Cos or SAM as
the loss function for each dataset. On the other hand, inthe results of chrominance components on ICVL dataset
shown in Table 2, the proposed method using Cos is worse
than the end-to-end method in terms of SAM . This may
be because the accuracy of the chrominance components
is suﬃciently high even using MSE Loss. The spectral distributions on ICVL dataset contain many similar distributions, and it is diﬃcult to improve Similarity further
by learning with Cos as the loss function. By learning
chrominance components with SAM as a loss function,

Table 1 Quantitative evaluation of chrominance components on CAVE dataset

Propoesd2

We also compare a conventional end-to-end method using MSE Loss.

4.2 Evaluation metrics
We use root mean square error (RM SE) to evaluate theerror in spectral intensity values, and also use Similarity
and SAM to evaluate the similarity between the spectral
distributions.
RM SE is shown as follows:

Similarity ↑0.942 ± 0.043
0.981 ± 0.011
0.980 ± 0.011

1
n i=1
n

Similarity =hyper-spectral images downsampled to 31 bands. Notethat the re-generated RGB images are diﬀerent in color
from the RGB images in the dataset, where many of the
RGB images included in the dataset are underexposed oroverexposed. Such RGB images are not suitable for themain purpose of this research. Therefore, we re-generated
the RGB images according to the method described in the
instruction attached to the dataset.

Loss
MSE
Cos
SAM

n i=1

 − S(i)S(i)

Similarity is shown as follows:

such as faces, food, and paintings. Since the total numberof images is small at 32, we use 2-fold cross-validation toevaluate the results.
4.1.2 ICVL dataset
The ICVL dataset consists of 201 images taken outdoors, with a size of 1300 × 1392. The dataset provides hyper-spectral images of 519 bands sampled from
400 nm to 1000 nm with a bandwidth of 1.25 nm, and31 bands downsampled from 400 nm to 700 nm with abandwidth of 10 nm. In our experiment, we employed

MethodEnd-to-End



n
1 

4.3.2

SAM ↓
2.391 ± 0.900
2.400 ± 0.779
2.216 ± 0.835

Reconstructed hyper-spectral images

Quantitative evaluation of reconstructed hyperspectral images of each dataset is shown in Tables 5
and 6.

Table 3 Quantitative evaluation of luminance components on CAVE dataset
MethodLoss
RM SEXY Z ↓RM SEX ↓RM SEY ↓RM SEZ ↓End-to-End MSE(Spectral) 5.446 ± 3.375 4.980 ± 3.638 5.090 ± 3.899 5.901 ± 3.885
Proposed
MSE(XYZ)
5.180 ± 3.038 4.812 ± 3.247 4.894 ± 3.333 5.454 ± 2.903

Table 4 Quantitative evaluation of luminance components on ICVL dataset
MethodLoss
RM SEXY Z ↓RM SEX ↓RM SEY ↓RM SEZ ↓End-to-End MSE(Spectral) 0.399 ± 0.067 0.337 ± 0.052 0.399 ± 0.058 0.410 ± 0.119
Proposed
MSE(XYZ)
0.278 ± 0.041 0.230 ± 0.049 0.275 ± 0.029 0.320 ± 0.057

32

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 5 Quantitative evaluation of reconstructed hyper-spectral images on CAVE dataset
MethodL. Loss C. Loss
L. conversion
RM SESpectral ↓End-to-End MSE(Spectral)
8.168 ± 4.047
Y
8.679 ± 3.756
MSE
X +Y +Z
8.085 ± 3.566
Cos
(XYZ)
max(Xl , Yl , Zl )8.524 ± 4.083
max(Xc , Yc , Zc )8.288 ± 3.789
ProposedY
8.784 ± 3.667
MSE
X +Y +Z
8.125 ± 3.441
SAM
(XYZ)
max(Xl , Yl , Zl )8.563 ± 3.924
max(Xc , Yc , Zc )8.320 ± 3.671

Table 6 Quantitative evaluation of reconstructed hyper-spectral images on ICVL dataset
MethodL. Loss C. Loss
L. conversion
RM SESpectral ↓End-to-End MSE(Spectral)
3.631 ± 1.629
Y
3.868 ± 1.542
MSE
X +Y +Z3.856 ± 1.572
Cos
(XYZ)
max(Xl , Yl , Zl )3.860 ± 1.542
max(Xc , Yc , Zc )3.860 ± 1.541
ProposedY
3.546 ± 1.570
MSE
X +Y +Z
3.527 ± 1.582
SAM
(XYZ)
max(Xl , Yl , Zl )3.535 ± 1.578
max(Xc , Yc , Zc )3.534 ± 1.578

hyper-spectral image because the image contains brightline spectra with high peak values. The upper and lower
ﬁgures show that the proposed method properly recognizes the peak wavelength, although the spectral intensities are lower than the ground truth.
The reconstructed spectral intensities of each method
on ICVL dataset are shown in Fig. 6. Because ICVL
dataset consists of images taken outdoors, many images
contain the sky. The error in the reconstructed spectral intensities of the sky is small as shown the lower
ﬁgure. On the contrary, the reconstructed spectral intensities have some diﬀerence from the ground truth in thearea containing ﬁne patterns as shown in the upper ﬁgure.
In both ﬁgures, there is no signiﬁcant diﬀerence between
the proposed method and the end-to-end method.
The common feature in Fig. 4, 5 and 6 is that the reconstructed spectral intensities have larger errors on thelong wavelength side. One of the reasons is that the pro-In the evaluation on CAVE dataset in Table 5, theproposed conversion method adopting (X + Y + Z) as
spectral magnitude achieved better results than the otherconversion methods. This conversion method using allof the tristimulus values outperformed the end-to-end
method (MSE(Spectral)). In addition, the conversion
method adopting max(Xc , Yc , Zc ) outperformed the conversion method adopting max(Xl , Yl , Zl ). Therefore, it
seems that the former is more accurate for calculating
the maximum value of (X, Y, Z) values.
The evaluation on ICVL dataset in Table 6 shows similar results as the evaluation on CAVE dataset, although
the best loss function is diﬀerent. The proposed method
learning chrominance components with SAM as a loss
function achieved better results for reconstructed hyperspectral images on ICVL dataset.
4.4

Qualitative evaluation

posed method does not consider the response characteristics of the color matching functions. The loss functionused in our experiments uniformly deals with the wavelength range of the spectral intensities, and the network
is optimized by averaging the error at each wavelength.
It is diﬃcult to estimate the spectral intensities on thelong wavelength side accurately because the color matching functions has low response characteristics on the longwavelength side. This is a common problem with thespectral super-resolution and is one of the future work.

The reconstructed spectral intensities of each method
on CAVE dataset are shown in Fig. 4. In the upper
ﬁgure, spectral intensity of the proposed method is successfully reconstructed to the same spectral magnitude as
the ground truth (GT). In the lower ﬁgure, the proposed
method achieved better results compared to the end-toend method (MSE(Spectral)) in the wavelength range of
400 to 600 [nm].
The reconstructed spectral intensities of “CD” images
are shown in Fig. 5．It is diﬃcult to reconstruct the33intensity

intensity

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

intensity

intensity

Fig. 4 Qualitative comparison of each method on “Balloon” images from CAVE dataset where “Our” is the
proposed method that performed luminance conversion using Cos for chrominance components

intensity

intensity

Fig. 5 Qualitative comparison of each method on “CD” images from CAVE dataset where “Our” is the proposed method that performed luminance conversion using Cos for chrominance components

Fig. 6 Qualitative comparison of each method on ICVL dataset where “Our” is the proposed method that
performed luminance conversion using SAM for chrominance components

5.

components separately. In the proposed method, thehyper-spectral image is ﬁnally reconstructed from the estimated chrominance component through a linear transformation using the estimated luminance component. Us-

Conclusion

We proposed a novel spectral super-resolution using a
deep learning to estimate luminance and chrominance

34

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

ing all of three tristimulus values, the proposed method

12) J. He, J. Li, Q. Yuan, H. Shen, L. Zhang: “Spectral Response
Function Guided Deep Optimization-Driven Network for Spectral Super-Resolution”, IEEE Trans. on Neural Networks andLearning Systems, pp. 1-15 (2021).

converts the spectral magnitude considering the entire
wavelength range. Compared with the conventional endto-end method that learns spectral intensities directly,
the proposed method reduces the error of each component
and achieves superior results for reconstructing hyperspectral images.
Future work includes the selection of appropriate networks for learning each component and the extension of
our method to HDR images to reproduce a spectral image with higher dynamic range than LDR. Another future
work includes experiment with a larger number of images
to verify the further usefulness of the proposed method.

13) J. Li, C. Wu, R. Song, Y. Li, F. Liu: “Adaptive Weighted
Attention Network with Camera Spectral Sensitivity Prior
for Spectral Reconstruction from RGB Images”, Proc. of the2020 IEEE/CVF Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), pp. 1894-1903 (2020).

The challenge of solving the metamerism problem is also
one of the important issues of spectral super-resulution.

Masahiro SAKAMOTO

14) F. Yasuma, T. Mitsunaga, D. Iso and S. K. Nayar: “Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum”, IEEE Trans. on Image
Processing, Vol. 19, No. 9, pp. 2241–2253 (2010).

(Received June 4, 2021)(Revised January 17, 2022)

He received his B.E. and M.E in information engineering from Hiroshima University,
Japan, in 2020 and 2022. His research interests include spectral super-resolution.

References
1) M. Moroni, A. Mei, A. Leonardi, E. Lupo, F. L. Marca: “PETand PVC Separation with Hyperspectral Imagery”, Sensors,
Vol. 15, No. 1, pp. 2205–2227 (2015).
2) E. Belluco, M. Camuﬀo, S. Ferrani, L. Modenese, S. Silvestri,
A. Marani: “Mapping Salt-Marsh Vegetation by Multispectraland Hyperspectral Remote Sensing”, Remote Sensing of Environment, Vol. 105, No. 1, pp. 54–67 (2006).

Kazufumi KANEDA (Member)He received his D.E. in system engineeringfrom Hiroshima University, Japan, in 1991.
Currently, he is a professor at Graduate
School of Advanced Science and Engineering,
Hiroshima University, Japan. His research interests include computer graphics, data visualization and medical graphics.

3) O. Yilmaz, O. Selimoglu, F. Turk, M. S. Kirik: “SNR Analysis
of a Spaceborne Hyperspectral Imager”, Proc. of the 2013 6thInternational Conference on Recent Advances in Space Technologies (RAST), pp. 601-606 (2013).
4) H. Otsu, M. Yamamoto, T. Hachisuka: “Reproducing Spectral
Reﬂectances from Tristimulus Colours”, Computer Graphics
Forum, Vol. 37, No. 6, pp. 370–381 (2018).

Bisser RAYTCHEV

5) W. Jakob, J. Hanika: “A Low Dimensional Function Space forEﬃcient Spectral Upsampling”, Computer Graphics Forum,Vol. 38, No. 2, pp. 147–155 (2019).

He received his Ph.D. in Informatics from
Tsukuba University, Japan, in 2000. Afterbeing a research associate at NTT Communication Science Labs and AIST, he is presently
an associate professor in the Department ofInformation Engineering, Hiroshima University, Japan. His current research interests include machine learning, computer vision, natural language processing and brain-inspired
computing.

6) R. Kawakami, J. Wright, Y. -W. Tai, M. B. Ezra, K. Ikeuchi,
Y. Matsushita: “High Resolution Hyperspectral Imaging via
Matrix Factorization”, Proc. of the 2011 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), pp.
2329–2336 (2011).
7) S. Guadarrama, R. Dahl, D. Bieber, M. Norouzi, J. Shlens,K. Murphy: “PixColor: Pixel Recursive Colorization”, Proc.
of the British Machine Vision Conference (BMVC), pp. 112.1112.13 (2017).
8) B. Arad, O. B. Shahar: “Sparse Recovery of Hyperspectral Signal from Natural RGB Images”, Proc. of the European Conference on Computer Vision (ECCV), pp. 19–34 (2016).
9) M. H. Nguyen, D. K. Prasad, M. S. Brown: “Training-Based
Spectral Reconstruction from a Single RGB Image”, Proc. of
the European Conference on Computer Vision (ECCV), pp.
186–201 (2014).
10) S. Jégou, M. Drozdzal, D. Vázquez, A. Romero, Y. Bengio: “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation”, Proc. of the2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1175-1183 (2017).
11) S. Galliani, C. Lanaras, D. Marmanis, E. Baltsavias,
K. Schindler: “Learned Spectral Super-Resolution”, arXiv
preprint arXiv:1703.09470, (2017).

35

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）-- Special Issue on Journal Track Papers in IEVC2021Part II --Evaluating YOLOv3 for Identification and Classification of
Functional and Sclerosed Glomeruli
Thalita Munique COSTA† (Student Member), Lourenço BARBOSA ††, Yoko USAMI⁑,
Mai IWAYA⁑, Kiyoshi TANAKA⁑⁑ (Fellow), Fabio SCHNEIDER††,⁑†
Shinshu University Graduate School of Medicine, Science and Technology, ††Federal University of Technology-Parana/CPGEI,
Department of Laboratory Medicine Shinshu University Hospital, ‡‡Academic Assembly (Institute of Engineering), Shinshu University<Summary> In Digital Pathology, histological slides digitized allow the use of techniques for automatic proceduresin histopathology, permitting the automatic quantification of the rate of sclerosed glomeruli to order the biopsy slides sothat the most serious cases can be identified more quickly. In this work, we evaluate the YOLOv3 as deep neural network
to identify glomeruli in WSI and classify in functional and sclerosed glomeruli. This work used the framework YOLOv3,
with 53-layers convolutional neural network, and 30 complete slides from the Bio-Atlas repository (Pennsylvania State
University), which resulted in 2448 images of 1024x1024 pixels with one or more glomeruli, used for training andperformance evaluation. A total of 585 sclerosed glomeruli and 3383 functional glomeruli were labeled. Through theexperiments, we achieve high performance in identification and classification of glomeruli (e.g., recall of 96.8%,
precision of 95.9%, accuracy of 98.1%, and an F1 score of 96.3%). The method is capable to identify and report thelocation of the glomeruli on the slide, classify the glomeruli in functional and sclerosed, and precisely provide thepercentage of sclerosed glomeruli, allowing support for a histopathological study of kidney diseases in the medicalroutine.
Keywords: Digital Pathology, Glomerular Sclerosis, Glomerular Hyalinization, YOLOv3, Deep Learning, Whole
Slide Image (WSI).

glomeruli and the accumulation of material hyaline
occurs, causing permanent loss of its function6). These
alterations cause a chain reaction. Healthy glomeruli
suffer hypertrophy to compensate for the loss of othernephrons seeking to maintain renal function throughchanges in structure, resulting in its sclerosis2). Renal
disease is in progress to its final stage when thedestruction of nephrons reduces the glomerular
filtration rate to less than 50% of normal2).
For the analysis of renal tissue, a renal biopsy is
performed, an exam indicated when there is a renal
failure of rapid progression, nephrotic syndrome,systemic
diseases,or
unexplained
renaltransplantation dysfunction6). Stained histologicalsections are evaluated microscopically to identify thenature of the disease6). One of the important items isthe evaluation of the degree of glomerular healing (i.e.,sclerosis). In these situations, it is sought to evaluate
the percentage of affected glomeruli, whether the
disease is focal (affects less than half of the glomeruli)
or diffuse (affects more than half of the glomeruli)6).
Typical images of functional and sclerosed glomeruli
on a renal biopsy slide are shown in Fig. 1.
In the medical routine, there may be many slides
to be analyzed at the same time. It becomes interesting
to use a tool that identifies slides that, for not having aminimum number of glomeruli, should be discarded,
while orders the slides by those that appear to have amore serious diagnosis, which can also be

1. Introduction
The functional unit of the kidneys responsible forthe formation of urine is called Nephron. Initially, ahealthy kidney has between 800,000 and 1,000,000
nephrons and has no regenerative capacity1). This
number gradually reduces with aging and the
reduction may be associated with several diseases suchas diabetes, systemic lupus erythematosus, ingestionof drugs, and infections2). Studies have scored andanalyzed recurrent cases of acute kidney injury caused
by Coronavirus infection 3) - 5). It is estimated that 6.7%of patients with Coronavirus acquire acute kidney
injury and among these 91.7% cannot survive5). Each
Nephron contains a glomerulus, which is formed by atangle of capillaries, called glomerular capillary tuft,
wrapped by Bowman's capsule1). They filter a large
amount of liquid and other elements from the blood.
The filtered liquid passes through a sequence of
tubular structures, where selective resorption of waterand ions occurs, and the residual elements, includingexcess water, are eliminated in the form of urine1).
Each component of renal tissue (e.g., glomeruli,
tubules, interstices and blood vessels) is usuallyaffected differently for each disease2). The glomeruli
may be affected resulting in a terminal lesion called
glomerular sclerosis or hyalinization6),2), when plasma
proteins are deposited in large quantities inside the36

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

2. Proposed Method
2.1 Entire flow
In this work we proposed to use a neural networkarchitecture capable of identifying and classifying the items
at the same time, using a prepared train set (i.e., teacher
database) for supervised training of the neural network withimages of 1024×1024 pixels of 26 varied WSIs. Also, we
use early stopping to avoid overfitting, set a threshold toincrease the accuracy of the system, reconstruct the WSI
with the labels, treat the data to eliminate the counting ofduplicate labels and bring the total number of glomeruli andthe percentage of sclerosed glomeruli. We show the entire
workflow of the proposed method in Fig.2

Fig. 1 Glomeruli found in renal biopsies. A: Functional
glomerulus; B: Sclerosed glomerulus1)

applied to identify kidneys that are eligible fortransplantation7). The glomerular sclerosis rate can bea good parameter to make this classification. Fully
digitized slide images (i.e., WSIs) are being widelyused in digital pathology. WSIs make thehistopathological evaluation possible in a virtualenvironment and allows the biopsy result to be sharedwith other specialists. Several laboratories use themfor diagnostic routine. With the use of stains (e.g.,
hematoxylin-eosin, Schiff's periodic acid), performed
to highlight and identify specific histological elements,
in different colors and shades, there is the possibilityof using the WSI for the automation of the analysis ofhistopathological alterations8).
The human vision and processing capacity in a
trained and rested subject are accurate and capable ofidentifying an object instantly9). On the other hand,automation allows benefits such as the reduction of theeffects of human subjectivity in diagnosis and theability to identify minimal variations in histologicalfindings that would not be perceived by humans8). In
this context, neural networks emerged to try to model
the human thought process using logical rules10).
Convolutional Neural Networks (i.e. CNN), using, forexample, Deep Learning algorithms can help inclassifications that are traditionally done manually11).
Deep learning algorithms use a large amount oftraining data in conjunction with multi-layered neuralnetwork architectures12). In many histopathologicalanalyses, one of the important threads is theclassification of the evaluated components. In thesesituations, deep learning can present interesting resultsin the sense of adequate classification13). In this work,
it is proposed to evaluate the performance of the deeplearning architecture YOLOv3 in the identificationand classification of glomeruli into functional andsclerosed, to obtain the percentage of sclerosed
glomeruli. YOLOv3 architecture was selected for thiswork because it presents many of the desired steps(training of convolutional neural network layers,
prediction of bounding boxes, calculation of box
probability and class probability) in a single
architecture while presenting a superior performancefor several applications when compared to some otherarchitectures9).

The first step consists of preparing the train set,
selecting images, from the Bio-Atlas repository14),which have a good number of glomeruli, different
shades and stains, and various diagnoses. Then thesmaller images (i.e., 256×256 pixels) areconcatenated creating bigger images with the size
chosen (i.e., 1024×1024 pixels). Some of theseimages are separated in another file (i.e., test set) for
performance evaluation. After that, text files werecreated with the coordinates and classes of theglomeruli using the YBAT software15). The files werecreated with a supervision of a pathologist.
The next step is configuring the neural network by
setting the hyperparameters of the neural network usedaccording to the specifications of the computer used,the data augmentation characteristics that you want toapply and resizing done in the first layer of the neuralnetwork (i.e., 512×512 pixels, a resize that maintains
the shape of the image with a simple scaling factor).
Next, the training can start and should be stopped assoon as the accuracy value is stabilized to avoid
overfitting.
After training, the threshold value for classconfidence must be configured. Only items found witha confidence value greater than the threshold will beconsidered. With the trained neural network, it ispossible to deliver slide images cut into pieces of 1024
×1024 pixels, never seen before by the neuralnetwork, to be labeled (i.e., images separated for
performance evaluation in the test set). The labeledimages of 1024×1024 pixels are reconstituted in the
WSI while the coordinates of each glomerulus foundare calculated, duplicate tags are eliminated from thecount, and a text file is generated with the informationof the number of glomeruli found, number of sclerosed
glomeruli, location of each glomerulus and percentage
of sclerosed glomeruli in the slide.

37

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 2 Entire workflow of the proposed method

38

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

2.2 Deep neural network
The deep network architecture used in this work was
YOLOv316), which is an evolution of YOLO (i.e. You
Only Look Once). YOLO is a framework that brings afast solution for the identification of objects in images,
defining bounding boxes and class probabilities.
YOLOv3 architecture has 53 layers (i.e., Darknet-53) asshown in Fig. 3 and Table 1. The network has
convolutional layers with varied numbers of 3×3 filters,pooling for image resizing, residual layers, a layer
entirely connected at the output and a SoftMax layer,
and an avgpool layer that despite being part of the
network are not used in this version of YOLO16). The
open-source YOLOv3 architecture can detect andclassify the object found among more than one class andconsiders in the decisions the context in which the object
is involved16).
2.3

Fig. 3 Representation of YOLOv3 network architecture.

Table 1 Layers of neural network Darknet-53

Database, training, and validation
14)

The repository used (i.e., Bio-Atlas ) in the process
to build the database (i.e., training set and test set) is
composed of pieces of 256×256 pixels of WSI. This
image size is not appropriate to the training once that
does not show an entire glomerulus and its context in theimage. To attend to these requirements while still
maintaining the shape and a simple scaling factor in thefirst layer of the neural network, the image size that waschosen to be used is 1024×1024 pixels. A script wasdeveloped to download zoomed ×20 images, of size
256×256 pixels, from 26 human kidney WSIs of the BioAtlas repository14), and concatenate the images building
2,759 images of 1024×1024 pixels. Also, for each image,a text file was built with the class and coordinates of theglomeruli presented on them.
The kidneys on the images are healthy and diagnosed
with septic infarct, microscopic polyarteritis, malignant
hypertension, tuberous sclerosis, chronic obstructive
pyelonephritis, amyloidosis, interstitial nephritis, acuteand chronic pyelonephritis, atheroembolism, chronictransplant rejection, myeloma nephropathy, focal
segmental glomerulosclerosis, renal infarct, renal cell
carcinoma, acute tubular necrosis from ethylene glycol,
nephrocalcinosis, eclampsia, arteriolosclerosis, acutetubular necrosis, systemic lupus erythematosus and,polycystic kidney disease. The WSIs in the Bio-Atlas14)
repository are from laboratories from University of
Kansas, University of New South Wales, University of
Iowa, Medical College of Ohio Toledo, Universitas
Basiliensis, Switzerland and from AACR Workshop. 26
WSIs were selected to compose the training set and 4
slides to compose the test set. The training set has 25
H&E-stained WSIs and one PAS-stained WSI. All WSIs
in the test set are H&E-stained.
In the WSIs, 585 sclerosed and 3,383 functional
glomeruli were located and classified. We used data

Layer

No. ofLayers1º2º1
13º- 4º1

5°1

6º- 9º2

10º1

11º- 26º

8

27º

1

28º- 43º

8

44º

1

52º

4

53º

Type

Filters

Size offilters

Convolutional
Convolutional
Convolutional
Convolutional
Residual
Convolutional
Convolutional
Convolutional
Residual
Convolutional
Convolutional
Convolutional
Residual
Convolutional
Convolutional

32
64
32
64

3x3
3x3/2
1x1
3x3

128
64
128

3x3/2
1x1
3x3

256
128
256

3x3/2
1x1
3x3

512
256

3x3/2
1x1

512

3x3

1024
512
1024

3x3/2
1x1
3x3

Convolutional
Residual
Convolutional
Convolutional
Convolutional
Residual
Avgpool
Connected
SoftmaxOut
256x256
128x128

128x128
64x64

64x64
32x32

32x32
16x16

16x16
8x8

8x8
Global1000

augmentation techniques, with rotation and HSV color
map variations, generating 803,200 images in the set.
For the localization and classification of theglomeruli, the YBAT15) software was used. In thisapplication it’s possible to build a file with theclassification and coordinates of the objects in each
image in the structure necessary for network training. In
this procedure, besides the class of the object, twocoordinates are recorded that indicate the displacementfrom the left superior corner of the image (i.e., x and y)as well as the width and height (i.e., w and h) of the

39

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 2 Network architecture parameters configured in theexperiments performed.
Parameter

Value

Percentage for training

80%

Percentage for error analysis

20%

Fig. 4 Fragments of the 4 slides used in performance validation1)

Batch

16

resilient to these variations12). The WSIs in the test set
were detected with renal cell carcinoma, renal infarct,
systemic sclerosis and disseminated intravascular
coagulation, microthrombi microinfarcts. In the medicalroutine, is expected that the number of functional
glomeruli found will be much higher than the number ofsclerosed since the glomeruli may be compromised byseveral other diseases that are not in the hyalinization
stage and even a kidney that is undergoing ahyalinization process will not have all its glomeruli
compromised at the biopsy time. One of the validation
slides had a region with failed fixation and unfocused.
The images that presented this flaw were disregarded.

Subdivision

8

Width

512

Height512

Channel3

Momentum

0.9

Decay

5

Angle

0

Saturation

1.5

Exposure

1.5

Hue

0.1

rectangle containing the classified object.
For the training, the images were randomly
separated into two groups. The first, containing about
80% of the images, was intended for training, and thesecond, containing about 20% of the images, was usedby the neural network itself to calculate the loss averageduring training. Table 2 shows the hyperparameters
used for the experiments. For other parameters, theinitial settings of the YOLOv3 architecture weremaintained. Batch and sub-division influence the speedof training, which tends to be higher with larger batchprocessing. The parameters had to be adjusted according
to the processing capacity of the hardware used. In thiswork, a 9th generation Intel Core I7 processor, 16GB
RAM and Nvidia GeForce RTX2060 6GB GDDR6VRAM card were used. The values entered in width andheight are used to resize the images in the input level ofthe neural network. Images were resized to 512×512
pixels. The channel used was 3, because the input items
are RGB images. The parameters angle, saturation,
exposure, and hue define how additional images will be
generated for training.

3.2 Evaluation metrics
For the evaluation of results, was considered thenumber of True Positives TP (i.e., glomeruli found andcorrectly labeled), False Positives FP (i.e., areas with noglomerulus labeled as a glomerulus), False Negatives
FN (i.e., areas with glomerulus not labeled) and True
Negatives TN (i.e., regions with no glomerulus notlabeled as a glomerulus). The region considered for theTrue Negatives is each region of 512×512 pixels notlabeled in the image that does not have a glomerulus. A
glomerulus can be found in a wide range of sizes, but forx20 zoomed images, the size 512×512 pixels comprises
most glomeruli. The true positive is the glomerulus in
the image correctly labeled as a glomerulus. For theseresults, the images were verified manually with asupervision of a pathologist. The metrics evaluate how
well the method can report the quantities and thepercentage of the functional and sclerosed glomeruli.
For this purpose, the bounding box of the detected
glomeruli doesn’t need to precisely have its size. The
bounding box can have different sizes as long it
identifies the glomerulus correctly and is a rectangle
including just one correct glomerulus. A mark should be
penalized (considered as a marking error) if thebounding box is much larger than the glomerulus (i.e.,
more than 50% bigger than the glomerulus and which
the area is bigger than 262,144 pixels, which is the areaof 512×512 pixels square). Because the WSI was cut
into smaller pieces to be analyzed, when the WSI wasreconstructed, there are glomeruli with more than one
bounding box. The data passed to the post-processing
analysis algorithm where all duplicate marks were
eliminated from the count before the report wasconstructed.
We used as notations Recall R (i.e. sensitivity) in
Eq.(1), Precision P in Eq.(2), Specificity S in Eq.(3),3. Performance Validation
3.1 Experimental setup
For performance validation of the proposed method,
4 WSI containing sclerosed and functional glomeruli(i.e., 153 sclerosed and 540 functional glomeruli) wereused as a test set, additionally to those 26 used fortraining. The 4 chosen slides have different shades asshown in Fig. 4, and different proportions of functionaland sclerosed glomeruli. The slides in the test set werechosen to analyze the response to different diseases anddifferent shades, once one of the main challenges in thecomputational interpretation of WSIs is the color
variations due to the slide preparation, and scanning.
The algorithms used in digital pathology have to be

40

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

0.46 and 0.53) that exclude a higher number of false
positives compared to true positives represented with avertical line. The test set were labeled again by theneural network, considering just images with confidence
higher than each threshold.

True Negative Rate RTN in Eq.(4), Accuracy A in Eq.(5),
Jaccard coefficient J in Eq.(6), False Positive Rate RFP
in Eq.(7), False Negative Rate RFN in Eq.(8), F1-Score
F1 in Eq.(9) and Matthews Correlation Coefficient M inEq.(10)17),18). Among these metrics, accuracy stands out
as it is commonly discussed and analyzed in clinicalpractice. To evaluate our method in comparison withother similar works, we also used the same metrics usedin recent similar research (i.e., Precision, Recall, F1Score and Accuracy).
୘୔R = ሺ୘୔ା୊୒ሻ

(1)

P = ሺ୘୔ା୊୔ሻ

(2)

S = ሺ୘୒ା୊୔ሻ

(3)

RTN = ሺ୘୒ା୊୒ሻ

(4)

A = ሺ୘୔ା୊୔ା୘୒ା୊୒ሻ

(5)J = ሺ୘୔ା୊୔ା୊୒ሻ

(6)

RFP = ሺ୊୔ା୘୒ሻ

(7)

RFN = ሺ୊୒ା୘୒ሻ

(8)

F1 = ሺ୔ାୖሻ

(9)

୘୔

୘୒

୘୒

୘୔ା୘୒

୘୔

୊୔

୊୒

ଶ∗୔∗ୖ

M=

3.4 Results and discussion
The results of the threshold analysis for theexperiments are shown in Table 3 and Fig. 6.

୘୒∗୊୔

ඥ(୘୒ା୊୒)(୘୔ା୊୔)(୊୔ା୘୒)(୘୔ା୊୒)(10)

3.3 Threshold analysis
When an image is run in the trained YOLOv3
architecture, the result presents the probability of a given
object labeled of belonging to the identified class. It wasobserved that the number of false positives could be
reduced by properly selecting the minimum probability
that should be considered for assuming that a glomerulus
was found. The performance evaluation test with thethreshold of 0.20 also presented 13 marks with classerror (i.e., object was correctly found but wrong
classified), where 12 were items labeled twice, with thecorrect and with the wrong classification. It wasobserved that this problem could also be avoided withthe correct threshold.
To analyze how the class probabilities can be used to
improve the metrics, 490 marks (i.e., about 70% of themarks) were classified into FP and TP and theirprobabilities were plotted in Fig. 5 (spread vertically),
so it would be possible to understand how a change inthe threshold value can change the number of TPs andFPs. In this graph, we chose 4 thresholds (0.20, 0.32,

Fig. 5 Graph of the class probabilities of the markings made by thesystemTable 3 Results of threshold experiments.

Threshold
0.20
0.32
0.46
0.53

41

Recall
0.97300.9711
0.96820.9655

Precision
0.87390.9161
0.95880.9324

Accuracy
0.95780.97320.98100.9721F1-Score
0.9202
0.94270.9634
0.9535IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 4 Results obtained in Experiment considering both classes
using 0.46 as cut-off threshold.

Metric

R
P
S
RTN
A
J
RFP
RFN
F1

Fig. 6 Graph of Threshold experiments. R: Recall; P: Precision;
A: Accuracy; F1: F1-Score

M

The experiment with the threshold of 0.46 waschosen to be better discussed once it presented the best
results for precision, accuracy, and F1-Score. As
expected, comparing this and the experiment with thestandard value (i.e., 0.20), there was a decrease in recall
and an increase in other parameters. While the Recall
decreased 0.0048 percentual points, Precision increased
0.0849, Accuracy 0.0232 and F1-Score 0.0432
percentual points. This means that the evaluated method
has lost some sensitivity in finding the right items but
has become significantly less likely to identify wrong
objects. The class error marks were reduced to 1 with noitem labeled twice.
The results of the experiment separated by WSI arepresent in Table 4. The slides were analyzed separately
so it becomes easier to identify the characteristics that
may interfere with the results. Slide 3 for example,
doesn’t have sclerosed glomeruli and has the best results
for recall and accuracy, so these parameters can be
expected to be higher in samples with a smaller numberof sclerosed glomeruli.
As a reference, to verify how the PAS-stained WSI
present in the training set influences the results,
experiments were performed by removing this WSI.
Table 5 shows the results for the same thresholds usedin the previous experiments. The results were a bit
deteriorated for some metrics by comparing with Table
3 because the number of images used for the training
was considerably reduced by removing the PAS-stained
WSI. Therefore, to improve the results, we should
increase the training data stained by PAS, and this issue
is one of our future works.

S1
0.96950.94080.98200.99500.96700.91380.01000.03000.95500.9475

S2
0.97350.9607
0.9921
0.99600.97730.93630.00600.02600.9671
0.9622

S3
1
0.98481
10.99720.98490.000900.9924
0.9920S4
0.93000.94890.99600.99690.98260.88570.00220.07000.93900.9368Total
0.96820.95880.99250.99700.98100.93010.00480.03140.96340.9596

Table 5 Results of experiments with the training set of 25 H&Estained WSI by removing the PAS-stained WSI.

Threshold
0.20
0.32
0.46
0.53

R
0.92580.89210.86390.8607

P
0.89490.90560.9201
0.9224

A
0.97790.9764
0.97570.9757F1
0.90720.89850.89100.8903are sclerosed and the coordinates in the WSI where theglomeruli are located, as shown in Fig 9, where thelocation of the items is composed by the coordinates ofthe position of each glomerulus found and the size of thebox that is marking each one.
Unlike some other related works, this work not justpresents the WSI reconstructed with the glomeruli andthe class confidence values labeled, but also theinformation about the number of glomeruli found, thepercentage of them that are sclerosed, and the positionof each glomerulus is reported, facilitating the use in
medical routine. The duplicate glomeruli are avoided in
the count checking the proximity between markings.
The results still present false positives (i.e., objects B
and D in Fig. 10), which generally occur because of thesimilarity of other kidney units with the glomeruli,
especially with sclerosed. Figure 10 also shows
examples of correct labels and a false negative area.
High results were obtained, mainly in terms ofaccuracy, true negative rate, and sensitivity. The three
metrics consider the value of true negative, so it isconcluded that the method has a good ability to not
identify as a glomerulus an area where there is noglomerulus. The metric that brought the lowest result
was the Jaccard coefficient, a metric that does notconsider the values of true negatives, and to increase thisvalue, the number of false positives and false negatives
need to be reduced. Still, the experiment obtained goodresults for all metrics, indicating the method's ability tofind and classify items correctly and the ability to notmake wrong marks.
In the test data, there were also images smaller than

3.5 Analysis of the results
Our method identifies the location of sclerosed andfunctional glomeruli and returns the quantities andpositions of the items found. The method returns animage of a WSI with the found glomeruli marked by aframe (i.e., rectangle box) and with their classconfidence value, as shown in Fig. 7 and Fig. 8, andreports the percentage of sclerosed glomeruli, the totalof glomeruli found, the number of these glomeruli that42

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 6 Results obtained in experiment considering only functional
glomeruli.

Metric
R
P
S
RTN
A
J
RFP
RFN
F1
M

S1

0.99120.99120.99810.99900.99500.98260.00090.00880.99120.9683

S2

0.9916
0.99150.99810.99900.9951
0.98330.00100.0084
0.99160.9708S3

10.98481
10.99730.98480.000900.99240.9807

S4

0.98630.98630.99870.9996
0.99700.97300.00040.01370.98630.9358Total

0.99230.98850.99870.9994
0.99610.98090.00080.00770.99040.9640Fig. 7 Example of image labeled by the proposed method
Table 7 Results obtained in experiment considering only sclerosed
glomeruli.

Metric
R
P
S
RTN
A
J
RFP
RFN
F1
M

Fig. 8 WSI labeled by the evaluated method

S1

0.92000.83630.9856
0.99640.96180.77970.00810.08000.87610.8631

S2

0.90620.85300.99470.99730.9751
0.78380.0044
0.09370.87880.9288

S3

1
1
0
-

S4

0.77780.84000.99750.99740.98380.67740.00170.22220.80770.7547

Total

0.86800.8431
0.9926
0.99780.97360.74700.00470.13200.85420.8489those used in the training, corresponding to the edges ofthe slide. These images were analyzed separately andhad an unsatisfactory result, showing the importance of
maintaining the dimensions of the images used in the
network.
Fig 9 Percentage of sclerosed glomeruli, total number of glomeruli,
number of sclerosed glomeruli, the location of the items and thesize of the box marked in the output file

3.6 Class analysis
Analyzing the results of each item separately, inTable 6 for functional and Table 7 for sclerosed
glomeruli, the results were worse for the sclerosed,
which can be explained by the smaller number of imagesof sclerosed glomeruli used in the training compared tofunctional, by the variety of the item searched andsimilarity with other elements shown in WSI. Bloodvessels undergoing sclerosis are still commonly
mistaken for sclerosed glomeruli, thus, as future work,
traditional segmentation techniques can be used to avoid
these misclassifications.
3.7 Comparison with other approaches
The methods used in works that have similar goalsof using neural networks to identify and classify
sclerosed and non-sclerosed glomeruli (i.e. Marsh et al.7),
Gallego et al.19), and Bueno et al.20)) are presented inFig. 10 A: False Negative of Sclerosed Glomerulus; B: False Positiveof functional glomerulus; C: True Positive of Functional
Glomerulus; D: False Positive of Sclerosed Glomerulus E:
True Positive of Sclerosed Glomerulus

43

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 8 Configurations of similar research7),19),20)

Proposed

Marsh et al.7)

Gallego et al.19)

Bueno et al.20)

No. of WSI
Method Used

26
Identification andclassification ofbounding boxes in asingle CNN architecture48
CNN with semantic
segmentation, PatchBased model + fully
convolutional model51
CNN withsemantic
segmentation +post-processing

47
CNN with semantic
segmentation

No. Sclerosed
No. Functional
Image’s size

585
3383
1024×1024

870
2997
448×448 and 1024×1024

564
1865
256×256303
942
2000×2000

Total of Images
after DA
CNN

8032004713961126733

25320

YOLOv3

VGG16

U-Net

SegNet-VGG19

Table 9 Results from similar research7),19),20)

AuthorPrecision

Recall

F1-score

Accuracy

Proposed

0.95880.96820.96340.9810Marsh et al.

0.70990.79140.7483-

Gallego et al.

0.96300.8610--

Bueno et al.

0.74990.92050.81910.9667

Table 10 Results in similar research separated by class 7),19),20)

MetricMarsh et al.
Func.
Scle.

Gallego et al.
Func.
Scle.

Func.

Scle.

ProposedP

0.81280.60700.97600.90600.9885

0.9431R

0.8852

0.69770.91700.66700.99230.8680F1

0.84750.64920.94500.76800.99040.8542structures) using convolutional neural networks incascade, and another method using one unique neuralnetwork for 3 class segmentation problem similar to the
proposed method (i.e., non-glomerular, normal
glomerular, and sclerosed structures). For a faircomparison, in this paper, only the results involving 3classes and a single CNN will be analyzed. As an extra
information, Bueno et al. used just images stained withPAS.
Marsh et al.7) used H&E-stained frozen samples
images. Gallego et al.19) used images stained with PAS
and H&E. Another difference between the methods was
the size of the images used. While Marsh et al.7) used
1024×1024 pixels image sets and isolated glomeruli in
448×448 pixels images, Gallego et al.19) used 256×256
pixels isolated glomerulus images and Bueno et al.20)
used isolated glomeruli images of 2000×2000 pixels.
Observing that the best results for the metrics analyzed
were obtained in the proposed method, it is concluded
that the analysis of the context where the object isTable 8 and the results are shown on Table 9.
Each author used a different method and different
convolutional neural networks, although they all usedsemantic segmentation. The problem and method
discussed in this work can’t be exactly compared withthese other works, once that they aim to define which
regions of the WSI belong to the health and sclerosed
glomeruli, while this work aims to identify the locationand classification of the glomeruli and identify thequantities and percentage of glomeruli that are sclerosed.
We understand that when comparing the metrics with theones obtained in similar works, we are not looking forthe best method between them, but rather consideringthat each one could be better in different situations.
Marsh et al.7) used Patch-Based and Fully
Convolutional Model in the training. Gallego et al.19)
used size analysis as post-processing. Bueno et al.20)
used two methods in the paper, i.e., one withsegmentation problem similar to the proposed method(i.e., non-glomerular, normal glomerular, and sclerosed

44

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

inserted is important for neural network learning. Somefactors could explain why the proposed experiments had
higher results compared to those presented by Bueno etal.20). Among them is the least scaling factor, as it is
known that in the scaling process the image may lose
relevant information. In addition, even though Bueno's
research used a higher number of slides, the total numberof glomeruli tagged in the present work is higher.
In Table 10, we can see that our method achieves
significantly higher results in the identification andclassification of sclerosed glomeruli compared to theseother methods. Another observation is that theperformance for sclerosed glomeruli detection is lower
than for functional glomeruli in these methods. This isbecause the smaller number of sclerosed glomeruli in the
database and blood vessels being wrongly labeled assclerosed glomeruli by the neural network. This issue
should be solved in the future.

4) J. Ng, Y. Luo, K. Phua, A. Choong: "Acute Kidney Injury in
Hospitalized Patients with Coronavirus Disease 2019 (COVID-19): A
Meta-Analysis". J. Infect. - Lett. to Ed. 81, pp. 661–664 (2020).
5) F. Zhang, Y. Liang: "Potential Risk of the Kidney Vulnerable To
Novel Coronavirus 2019 Infection". Am J Physiol Ren. Physiol 318,
pp. 1136–1137 (2020).
6) R. Bonegio, D. Salant: "Glomerular Diseases". DECKER Intellect.
Prop. INC- ACP Med. 2011;1-27 (2011).
7) J. Marsh, M. Matlock, S. Kudose, T. Liu, T. Stappenbeck, J. Gaut , J.
Swamidass: "Deep Learning Global Glomerulosclerosis in Transplant
Kidney Frozen Sections". IEEE Trans. on Med. Imaging (2018)
doi:10.1109/TMI.2018.2851150.
8) Y. V. Eycke, J. Allard, I. Salmon, O. Debeir, C. Decaestecker: "Image
Processing in Digital Pathology: An Opportunity to Solve Inter-Batch
Variability of Immunohistochemical Staining". Sci. Rep. (2017).
doi:10.1038/srep42964.
9) J. Redmon, S. Divvala, R. Girshick, A. Farhadi: "You Only Look
Once: Unified, Real-Time Object Detection". Proc. of the IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition (2016). doi:10.1109/CVPR.2016.91.
10) S. Skansi: Introduction to Deep Learning. From Logical Calculus to
Artificial Intelligence, Springer (2018).
11) J. Ferreira, J. Raniery, N. Correia, P. Marques: "Aprendizado De
Máquina Na Atenção À Saúde Humana". Computação Brasil - Revista
da Sociedade Brasileira de Computação (SBC) 39oed pp.37–39 (2019).
12) A. Madabhushi, G. Lee: "Image Analysis and Machine Learning in
Digital Pathology: Challenges And Opportunities". Med. Image Anal.
33, pp.170–175 (2016).
13) M. Najafabadi, F. Villanustre, T. Khoshgoftaar, N. Seliya, R.
Wald, E. Muharemagic: "Deep Learning Applications and Challenges
in Big Data Analytics". J. Big Data. doi:10.1186/s40537-014-0007-7(2015).
14) Bio-Atlas, http://zfatlas.psu.edu/, NIH grant 5R24 RR01744, Jake
Gittlen Cancer Research Foundation, and PA Tobacco Settlement Fund.
Slides 569, 603, 651, 657, 671, 704, 717, 832, 837, 842, 849, 868, 992,
1060, 1118, 1119, 1150, 1154, 1289, 1292, 1298, 1307, 1313, 1322,
1330, 1379, 1434, 1555, 1571, 1575, http://bio-atlas.psu.edu/
view.php?s =603*.
15) S. Draining: Ybat - YOLO BBox Annotation Tool.
https://github.com/drainingsun/ybat
16) J. Redmon, A. Farhadi: "YOLOv3: An Incremental Improvement".
Tech Rep. (2018).
17) D. Powers: "Evaluation: From Precision, Recall , F-Factor To ROC,
Informedness, Markdness & Correlation". J. Mach. Learn. Technol. 2,
pp.37–63 (2011).
18) D. Chicco, G. Jurman: "The Advantages of the Matthews
Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary
Classification Evaluation". BMC Genomics, doi:10.1186/
s12864-019-6413-7 (2020).
19) J. Gallego, Z. Swiderska-Chadaj, T. Markiewicz, M. Yamashitae, A.
Gabaldong, A. Gertych: "A U-Net Based Framework to Quantify
Glomerulosclerosis in Digitized PAS and H&E Stained Human
Tissues". Comput. Med. Imaging Graph. 89, (2021).
20) G. Bueno, M. Fernandez-Carrobles, L. Gonzalez-Lopez, O. Deniz:"Glomerulosclerosis Identification in Whole Slide Images Using
Semantic Segmentation". Comput. Methods Programs Biomed. 184,
105273 (2020).

4. Conclusion
In this paper, we evaluated the performance of
YOLOv3 as deep neural network to identify the positionand classify glomeruli, into functional and sclerosed, in
WSIs from kidney biopsies. Also identify the total
number of glomeruli and the rate of sclerosed glomeruli,
to select WSIs that must be discarded for not having theminimum number of glomeruli for evaluation and toorder the WSIs by damage.
Our method employs YOLOv3 with 53 layers neuralnetwork, to identify and classify glomerulus between
functional and sclerosed. We prepared a database withimagesfrom
Bio-Atlas
repository,
properlyconcatenated and classified, with a supervision of apathologist, for training the neural network.
Our method achieved a high-performance withrecall of 96.8%, precision of 95.9%, accuracy of 98.1%,
and an F1 score of 96.3%. The results obtained weresignificant compared from those obtained in similar
research. Also, we can show the WSIs reconstructed andwith the glomeruli labeled, and a file containing thenumber of glomeruli found, the coordinates of eachglomerulus on the slide and the percentage of sclerosed
glomeruli, being useful in the medical routine.
As future works, we should increase the databasewith PAS-stained WSI and verify the results also withthis kind of stain. Also, we should investigate anotherapproach, as neural network cascade and semantic
segmentation. Besides that, we are planning to extend
our method to other specific diseases in medical routine.

(Received May. 27, 2021)(Revised Jan. 31, 2022)

References
1) A. Guyton, J. Hall: Tratado De Fisiología Medica. (2017).
2) V. Kumar, A. Abbas, K. Aster, J. Robins: Patologia Básica. (2013).
doi:10.1016/b978-0-7020-3369-8.00001-x.
3) B. Henry, G. Lippi: "Chronic Kidney Disease is Associated with
Severe Coronavirus Disease 2019 (COVID-19) Infection". Int.
Urology and Nephrology. 52, pp.1193–1194 (2020).

45

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Kiyoshi TANAKA (Fellow)

Thalita Munique COSTA(Student Member)He received his B.S and M.S. degrees
in Electrical Engineering and Operations
Research from the National Defense
Academy, Yokosuka, Japan, in 1984 and1989, respectively. In 1992, he received theDr. Eng. Degree from Keio University,
Tokyo, Japan. In 1995, he joined theDepartment
ofElectricalandElectronic Engineering, Faculty ofEngineering, Shinshu University, Nagano,
Japan, and currently, he is a full professor
in the academic assembly (Institute ofEngineering) of Shinshu University. He is
the former Vice-president of Shinshu
University in charge of international affairs
and the former director of the Center for
Global Education and Collaboration of
Shinshu University. His research interests
include image and video processing, 3D
point cloud processing, information hiding,human visual perception, evolutionary
computation, multi-objective optimization,
smart grid, and their applications.
Currently, he is the president of IIEEJ, afellow of IIEEJ, a member of IEEE, IEICE,IPSJ, JSEC, and so on.

She received her B.S. and M.E. degrees
in Electronic Engineering, and in Science,
Area of Concentration: Biomedical
Engineering, from Federal University ofTechnology - Paraná (UTFPR), Brazil in2018 and 2020, respectively. She is now a
Dr. Eng. Candidate in Telecommunication
System at Shinshu University, Nagano,
Japan. Her research interests include Digital
Pathology, Artificial Intelligence, and Image
Processing.

Lourenço BARBOSA

He graduated as an Electronics
Technician at Federal Technological Centerof Rio Grande do Sul (CEFET) in 2003,
bachelor’s in Information Systems at Paraná
Society of Education and Informatics (SPEI)
in 2010. In 2020, he received the title ofMaster of Science from the Federal
University of Technology - Paraná(UTFPR), He is a Technical Leader in
Software Evolution and Supports at NTT
DATA Business Solutions Brazil. His
research for the medical field encompasses
the topics of image/video processing, deeplearning, and machine learning. Currently,
as a candidate for a Doctorate at UTFPR.

Fabio SCHNEIDER

He received his B.S and M.S. degrees
in Electrical Engineering and Biomedical
Engineering from the Federal University ofTechnology - Paraná (UTFPR), Brazil, in
1989 and 1995, respectively. In 2006, hereceived the Ph.D. degree from Universityof Washington, Seattle, U.S.A. In 1995, hejoined the Department of Electronics andTelecommunication at UTFPR, Brazil
where he is currently a full professor. He is
a former Vice President for Research andPost Graduate School of UTFPR. His
research interests include image and video
processing, Artificial Intelligence and Deep
Learning development and application,
ultrasoundimaging,computationalalgorithms, and architectures.

Yoko USAMI
She received her Ph.D. degree from
Tokyo Medical and Dental University,
Tokyo, Japan in 2013. She joined theDepartment of Laboratory Medicine at
Shinshu University Hospital, Japan where
she is currently a vice tech supervisor. Her
research interests include Laboratory
Medicine, Clinical Chemistry, and ClinicalLaboratory Immunology.

Mai IWAYA
She received her M.D. degree from
Shinshu University in 2005. She completed
anatomical pathology residency at Shinshu
University Hospital in 2011 and completed
gastrointestinal pathology fellowship at
University of Toronto, Canada in 2018. She
received the Ph.D. degree from Shinshu
University in 2019. She joined thedepartment of laboratory medicine at
Shinshu University in 2018 and currently,
works as an associate professor/lecturer. Her
research
interests
includeartificial
intelligence in the pathology field.

46

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2021Part II --Measurement of Eye Size Illusion Caused by Thickness of Eyeliner
on Double-Eyelid Eyes
Rena OKURI† , Shuhei KODAMA† †† , Tokiichiro TAKAHASHI† †† (Fellow )† Tokyo Denki University , †† ASTRODESIGN, Inc.

<Summary> Eyeliner is one of the makeup methods that enhances the attractiveness of female by ageometric illusion that makes the eyes appear larger. Eyeliner has no ﬁxed shape, and its thickness andlength can be freely adjusted. In this paper, we experimentally verify the relationship between thickness ofeyeliner and the perceived eye size. In addition, by examining the sense of incongruity of the eyeliner, weclarify the optimal thickness of eyeliner as makeup to make the eyes appear larger. The results showed thatthe thicker eyeliner, the larger the eyes were perceived to be. However, the excessive thickness of eyeliner
increased the sense of incongruity and reduced the illusion eﬀect. There were gender diﬀerences in the resultsof these experiments. For male, thickness of eyeliner had a signiﬁcant eﬀect on the perception of eye size,
while for female it had no signiﬁcant eﬀect.
Keywords: makeup, eyeliner, face, eyes, geometric illusion1.

ness of eyeliner as eye makeup by conducting a subjective evaluation questionnaire of the sense of incongruity
for the eyes with the eyeliner of each thickness. These
results are used to verify the best thickness of eyeliner tomake the eyes appear larger. Since it is known that there
are gender diﬀerences in the perception of attractivenesswith makeup5), we also verify the gender diﬀerences inIntroduction

In recent years, with the growing interest in facial
beauty and attractiveness, facial imaging studies havebeen actively conducted in many ﬁelds. According to experimental studies on facial morphology, large eyes haveone of the features that increase the attractiveness ofa female face1) because they give the sense of childishness and femininity2),3). Makeup is a means of improving

the results.

2.

the attractiveness of a female face4),5). In particular, eye

Related Work

Jones et al.7) tested the hypothesis that the apparent
size of facial features (eyebrows, eyes, nose, and mouth)changed by makeup. They found that the eyes appeared
to be larger after applying makeup both by the participants themselves and by professionals. Furthermore, they

makeup improves the attractiveness of a female face bymaking the eyes appear larger and emphasizing them5).
Matsushita et al.6) revealed that eyeliner, a makeup
method in which a line is drawn around the eyelash line,
has the eﬀect of making the eyes appear larger by the
Delboeuf illusion. They measured how much the eyes
appear to be larger due to the illusion (called the amountof illusion) by adjusting the color and range (upper eyelid
only, upper eyelid + lower eyelid) of eyeliner.
In this paper, we assume that thickness of eyeliner af-

compared whether the change in apparent feature sizedue to cosmetics was caused by the enhancement of ﬁne
details or by the coarse changes in the entire region byﬁltering the spatial frequency of the face image. As a result, the eyes appeared larger with makeup regardless ofthe spatial frequency, indicating that the eﬀect of cosmetics on apparent feature size is not limited to whether the
spatial frequency is low or high. These results support
the idea that modiﬁcation of the apparent size of facial
features by makeup can enhance facial attractiveness.
Several studies focus on the eyes among the facial
features. Morikawa et al.8) found that the eyes withfects the amount of the illusion in addition to the color
and range of eyeliner as Matsushita et al.6) focused on.
We measure the amount of illusion between thickness ofeyeliner and the size of the double-eyelid eyes to clarify the relationship between thickness of eyeliner and theamount of illusion using a psychophysical measurement
method. Moreover, we investigate the appropriate thick-47

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
520px

140px

260px
ﬂip horizontally(a) original image (le eye)(b) ﬂipped image (right eye)Fig. 1 Delboeuf illusion

eye shadow appear larger in their experiment using psychophysical methods. They showed that the perceived
size of the eyes with eye shadow is related to the assimilation eﬀect between the eye and the eye shadow, andthe assimilation eﬀect between the eyebrow and the eyewhen the eye shadow ﬁlls the area between the eye andthe eyebrow. In addition, the perceived the eyes with eye
shadow is aﬀected by the viewing distance. They also
experimentally showed that the eye size illusion causedby eye shadow and eyebrows is based on the same psychological eﬀect as the Delboeuf illusion. The Delboeuf
illusion is the illusion that the inner circle of a double circle appears to be larger than that of a single circle (theinner circle is the same size as the single circle), as shownin Fig. 1. This illusion is caused by the assimilation of
the outer and inner circles of the double circles.
Matsushita et al.6) experimentally showed that eyeliner
has the eﬀect of making the eyes look larger based on the
Delboeuf illusion and that the perceived eye size changes
depending on the color and range of eyeliner. They veriﬁed the eﬀect of color and range of eyeliner on the eye(c) To (both eyes without eye makeup)

scaling areas(d) Scaling areas of comparison smulus image

Fig. 2 Procedure for generating To which is both eyes
image without eye makeup and the comparison
stimulus image generated by scaling To image

and orientation of the display should be ﬁxed, and theluminance should be measured with a color luminancemeter (CS-100A, Konica Minolta) to be approximately
250 cd/m2 . The aspect ratio of all the displays used in
the experiment was 16:9, and their vertical sizes H of thedisplays were between 14.9 cm and 33.6 cm. The observation distances between the subjects and the displays
were set to 3H. All participants were proﬁcient in IT literacy, and they prepared the experimental environment
as instructed and followed the experimental procedures.
3.2

size illusion, but they did not verify the eﬀect of eyeliner
thickness on the illusion.
In this paper, we experimentally examine the eﬀect ofeyeliner thickness on the perceived eye size. Furthermore, we discuss the results focusing on gender diﬀerences, which have not been veriﬁed in the previous studies6),8).

3.

Standard stimulus images and comparison
stimulus images

The comparison stimulus images and the standard
stimulus images used in the experiment were generated
as follows.
The pixels resolution of both comparison and standard
stimulus images was 520 px× 140 px, and they were displayed on the screen so that 1 px of the image was 0.2
mm (rounded to the second decimal place).
In this experiment, we generate and use symmetricaleyes images as stimulus images. To generate symmetrical eyes images, ﬁrst, we prepare an image of the eye toIllusion Measurement Experiment

3.1 Experimental environment and apparatus
Since it was diﬃcult to gather all participants for theexperiment due to the COVID-19 pandemic, we distributed the experimental program and instructed eachparticipant to prepare the execution environment of theexperimental program before conducting the experiment.
During the measurement, we instructed that cold lighting
should be used, the brightness should be constant, andthe direction of the lighting should be ﬁxed. The position

the left of the root of the nose (left side of the screen)of a Japanese woman with double-eyelid and no makeup(Fig. 2(a)), as an original image. The width of each eye
is set to about 30.0 mm, based on the average interocularbreadth of 36.6 mm and biectocanthion breadth of 90.0mm, which are typical for Japanese young adult female.9)48

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

・・・Standard Smuli

To(0.0mm)

88%T₁(1.2mm)

・・・100%(To)

116%

Fig. 4 Comparison stimulus images (88%，100%，116%)

T₂(1.8mm)

T₃(2.4mm)

eyeliner

T₄(3.0mm)

outer corner

thickness

inner corner
medial canthus

T₅(3.6mm)

T₆(4.2mm)T₇(4.8mm)

lateral canthus
6px

Fig. 3 Original eye image without eye makeup (To ) andthe standard stimulus images (T1 to T7 )18px

Fig. 5 Deﬁnition of eyes and eyeliner

follow the contour of the eye. In this experiment, the inner corner of the eye is set between the medial canthus
and the point 18 px inward from the medial canthus. The
outer corner of the eye is deﬁned as the point between thelateral canthus and the point 6 px outward from the lat-

Next, we generate an image of the right eye (right sideof the screen) by ﬂipping the original image horizontally(Fig. 2(b)). Finally, we generate a symmetrical image ofboth eyes To (thickness original) as shown in Fig. 2(c),based on the original image and the right eye image.
As standard stimulus images, we generate seven images
T1 , T2 , T3 , T4 , T6 , and T7 with eyeliner of thickness 6, 9,

eral canthus. Figure 5 shows the deﬁnitions of eyes andeyeliner.
3.3

12, 15, 18, 21, and 24 px (1.2, 1.8, 2.4, 3.0, 3.6, 4.2 and 4.8
mm in real size) drawn with paint software on To which

Experimental procedure

For each standard stimulus, we obtain an equivalent
comparison stimulus, i.e., the points of subjective equivalence (PSE), and deﬁne the scaling rate of the comparison stimulus as the amount of illusion. The procedure of
the experiment is shown in Fig. 6.
We used the staircase method to measure the amountof illusion. First, Image 1 of Fig. 6 (a gray scale image,
R=G=B=153) is displayed for 7000 ms. Then, Image
2 of Fig. 6, in which standard and comparison stimulus
are aligned on the left and right sides of the image, isis the image of both eyes without eye makeup. The ith
image Ti is an image with eyeliner of thickness (i × 3 +3) px or (i × 0.6 + 0.6) mm drawn by paint software.
Only the left eye part of To and the standard stimulus
images (T1 to T7 ) are shown in Fig. 3.
As comparison stimulus images, we generate images inwhich the area around each eye of To (Fig. 2(d)) is scaled
by 2% from 88% to 116% without changing the aspectratio of the original image in the horizontal and verticaldirections. Here, To is the original image of the comparison stimulus image, scaled to 100%. For gaps in theimage caused by scaling, we blur so that the merged area
would not seem unnatural. Figure 4 shows only the left
eye part of the comparison stimulus images (88%, 100%,

displayed on a white background for 3000 ms. The participants compare both stimulus images, then answer inthree ways: “I feel that the eyes on the right side of theimage are larger,” “I feel that the eyes on the left sideof the image are larger,” or “equal/not sure.” Next, theImage 1 is displayed again for 4000 ms. Then, the task
is repeated to answer the question by comparing the eye
sizes of the stimulus images displayed on both the left
and right sides of Image 2 and to watch Image 1 on thedisplay.
The comparison stimulus is changed by the partici-

116%). It is unknown to what extent thickness of eyeliner
aﬀects the apparent eye size both vertically and horizontally. We conducted the experiment assuming that theapparent eyes are isotropically scaled in horizontal andvertical directions. It would be a future task to verify thescaling ratios of the apparent eyes are either isotropic or
anisotropic.
When the eyeliner is drawn in a smooth curve that follows the contour of the upper eyelid margin, the thicknessof eyeliner is deﬁned as the distance extended verticallyupward from the curve. The eyeliner gradually becomesthinner toward the inner and outer corners of the eye topant’s judgment in each trial. When the participants
judge that the standard stimulus is larger, the staircase
direction of the comparison stimulus is upward (88% to116%), while when they judge that the comparison stimulus is larger, the staircase direction of the comparison
stimulus is downward (116% to 88%). The measurement

49

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Image 2Sub-images of standard
and comparison smulus
are aligned side by side

Image 1gray image(1) Display Image 1 for 7000ms.
(2) Display Image 2 for 3000ms.
While displaying, parcipants answer which side of the sub-images in Image 2 they feel eyes are larger on.
(3) Display Image 1 for 4000ms.
(4) Update the sub-images in Image 2 according to the answer in Step (2).
Repeat steps (2) to (4).

Fig. 6 Experimental procedure for measuring the amount of illusion

is completed when the direction of ascending or descending of the comparison stimulus is reversed 8 times, referring to the experimental method of Matsushita et al.6).

Amount of illusion(perceived eye size)[%]110

Then, the mean of the stimulus intensity (%) of 8 times of
the comparison stimuli is used as the amount of illusion
under the condition of that standard stimulus.
The standard stimulus and the comparison stimulus
are displayed on the left or right side of the screen at
random. A break time of at least one minute is provided
before changing the standard stimulus image and moving

102
100
1.2
(T₁)

1.8
(T₂)

2.4
(T₃)

3.0
(T₄)

3.6
(T₅)

4.2
(T₆)

4.8
(T₇)

Eyeliner thickness[mm](about 105.7%, 107.9%, 109.6%, 110.1%, 109.4%, 107.0%,
106.8% in area, respectively). For example, the amountof illusion of 105% makes the eyes appear to be 5% larger
than their real size. The amount of illusion tended to increase when thickness of eyeliner was 1.2 mm to 3.0 mm,
to reach the maximum at 3.0 mm, and then to decrease
from 3.0 mm to 4.8 mm. We refer to the detailed analysisof the measurement results in section 4.3.

Participants

Thirty-three participants (mean age 22.7 years, SD =1.48, 17 males and 16 females) with normal visual acuity
and color vision participated. We explained the contentsof our experiment in verbal and written, then obtained
the participants’ consent before conducting the experiment.
4.2

104

Fig. 7 Thickness of eyeliner and the amount of illusion(Markers indicate mean values of the amount of
illusion and error bars indicate 95% conﬁdence
intervals)

Experimental Results and Statistical
Analysis

4.1

106

98

to the next measurement.

4.

108

4.3

Experimental results

Statistical analysis

We analyzed whether the eyeliner thickness factor affects the amount of illusion using the analysis of varianceand the eﬀect size for the results of the illusion measurement experiment. Furthermore, we compared between
groups using the multiple comparison method, 95% conﬁdence intervals, and eﬀect sizes. The results are described
below.

Figure 7 shows the results of the illusion measurement experiment. The amount of illusion in Fig. 7 indicates the mean amount of illusion of the participants.
In the standard stimulus images T1 to T7 , the amount of
illusions (perceived eye size) were about 102.8%, 103.9%,
104.7%, 104.9%, 104.6%, 103.4%, 103.3%, respectively

50

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

4.3.1

Eﬀect of eyeliner thickness factor

Do you feel the sense of incongruity?
2: a lot
1: somewhat
0: not at all

To analyze the eﬀect of eyeliner thickness on the
Eyeliner thickness[mm]double-eyelid eye size illusion, we checked the normality of the experimental data and the equality of variance
between groups and used the one-way ANOVA of withinsubjects factors. The results of the analysis of varianceshowed that the eﬀect of eyeliner thickness was statistically signiﬁcant (F(6, 32) = 3.80, p = .00 <.05). According to the eﬀect size η 2 , which is an indicator of the realeﬀect size regardless of the sample size, the strength of therelationship between the amount of illusion and thicknessof eyeliner was slightly medium10)(η 2 = 0.04). In otherwords, thickness of eyeliner aﬀected the perception of thesize of double-eyelid eyes.
4.3.2 Intergroup comparison
By comparing each group of T1 to T7 , we can grasp the

1.2(T₁)
1.8(T₂)2.4(T₃)3.0(T₄)
3.6(T₅)
4.2(T₆)
4.8(T₇)
0

25

50

75

100(%)

Fig. 8 Subjective evaluation questionnaire results

5.

Subjective Evaluation Questionnaire

Although there is no rule for thickness of eyeliner, thickeyeliner is not appropriate as makeup if it causes the senseof incongruity. Thus, we investigated the sense of incongruity for each thickness of the eyeliner in the illusion
measurement experiment using a subjective evaluationquestionnaire. Twenty-nine participants (17 males and
12 females, aged 20-28 years) answered the questionnaire.

trend of the change in the amount of illusion. We compared the diﬀerences in the means between the groups
using the multiple comparison method (Tukey-Kramer
method) and found that T3 , T4 , and T5 were signiﬁcantly
larger than T1 . Therefore, it can be said that the illusion
increased gradually as thickness of eyeliner increased from
T1 to T4 , and was the largest around T4 . On the other

5.1

Contents

We conducted a questionnaire to investigate the senseof incongruity of the standard stimulus images T1 to T7
used in the illusion measurement experiment as makeup.
One of the standard stimulus images T1 through T7 washand, there were no statistically signiﬁcant diﬀerences
between the other groups. However, looking at the 95%
conﬁdence interval, which is an indicator of signiﬁcantdiﬀerence by not including 0, T6 for T4 (mean diﬀerence-1.52, 95%CI[-3.28, 0.24]) and T7 for T4 (mean diﬀerence-1.61, 95%CI[-3.36, 0.15]) slightly included 0. This indicates that the amount of illusion tends to decrease gradually with the eyeliner thickness after T4 (3.0 mm or more).

proposed in the questionnaire, and participants were required to answer the question “Do you feel the sense of incongruity with the following image?” by selecting from “0.
not at all,” “1. somewhat,” or “2. a lot.” This task was
performed for all standard stimulus images T1 to T7 . In
the ﬁeld of psychiatry, the three-option Brief Grief Questionnaire11)is often used for negative impressions such as

This suggests that above a certain thickness, the eyes do
not appear to be as large as expected even if the eyeliner
is thicker.
We then measured the magnitude of the experimental
eﬀect between each group using the eﬀect size dD (theeﬀect size that takes the correlation between the groups
into account). The eﬀect sizes of T1 for T2 and T2 for T3

anxiety and distress, so the three-point scale was also
used in this experiment to investigate negative impressions of the sense of incongruity.
5.2

Response results and discussion

As shown in Fig. 8, it can be found that the thicker theeyeliner from T1 to T7 , the more sense of incongruity theparticipants had. The number of participants who had
the strong sense of incongruity gradually increased fromT3 , reaching nearly 50% in T4 , and more than 50% in T5 ,T6 , and T7 , which suggests that they are not appropriate
as makeup. In T1 , some participants answered that they

were both slightly medium10)(dD = 0.35 and dD = 0.34,
respectively), which supports the monotonic increase of
the amount of illusion from T1 to T3 . In addition, theeﬀect size of T5 to T6 was nearly medium10)(dD = 0.43),
suggesting that the amount of illusion decreases from T5
to T6 . Moreover, the eﬀect sizes of T3 to T4 , T4 to T5 , andT6 to T7 were nearly small to extremely small10)(dD =0.07, dD = 0.13 and dD = 0.03, respectively), suggestingthat the change in the amount of illusion from T3 to T5
and from T6 to T7 are slight.

felt somewhat the sense of incongruity because they felt
the sense of incongruity that the eyeliner was not actuallyapplied but drawn with paint software and that the shapeof the corner of the eyes.

51

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

6.1

male

Discussion
Amount of illusion(perceived eye size)[%]6.

Eﬀect of eyeliner of diﬀerent thicknesses

From the experimental results on measuring theamount of illusion with various thicknesses of eyeliner,
it was shown that the amount of illusion increased monotonically from 1.2 mm to 2.4 mm, and slightly from 2.4
mm to 3.0 mm, and that the eyes were perceived to be
the largest around 3.0 mm. Furthermore, there was aslight decreasing trend from 3.0 mm to 3.6 mm, a decreasing trend from 3.6 mm to 4.2 mm, and a very small
decreasing trend from 4.2 mm to 4.8 mm. A questionnaire for subjective evaluation suggested that the eyelinerwith thickness of 3.0 mm or more gave the strong sense ofincongruity as makeup. Therefore, thickness of eyelinerthat can make the eyes appear the largest with the least
the sense of incongruity as makeup is about 2.4 mm.

108
106
104
102
100
98

1.2
(T₁)

1.8
(T₂)

2.4
(T₃)

3.0
(T₄)

3.6
(T₅)

4.2
(T₆)

4.8
(T₇)

Eyeliner thickness[mm]Fig. 9 Thickness of eyeliner and the amount of illusion
by gender (Markers indicate mean values of theamount of illusion and error bars indicate 95%
conﬁdence intervals)

Eyeliner thickness[mm]Do you feel the sense of incongruity?
2: a lot
1: somewhat
0: not at all

The reason why the illusion increases with thickness ofeyeliner is thought to be that eyeliner area is misidentiﬁed as the eye area. It is also thought that the thicker
eyeliner, the easier it is to perceive that the eye is surrounded by eyeliner and the Delboeuf illusion is more
likely to occur.
However, the eyes with excessively thick eyeliner makethe boundary between the face and the eyes clearer, thus
making the eyes appear to ﬂoat away from the face. Then,the eyes appear to be smaller in contrast to the size offace, which is called “size contrast illusion.” This illusion is also seen with eye shadow6). In addition, the senseof incongruity of stimulus images with excessively thickeyeliner may aﬀect the perception of the illusion and reduce the amount of the illusion. The measurement of
the amount of illusion with the strong sense of incon-

1.2(T₁)
1.8(T₂)2.4(T₃)3.0(T₄)
3.6(T₅)
4.2(T₆)
4.8(T₇)
0

25

50

75

100(%)

75

100(%)

Eyeliner thickness[mm](a) male
1.2(T₁)
1.8(T₂)2.4(T₃)3.0(T₄)
3.6(T₅)
4.2(T₆)
4.8(T₇)02550(b) female

Fig. 10 Subjective evaluation questionnaire results bygender

gruity makes it diﬃcult for the participants to perceive
the shape in a short time and to obtain the illusion eﬀect.
In addition, there is a possibility of large individual differences in the results because participants are hesitant
to judge.
Therefore, the eyeliner thickness illusion is not merely
an application of the Delboeuf illusion, but probably aninterrelationship of various illusions. Further experiments
are needed to clarify the relationship between the various
illusion eﬀects.
6.2

female110

results of the illusion measurement experiment by gender.
Figure 10 shows the results of the subjective evaluationquestionnaire by gender.
The results of the illusion measurement experiment for
male showed that the amount of illusion tended to increase when thickness of eyeliner was 1.2 mm to 3.0 mm,
to reach the maximum at 3.0 mm, and then to decrease
from 3.0 mm to 4.8 mm (Fig. 9). This trend is similar
to that of all the participants (Fig. 7). For male, thickness of eyeliner had a signiﬁcant eﬀect on the amount of
illusion (F(6, 16) = 6.00, p <.001). While for female,the variance was larger and thickness of eyeliner had no
signiﬁcant eﬀect on the amount of illusion (F(6, 15) =Diﬀerences between male and female

The results of the illusion measurement experiment andthe subjective evaluation questionnaire showed a large
diﬀerences between male and female. Figure 9 shows the

1.90, p = .09). This means the illusion eﬀect of eyeliner

52

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

illusion eﬀect only on male. In this paper, since we con-thickness in male was more marked than in female.
In the subjective evaluation questionnaire, about 50%

ducted on one type of eye sample, it is necessary to con-of the female participants already felt a little sense ofduct validation on multiple types of eye samples that takethe shape of the eyelid such as single-eyelid eyes into con-incongruity with the minimum thickness of eyeliner (1.2mm), suggesting that the sense of incongruity aﬀected
the measurement of the amount of illusion more than themale participants. Since female has more opportunities toconsciously look at eyeliner, they might be more likely tofeel the sense of incongruity about the shape and positionof the eyeliner.
Coren and Porac12) showed the correlation among individual diﬀerences in spatial ability and the size of illusion
for each type of illusion. They proved that the eﬀectof illusion is larger in people with higher spatial ability
for illusions related to the perception of area and direction, such as the Delboeuf illusion. Although male are
generally said to be better than female in spatial ability13), their study did not show clear and consistent sex

sideration. In the future, based on the veriﬁcation withmultiple types of eye samples, we will investigate thethickness model of eyeliner that makes the eyes appear
larger according to the shape of individual eyes.

Acknowledgments
We would like to thank Dr. Hiroto Inouye, assistantprofessor at Tokyo Denki University, for his comments
and suggestions in carrying out this paper. We would also
like to thank Mr. Shigeaki Suzuki and ASTRODESIGN,
Inc. for their kind support.

References
1) J. Y. Baudouin, G. Tiberghien: “Symmetry, Averageness, andFeature Size in the Facial Attractiveness of Women”, Acta Psychologica, Vol.117, No.3, pp.313–332 (2004).

diﬀerences in the correlations among the spatial abilities
measures and the visual-geometric illusion scores. However, the eyeliner illusion, which is not a mere geometric
illusion, is caused by the interaction of many complex
perceptual processes. It is also possible that male withhigher spatial ability is more likely to show this illusioneﬀect. We suspect that these diﬀerences between maleand female in the amount of illusion and the sense of incongruity on the eyes with makeup are one of the reasons
why there are gender diﬀerences in makeup preferences.

7.

2) D. Jones, C. L. Brace, W. Jankowiak, K. N. Laland, L. E.
Musselman, J. H. Langlois, L. A. Roggman, D. Pérusse, B.
Schweder, D. Symons: “Sexual Selection, Physical Attractiveness and Facial Neoteny: Cross-Cultural Evidence and Implications”, Current Anthropology, Vol.36, No.5, pp.723–748(1995).
3) D. I. Perrett, K. J. Lee, I. Penton-Voak, D. Rowland, S.
Yoshikawa, D. M. Burt, S. P. Henzi, D. L. Castles, S. Akamatsu: “Eﬀects of Sexual Dimorphism on Facial Attractiveness”, Nature, Vol.394, No.6696, pp.884–887 (1998).
4) K. Tagai, H. Ohtaka, H. Nittono: “Faces with Light Makeup
Are Better Recognized than Faces with Heavy Makeup”, Frontiers in Psychology, Vol.7, No.226 (2016).

Conclusion

5) R. Mulhern, G. Fieldman, T. Hussey, J. L. Lévêque, P. Pineau:“Do Cosmetics Enhance Female Caucasian Facial Attractiveness?”, International Journal of Cosmetic Science, Vol.25,
No.4, pp.199–205 (2003).

In this paper, we measured the apparent size of doubleeyelid eyes depending on thickness of eyeliner, i.e., theamount of illusion. The results of the measurement experiment showed that the illusion increased monotonicallyto a certain level as thickness of eyeliner increased, andthen decreased. This result indicates that thick eyeliner
causes misperception of the eye area and makes the eyes
appear larger, while too thick eyeliner has the opposite

6) S. Matsushita, K. Morikawa, H. Yamanami: “Measurementof Eye Size Illusion Caused by Eyeliner, Mascara, and Eye
Shadow”, Journal of Cosmetic Science, Vol.66, No.3, pp.161–
174 (2015).
7) A. L. Jones, A. Porcheron, R. Russell: “Makeup Changes the
Apparent Size of Facial Features”, Psychology of Aesthetics,
Creativity, and the Arts, Vol.12, No.3, pp.359–368 (2018).
8) K. Morikawa, S. Matsushita, A. Tomita, H. Yamanami: “A
Real-Life Illusion of Assimilation in the Human Face: Eye Size
Illusion Caused by Eyebrows and Eye Shadow”, Frontiers in
Human Neuroscience, Vol.9, No.139, pp.1–9 (2015).

eﬀect. We measured the amount of illusion by assumingthat the thickness of eyeliner is isotropically large in thehorizontal and vertical directions relative to the apparent
eye size. Verifying diﬀerent aspect ratios in the horizontal
and vertical directions is a future work. Moreover, further
experiments are needed to clarify whether the illusion effect of eyeliner thickness is aﬀected by the texture of theactual eyeliner, such as liquid or pencil type.
We also revealed that gender diﬀerences within the re-9) M. Kouchi, M. Mochimaru: “Anthropometric Database ofJapanese Head 2001”, National Institute of Advanced Industrial Science and Technology, H16PRO-212 (2008).
10) J. Cohen, Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Lawrence Erlbaum (1988).
11) M. Ito, S. Nakajima, D. Fujisawa, M. Miyashita, Y. Kim, M. K.
Shear, A. Ghesquiere, M. M. Wall: “Brief Measure for Screening Complicated Grief: Reliability and Discriminant Validity”,
PLoS One, Vol.7, No.2, e31209 (2012).

sults were also apparent. Thick eyeliner had a greater12) S. Coren, C. Porac:

53“Individual Diﬀerences in Visual-IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
Geometric Illusions: Predictions From Measures of Spatial
Cognitive Abilities”, Perception & Psychophysics, Vol.41,
No.3, pp.211–219 (1987).
13) M. C. Linn, A. C. Petersen: “Emergence and Characterizationof Sex Diﬀerences in Spatial Ability: A Meta-Analysis”, Child
Development, Vol.56, No.6, pp.1479–1498 (1985).

(Received May 31, 2021)(Revised April 30, 2022)

Rena OKURI
She received her B.E. degree from Tokyo
Denki University in 2020. She is currently
a Master’s student at Tokyo Denki University Graduate School. Her research interests
include facial image processing and cognitive
science.

Shuhei KODAMAHe received his Ph.D. (Engineering) from
Tokyo Denki University in 2019. He had beena visiting research student at Université deMontréal from March until June, 2018. He
had been an engineer at ASTRODESIGN,
Inc. from 2019 until 2021. He is currently an
assistant professor at Tokyo Denki University.
His research ﬁelds are NPR and HCI.

Tokiichiro TAKAHASHI (Fellow)
He received his B.E. degrees from Niigata
Uiversity in 1977, and Ph. D. from the University of Tokyo in 2005, respectively. He
had been working at Nippon Telegraph andTelephone Corporation and engaged in Pattern Recognition, Computer Graphics, andLearning Science since 1977. Since 2003, hehas been a professor at Tokyo Denki University. He has been a visiting researcher at
ASTRODESIGN, Inc. since 2017. His research interests include computer graphics,
image processing and visual computing.

54

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2021Part II --3D Distance Field-Based Apparel ModelingMasanori NAKAYAMA† (Member), Takami YAMAMOTO†† (Member),Issei FUJISHIRO† (Honorary Member/Fellow )† Keio University , †† Wayo Women’s University<Summary> This study aims to design an apparel item by converting a polygon model of the humanbody obtained through a 3D scan into a volume model. From the model, we can quickly generate a curved
surface that retains a constant distance from the surface of the body, thus being able to design an apparel
item that ensures accurate allowance by deﬁning the distance ﬁeld for the surface of the body. The curved
surface, which would be an ideal garment that can envelop the body, can be generated by tweaking thethreshold value for an isosurface from the surface of the body. Furthermore, by drawing design lines on
the curved surface, the pattern required for actual sewing can be designed in 3D space. The curved surface
trimmed by the design lines is converted into a polygon mesh. A dynamics simulation is applied to smooththe curvatures so that the desired ﬂat surface pattern can be obtained. As a result of actually sewing thevirtually designed pattern and trying it on, it was shown that we can generate clothes with the appropriate
allowance and proportions.
Keywords: 3D scanner, volume model, 3D distance ﬁeld, apparel items

1.

With this in mind, we propose a reverse pattern design
method of cutting a 3D curved surface and then expanding this into 2D space. In other words, this method ﬁrst
constructs a curved surface that expresses the ideal pieceof clothing, cuts it directly in 3D space, and then converts
it into a 2D pattern. This creates a seamless curved surface of the garment that would cover the human model
serving as the base; consequently, the 3D consistency at
the time of sewing can always be maintained, regardlessof body shape. Furthermore, various sizes and appropriate allowances can be freely set by enlarging or reducing
the size of the base human model.
In the current study, we have focused on the degree of
freedom and accuracy of the human body model. A human body model measured with a 3D scanner includes
body shape distortions and uneven noise. Therefore,
most tradtional methods copy individual body shape information by ﬁtting a simpliﬁed template model preparedwith an actual model. In other words, the accuracy ofthese other methods is inferior because the extremelycomplex human body shape is simpliﬁed into a few parameters, such as the bust or waist. This accuracy issue isparticularly signiﬁcant in ﬁelds where “form-ﬁtting clothing” is emphasized. We have adopted implicit function
modeling to generate a curved surface that envelops theIntroduction

In conventional apparel design, polygon models havecommonly been used to explicitly deﬁne fabric based ona methodology that mounts 2D patterns on a 3D human body1)–3). The apparel industry has a long historyof drafting patterns by hand. Over the course of digitalization, many kinds of pattern drafting software havebeen developed based on 2D industrial computer-aided
design (CAD). Indeed, the culture of 2D pattern drafting has continued for many years, and the mainstream
format in industrial design became the mainstream format in drafting. However, there is still no fundamental
change in the conventional methodology of enveloping a
3D human body with a 2D cloth. The 2D design mustbe forcibly extended and drawn, especially when developing garments of multiple sizes (i.e., grading), to ensure
an appropriate allowance. Therefore, major size changes
cannot be made, and any allowance must be based on
empirical evaluations. A typical work using this methodology is given by Umetani et al4), who developed a systemthat allows the user to draw a design and carry out theﬁtting in real time. However, the amount and positionof the darts and shape of the sleeves are not practical interms of clothing construction and have limited relevanceto the human body and clothing.

55

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 1 Pattern development ﬂow based on 3D distance ﬁeld

pattern is obtained while controlling distortion to a minimal degree by reducing the 3D curve of the fabric to 2D
stretch expansion.
Furthermore, to evaluate this approach, we created abasic pattern of the upper body for both women andmen. We also created a vest for men as another realapparel item. Our original intent was to provide a comprehensive framework for the design of apparel items, yetin the present study, we focus only on tight clothing because this type of clothing is the most sensitive to individual diﬀerences and, thus, is diﬃcult to design accurately.
Traditionally, apparel designers have only been able to
make 3D cuts using a standard torso. In our study, bystrictly maintaining the allowance, we establish a new design method that emphasizes the individual body shape.

human body. In other words, the human body model is
implicitly deﬁned as a distance ﬁeld based on the surfaceof the body, thereby facilitating the generation of a 3D
curved surface along the surface of the body. Speciﬁcally,
the human body model is deﬁned as a “volume model”,
which is done by using voxels.
The most prominent characteristic of the volume modelis that the curved surface retaining a constant distancefrom the surface of the body can be simply generated byestablishing the distance ﬁeld. In other words, designing a garment with an appropriately deﬁned allowance ispossible because the measured human body can be madethinner or fatter just by changing the threshold value forisosurfacing.
When the 3D scanned form is used without any changes
being made, an improved ﬁt to the body is achieved,
whereas the aesthetics of a well-proportioned garment decreases. To address this issue, a method to negate the left
and right disparity of the body is developed, here by utilizing the characteristics of the voxel model, which simpliﬁes the topological operations while maintaining personal
body shape characteristics5).
The ﬂat surface pattern needed for actual sewing is obtained by trimming the ideal garment surface deﬁned by
the isosurface. Of course, the trimmed surface has inherent 3D curvatures. However, by converting the trimmed
surface to a polygon mesh again, cloth simulation witha mass-spring model is applied. In other words, a 2D

2.

Volumetric Apparel Design

The pattern development ﬂow is shown in Fig. 1. First,a polygon model of the human body is obtained using a
3D scanner and is converted into a voxel model to deﬁne the distance ﬁeld. Next, the left–right diﬀerence
is removed by detecting the sagittal plane, and then, a
torso with a generous margin is generated by changing
the threshold for isosurfacing.
Generating a curved surface with a constant distanceto an arbitrary curved surface is very diﬃcult because
special care must be taken when it comes to the possible
changes in the topology. Computing the distance from56

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

the curved surface based on a polygon model would bevery expensive, and there would be a high possibility ofgenerating incorrect data structures because of the selfintersection of polygons. On the other hand, if we rely
on a volumetric model, it can always generate the correct
surface, no matter how much allowance is speciﬁed.
After doing this, design lines are drawn to create thebasic pattern for the upper body on the torso surface, andcurved pattern surfaces are generated as a polygon mesh.
Furthermore, a cut-out pattern of the curved surfaces isdeveloped into a ﬂat surface pattern. Finally, the fabric is cut using thick sheeting, sewn, and tried on while
conﬁrming any fabric distortion through a virtual ﬁtting;ﬁnally, the design quality is evaluated.

3.

Fig. 2 Gravity centers on the horizontal section position

Volume Model Generation

The core of the current research is a volume model ofthe human body. We can parametrically change its sizeby representing the surface of the body as an isosurface
from the distance ﬁeld. Conventional torso models thatcan be physically resized have been used. However, sucha torso model only divides the surface of the body into
multiple parts and deforms them mechanically. Our virtual parametric torso can deform while also maintaining
a smooth and continuous surface. The volume model isgenerated based on a real human body that has been 3D
scanned. Moreover, the process can be broadly divided
into fairing the model as a highly accurate human bodymodel and fairing it as a torso model that is useful forapparel design.
3.1

Fig. 3 Detecting the central axis of the trunkof the cross-sectional shape at various vertical positionsof the torso in Fig. 2 show that the range from the buttocks to the bottom of the bust reﬂects the vertical axis of
the trunk relatively well. Therefore, we extracted crosssections from two locations at the upper and lower endsof the limited trunk range instead of extracting the tilt
from the entire body; we then deﬁned the vertical axis
by connecting the plane centers of gravity, as shown in
Fig. 3.
The tilt of the human body model can be corrected if
the vertical axis of the human body is obtained. However,
the tilt in the rotational direction around the vertical axis
would still remain. This is not a problem when printing
with a 3D printer or conducting dressing simulations, but
extraction with complete symmetry is essential for generating a torso with an aesthetic sense we pursue. Therefore, we used the symmetry of the trunk, here centeringon the vertical axis obtained from the trunk, to determine the angle of rotation needed to correct the frontal
direction. Speciﬁcally, the horizontal cross-section of thecentral part of the trunk is extracted, after which the left
and right sides are reversed, here assuming a temporary
plane of symmetry. The plane of symmetry can be said tobe optimal if the overlapping area is the largest when thereversed cross-section and original cross-section overlap
with each other, as shown in Fig. 4.

Body model generation

Although a 3D body scanner digitizes a person’s body,the scanned data are generally output as a polygon
model. Therefore, the polygon data must be converted
into a volume model.
A human body scan is conducted, with the subject
somewhat spreading both legs and hands so as not to hide
their armpits and the space between their legs. However,human posture is always tilted, and there are also cases
where an accurate horizontal reference may not be guaranteed, depending on the measurement method, such as
when using photogrammetry. Therefore, the tilt is removed from the scanned data. The original vertical direction of the human body is determined only from theshape data. The trunk establishes the vertical directionof the human body, and this is not heavily aﬀected bytheir tilt. The results of calculating the center of gravity

57

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Smoothing

Fig. 4 Searching for symmetry planes on the humanbody(b) Perimeter preserving

Fig. 5 Denoising the body model

The smoothing process controls ﬂuctuations in the distance ﬁeld by averaging the voxel values between adjacent voxels. The surface of the body, which is deﬁned as
the isosurface, also changes when the voxel value changes.
Areas with a more signiﬁcant local curvature (e.g., minuteunevenness) are removed more often, resulting in a human body model with a smooth surface that still retains
its basic shape.
Because the smoothing process decreases the spatial
frequency of the voxel model, very complicated models
topologically come closer to a sphere, and protrusions
such as ﬁngers and arms are shaved oﬀ. As a result, thebody model deforms and deviates from the actually measured shape. However, by adjusting the threshold value
for isosurfacing, a body model that retains the surface
quality can be generated while maintaining the representative parameter values, such as the boundary length, asshown in Fig. 5(b).

Existing methods that correct the tilt in 3D models
generally take the approach of aiming for the model’s
overall shape (i.e., global optimization). However, a globally optimal solution often cannot be obtained when aprerequisite is the deformation of limbs. Therefore, we
conducted local optimization on the trunk of the humanbody by focusing on the production of an ideal torso
model rather than on an entire mannequin. This approach allows for the stable extraction of the torso without relying on the position of the subject’s limbs.
The polygon model can be voxelized after correcting
the vertical axis tilt and left–right rotation misalignment.
In our implementation, the voxel values inside the polygon model are set to 1 and those outside to 0. In addition,
for the points on the boundary surface, the intermediate values are used. In other words, these points near
the body deﬁne the distance ﬁeld from the surface of the
body. The initial polygon model is approximated by anisosurface extracted with a threshold value of 0.5.
The distance from the surface of the body should befound for all voxels when deﬁning a distance ﬁeld in a
general sense. However, ﬁnding the shortest distances
for positions far away from the surface of the body is ahighly computationally expensive process. Therefore, we
decided to form a distance ﬁeld only in the near ﬁeld close
to the surface of the body, here based on the premise ofdesigning an allowance with a reasonable range. The surface can be expanded or contracted within a reasonable
range for a human body by changing the threshold value.
However, when voxelization is carried out without adjustment, the actual polygon data include scanning errors
and noise because of the unevenness of the model’s underwear, which will deteriorate the usability as a torso.
To address this issue, a smoothing ﬁlter is used for the
3D voxel space in the same way as the denoising ﬁlter isused in image processing, as shown in Fig. 5(a).

3.2

Torso model generation

Because the human body has distortions in its postureand symmetry, even if the body model closely reproduces
the actual body shape, it would lack the aesthetics of a
torso for the garment design. To deal with this problem,the sagittal plane of the body is automatically detectedfrom the body model, and the postural sway and symmetry diﬀerences of the body shape are removed.
Figure 6 shows the symmetrization process. The left–right symmetry is obtained based on the left–right symmetrical plane of the trunk, which is detected when creating the body model. Adding the symmetrical voxel values
across the sagittal plane and updating the voxels on theleft and right sides so that they have the same values result in the body model being inverted and superimposedon itself. The tilt of the body model is removed before
voxelization; consequently, the shared volume is maximized. The completely overlapping sections have a voxel

58

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Cross-sectional lines)HDWXUH+HLJKW%XVW/LQH %/7KUHVKROG         

8QGHU%XVW/LQH 8%/     :DLVW/LQH :/     +LS/LQH +3    (b) Cross-sectional perimeters (mm)Fig. 7 Torso with a controllable threshold
Fig. 6 Symmetrization process

value that is almost double the original, but the nonoverlapping and nonsymmetrical parts will have less values
than this. Thus, the entire body shape could be madesymmetrical while retaining a complete intersection (i.e.,
the parts that are originally symmetrical) by smoothing
only the areas with low voxel values while maintaining
the intersections with high voxel values. Areas outside
the body where the voxel values are zero are also excluded
from smoothing to prevent the base of the voxel values
from widening because of smoothing. In other words,
smoothing is conducted only in areas on the human bodywhere the shape is ﬁxed and in the regions between thespaces outside the body. The voxel values are then relaxed and symmetry is established while maintaining thebasic shape as much as possible.
However, as with surface smoothing in Fig. 5(a), there
will be distortions in the bust shape and other features.
Therefore, by adjusting the threshold value, the isosurface is symmetrized while preserving the chest circumference length, as shown in Fig. 5(b).
Furthermore, to create a foundation suitable for designing a basic pattern for the upper body, the thresholdvalue is intentionally shifted, and a slightly inﬂated torso
model is generated. By deﬁnition of the distance ﬁeld,
at a position at any height, the closer to 1 the thresholdbecomes, the smaller the perimeter becomes. In otherwords, allowance is included in the shape in advance sothat the pattern can be directly cut on the torso. Figure 7 locates four representative cross-sectional feature(a) Design lines drawn on the
body surface(b) Trimmed isosurfaces

Fig. 8 Dividing by design lines

lines in (a) and lists their perimeter allowances according
to diﬀerent thresholds in (b).

4.

3D Patterning

To obtain the pattern needed to sew a garment, designlines (cut open lines) are drawn on the surface of the torso
model, as shown in Fig. 8(a). A princess line (a silhouettewith a narrow waist and ﬂared hem because of the vertical switching lines) is inserted in the center of the front
and back, and other design lines are inserted between theprincess line and side. Once the neck and sleeve lines
are inserted, the upper body pattern enveloping the upper body is complete. Fig. 8(b) shows the corresponding
trimmed isosurface.
The design lines are empirically arranged to decrease
the distortions when each of the areas divided by thelines is eventually converted into a ﬂat surface pattern6) .

59

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Trimmed isosurface components(b) Flattened patterns

Fig. 9 Pattern generating process(a) Before

Fig. 10 Virtual ﬁtting of the basic pattern

Convex areas such as the bust and shoulder blade areconsidered cone shaped, and the dividing boundary is arranged on the top.
The curved surfaces of the patterns placed on the torso
are converted into a cuttable ﬂat surface pattern, as
shown in Fig. 9. The polygon model that explicitly deﬁnes the surface is suitable for deforming the surface, so
the pattern drawn on the isosurface is converted into a
polygon mesh. Speciﬁcally, after the entire isosurface isconverted into a polygon mesh by the Marching Cubes
method7), it is trimmed according to the pattern region.
The polygons across the pattern boundary are divided
along the design lines, and a polygon mesh, which is completely separated for each pattern, is obtained in Fig. 9(a).
Although the original polygon mesh has 3D curvatures, apattern for a ﬂat surface can be developed by applying asimple cloth simulation in Fig. 9(b). Because deformation
occurs when the curved surfaces are pressed down into a
ﬂat surface to remove the curvatures, stretch distortion of
the fabric is unavoidable. However, the stretch distortionis minimized by imposing a mass-spring model with ridge
lines and the polygon’s vertices. The visible outline of theobtained polygon mesh becomes the basic pattern for theupper body. The geometric expandability is not strictly
guaranteed with the mass-spring simulation. However, it
is possible to minimize distortions through the design ofthe pattern line so that some expandability can be ensured.

5.

(b) After&RORU

6WUHWFKUDWLR

1RQH



:KLWH



<HOORZ

5HG

Fig. 11 Visualization of stretch vectors

essary to make the density of polygon meshes uniform byremeshing. After obtaining a ﬂat surface pattern, whichbecomes a visible outline of the polygon mesh, the oldpolygon mesh is temporarily disposed, and then, the newpolygon mesh is repacked. The vertices of the new polygon mesh are ﬁlled in using the Poisson-disc distribution8), and a uniform polygon mesh that connects the vertices
with a minimum distance is generated by Delaunay triangulation9).
A cloth simulator was implemented based on the DressMakingCAD10)freeware by Tomoyuki Ito, who is a research collaborator of this work11). If the pattern is designed on a ﬂat plane, fabric distortion in the wearing simulation is large, and the pattern must often be corrected.
However, because this pattern was originally designed in
3D space, fabric distortion in the wearing simulation isminimal, so no correction is needed. In Fig. 11, the elongation of the fabric is calculated along the polygon meshedges and visualized as three-dimensional bars; only bars
with elongation greater than 0.05 are shown, and the colors follow the legend. Incidentally, apart from the elastic distortion of the pattern, the distortion of the outline aﬀects the garment’s silhouette, so the cross-sectional
shapes of the collar, sleeves, and waist are corrected, as
necessary.

Virtual Fitting

The polygon mesh of the ﬂat surface pattern is always
distorted when converting it into a ﬂat surface, so even
if the ﬂat surface pattern is resewn into a 3D shape, itwill not match the original torso shape. Therefore, itis necessary to perform virtual ﬁtting to quantitatively
evaluate the amount of silhouette distortion. Figure 10
shows the patterns before and after the virtual ﬁtting.
To reproduce the realistic behavior of fabrics, it is nec-

60

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
Table 1 Working times (minutes)

3URFHVV

front body, right front side, right back body, and rightback side using thick sheeting and placing the target lines
along the body. In doing so, we did not reserve any allowance. Figure 12(d) shows the results of F1 trying on
the garment. The ﬁtting evaluation questionnaire showed
the following results: regarding the silhouette, an evaluation of “good” was obtained for the front, sides, and back;regarding the ﬁt and wearing comfort, an evaluation of“good/somewhat good” was obtained; and the free responses included the following: “The silhouette aroundthe chest is very pretty. The garment ﬁt very well along
the unevenness from the centers of the back to the sides”.
“The garment ﬁts perfectly on the body, and there is nowasted space”.
The free responses of F1 comparing the our new body
prototype in Fig. 12(c) and the traditional torso-based
body prototype in Fig. 12(d) included the following: “Thenew body prototype seemed to ﬁt thinner bodies better.
The garment ﬁt the chest curvature very well, and thesilhouette was pretty”. “There is little diﬀerence in thefrontal silhouettes of the new and traditional body prototypes, but the side silhouette of the new body prototype
is more beautiful”.
Comparing the body prototypes in Fig. 12(c) and
Fig. 12(d), ours better ﬁts the roundness of the chest in
the front. For the back side, ours better ﬁts the body,even though the body shape is prominent around thescapula. We were able to simulate this trend in virtualﬁtting as well in Fig. 13. Furthermore, in the comparison of the sagittal plane area, our prototype has a smallerarea of allowance, as shown in Fig. 14. In other words,
we quantitatively demonstrated that we could use ourmethod to make aesthetically pleasing clothes with a minimal allowance when compared with the conventional 2D
patterning.
Therefore, we created body prototypes for two more female research participants (F2, F3) to try on, as shown in
Fig. 15. As with research participant F1, we carried out
a four-step questionnaire. Both of them answered thatthe silhouette was “good” for all sides: front, sides, andback. The ﬁt was rated as “good” by both, but the wearing comfort was divided between “good” and “somewhat
good”. The free description responses included the following: “The ﬁt was strong due to the small allowance,
but it ﬁt the human body in an upright position verywell”.
Figure 16(a) shows the men’s prototype pattern (leftside), which was slightly modiﬁed by virtual ﬁtting and

2SHUDWLRQWLPH *HQHUDWLRQWLPH

3RVWXUHFRUUHFWLRQ

6PRRWKLQJ

6\PPHWUL]DWLRQ'3DWWHUQLQJ')ODWWHQLQJ

9LUWXDOrWWLQJ





6.

Results

Using this system, we created prototypes of apparelitems for seven women (F1,...,F7) and one man (M1).
The eight individuals were selected for their gender differences and body shape and posture characteristics, even
if the diﬀerences in BMI were small.
All experiments were conducted on a computer with anIntel Core i7-7700 3.60 GHz CPU with 32 GB of RAM
and a Windows 10 x64 operating system. The codes were
implemented using Java. Table 1 shows the operationand generation times. The generation time is longer than
expected, but we believe that it can be sped up by making
the program multithreaded.
A basic pattern for an upper body of female research
participant F1, which was slightly adjusted using the virtual ﬁtting apparel CAD, is shown in Fig. 12(a). Generally, a right pattern is prepared for women while a left
pattern is prepared for men. Cutting and sewing were
performed using thick sheeting. Sewing was performed
with a sewing machine, and an open fastener was attachedbecause the centers of the back formed a curved line.
The results of F1 wearing the body prototype pattern are shown in Fig. 12(c). An evaluation questionnairewith four response items regarding wearing the clothing
item was carried out: “good”, “somewhat good”, “somewhat poor”, and “poor”. The silhouette was evaluatedas “good” for all sides: front, sides, and back. Regarding
the ﬁt and wearing comfort, the evaluation was “good”.
In the free description section, responses such as “Thesilhouette of the chest curvature from the side is pretty.
The ﬁt of the sleeve and shoulder lines is appropriate”
were given. In addition, the comment “There is a sufﬁcient allowance, and no discomfort when moving” wasreceived.
Figure 12(b) shows the body prototype pattern createdby draping with the torso we developed for F1. A target line was placed on the torso using a 2 mm IC tape.
The draping was conducted by roughly cutting the right61

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Basic pattern (Ours)(b) Basic pattern (Draping)(c) Wearing the basic pattern (Ours)(d) Wearing the basic pattern (Draping)

Fig. 12 Evaluations by women’s try-on%DVLFSDWWHUQ &URVVVHFWLRQ*DS%XQNDVW\OH



2XUV



2ULJLQDOERG\



(a) Cross-sectional areas (mm2 )(a) Ours(b) Draping

Fig. 13 Simulation of the ﬁtting

apparel CAD. Figure 16(c) shows the results of participant M1’s (ﬁrst author of this paper) ﬁtting. The ﬁtting
evaluation questionnaire showed the following results: regarding the silhouette, an evaluation of “good” was obtained for the front, sides, and back; regarding the ﬁt
and wearing comfort, an evaluation of “good” was obtained. The free responses included the following: “Thearea around the waist was well done”. “The overall silhouette was clean”.
A men’s vest12)pattern is shown in Fig. 16(b), where
drawing diagrams of the vest based on the men’s basic pattern are shown on the left, and the development
diagram shown on the right is a completed vest pattern. The results of M1 wearing the vest are shown(b) Cross-sectional lines

Fig. 14 Evaluation of the ﬁttingin Fig. 16(d). In the evaluation questionnaire regarding
wearing, the silhouette was evaluated as “good/somewhat
good” for all sides: the front, sides, and back. Regarding the ﬁt and wearing comfort, the evaluation was also
“good/somewhat good”. In the free description section,
the responses were as follows: “Overall, the allowance isjust right; the height seems slightly longer”, “It ﬁts mysloped shoulders”, and “There is no unnecessary tightfeeling”.

62

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Basic pattern for F2(b) Basic pattern for F3(c) Wearing the basic pattern for F2(d) Wearing the basic pattern for F3

Fig. 15 Evaluations by women’s try-on (Ours)(a) Basic pattern(b) Vest pattern(c) Wearing the basic pattern(d) Wearing the vest

Fig. 16 Evaluation by men’s try-on (Ours)

In addition, we made prototypes for more research
participants (F4,...,F7) and performed virtual ﬁttings.
Figure 17 shows ﬁve prototypes with distinctive body
shapes. The rows show the simulation results for thesame research participant, while the columns show theThe men’s vest shown in Figure 16(d) had just the rightamount of allowance and resulted in a beautiful silhouette. The vest ﬁts so well without trying it on becauseof the appropriateness of the original prototype data forthe vest.

63

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
Ours

Bunka-style

F4: BMI=17.0

F1: BMI=19.1

F5: BMI=19.2

F6: BMI=22.7

F7: BMI=27.2
Fig. 17 Virtual ﬁtting of various body types

diﬀerent patterning methods. The four images for one
garment show the wrinkles and the degree of stretchingin the cloth as seen from the front and back of the human body. Many wires drawn on the surface indicate theexpansion and contraction of the fabric in terms of colorand direction. The colors used here follow the deﬁnitionin Fig. 11.
Note that BMI is used to show the degree of obesity.
The simulation shows that the clothes designed with our
method ﬁt the body better and wrinkled less than thosewith the traditional Bunka-style13)2D drawing. In addition, the conventional draping was less likely to cause
stretching of the fabric because of the large overall allowance, but in the case of characteristic body shapes, it
tended to stretch a great deal locally.

7.

Conclusion

In the current paper, a high-quality volumetric bodymodel for which noise was eliminated was created. Then,a highly aesthetic volumetric torso model with little symmetrical diﬀerences of the body and with a strictly designated allowance by thresholding the 3D distance ﬁeld was
generated. After designing a pattern for an ideal 3D formon the adaptive torso model, a ﬂat surface pattern withminimal distortion was expanded. A basic pattern for theupper body and a men’s vest were prototyped, and tryon evaluations were carried out to make sure there wasenough room for each of the body types. In future research, we would like to improve the volume of the torso
further and produce various apparel items.

64

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Acknowledgments

Masanori NAKAYAMA (Member)
He is currently working as the chief priest of
temple “Ankokuin” in Chiba Prefecture andas the CEO/artist of the company “LUXOPHIA”, which produces and promotes computational art. He received his B.E. and
M.E. at Keio University in 2002 and 2004.
His current research interests include photoreal rendering, free-form surface modeling,data processing based on spherical geometry, panoramic display, stereogram, apparel
CAD education, and 3D human body measurement. He is a member of IPSJ, IIEEJ,
and SAS.

We would like to sincerely thank Tomoyuki Ito for his
collaboration in programming and the staﬀ at BMD System Corporation for their cooperation in the 3D bodymeasurements. This work has been partially supported
by JSPS KAKENHI Grant Number JP19K02337.

References
1) DressingSim LSX (LookStailorX),
http://www.digitalfashion.jp/new/product/dressingsim lsx/
(2005).
2) Z. Stjepanovič, P. Tanja, A. Rudolf, J. Simona: “3D Virtual
Prototyping of Clothing Products”, Innovations in Clothing
Technology and Measurement Techniques, Technical University of Lodz, pp.28–41 (2012).

Takami YAMAMOTO (Member)
She is an Associate Professor in the Department of Fashion and Art at Wayo Women’s
University, Chiba, Japan. She completed adoctoral program at the graduate school of
Ochanomizu University and received a Ph. D.
in Integrated Science in 2006. Her current research interests include clothing construction
education, apparel CAD education, and 3D
human body measurement. She is a memberof IPSJ, IIEEJ, JES, JSET, SAS, The Japan
Society of Home Economics, and The Japan
Research Association for Textile End-Uses.

3) CREACOMPO2 Torso,
https://www.toray-acs.co.jp/products/creacompo2/
patternmagic2-3d/ (2014).
4) N. Umetani, D. Kaufman, T. Igarashi, E. Grinspun: “Sensitive
Couture for Interactive Garment Modeling and Editing”, ACM
Trans. on Graphics, Vol.30, No.4, pp.1–12 (2011).
5) T. Yamamoto, M. Nakayama, I. Fujishiro: “Development of
Individual Torso Based on 3D Distance ﬁeld”, Abstracts of the72nd Annual Meeting of Home Economics of Japan, p.7 (2020)(in Japanese).
6) T. Masuda, H. Imaoka: “Estimation of Body Form Factor
for Developing Trunk Surface (Part 2), Comparison of TightFitting Pattern of Front Bodice with Development Pattent ofFront Trunk”, Journal of Home Economics of Japan, Vol.47,
No.4, pp.343–355 (1996).

Issei FUJISHIRO(Honorary Member/Fellow )He is currently a professor of information andcomputer science at Keio University, Yokohama, Japan. He received his Doctor of Science in information sciences from the University of Tokyo in 1988. His research interests
include modeling paradigms and shape representations, applied visualization design andlifecycle management, and smart ambient media with multi-modal displays. He is a member of Science Council of Japan, a fellow of
JFES, an honorary member of IIEEJ, a fellow of IPSJ, and a senior member of IEEEand ACM. He is a 2021 inductee to IEEE Visualization Academy.

7) W. Lorensen, H. Cline: “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, Proc. of ACM SIGGRAPH Computer Graphics, Vol.21, No.4, pp.163–169 (1987).
8) A. Ahmed, H. Perrier, D. Coeurjolly, V. Ostromoukhov,
J. Guo, D. Yan, H. Huang, O. Deussen: “Low-discrepancy
Blue Noise Sampling”, ACM Trans. on Graphics (Proc. of SIGGRAPH Asia), Vol.35, No.6, Article No.247 (2016).
9) J. Guo, D. Yan, G. Bao, W. Dong, X. Zhang, P. Wonka: “Eﬃcient Triangulation of Poisson-Disk Sampled Point Sets”, The
Visual Computer, Vol.30, No.6–8, pp.773–785 (2014).
10) DressMakingCAD,
https://xn--6xw240d.net (2015) (in Japanese).
11) K. Choi, H. Ko: “Stable but Responsive Cloth”, ACM Trans.
on Graphics, Vol.21, No.3, pp.604–611 (2002).
12) T. Kaneko: MEN S Clothes for All Seasons, NIHON VOGUE
Corp. (2015) (in Japanese).
13) Bunka Fashion College (Ed.): Garment Design Textbook (1)Fundamentals of Garment Design, Bunka Publishing Bureau,(2009) (in Japanese).

(Received May 31, 2021)(Revised December 14, 2021)

65

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2021Part II --Invertible Fingerprint Replacement for Image Privacy Protection
Kazuya NAKAMURA† , Shugo YAMAGUCHI† , Hideki TSUNASHIMA† , Shigeo MORISHIMA†† (Member)† Waseda University , †† Waseda Research Institute for Science and Engineering<Summary> The demand for privacy protection has been increasing with the widespread use of devices
that can easily take high-resolution images, such as digital cameras and smartphones. In particular, ﬁngerprint information is one of the targets for privacy protection, but there is no research which speciﬁcallydeals with ﬁngerprint information removal to our best knowledge. In this paper, we propose a method forreversibly replacing ﬁngerprints in an image with another fake ﬁngerprint. This method makes it possible toautomatically remove the original ﬁngerprint information in the input image and generate a natural image.
Moreover, the input image s ﬁngerprint information can be easily restored from the output image only by
speciﬁc persons who know the key used in the image generation process. In conclusion, we conﬁrmed that theﬁngerprint information removal and restoration methods are eﬀective by using the ﬁngerprint authentication
model.
Keywords: image processing, privacy protection, ﬁngerprint, texture synthesis

1.

Networks (GAN)5). This method mainly deals with faces
and cars as privacy content. However, there is no method

Introductionthat replaces ﬁngerprint information in color images andgenerates natural images for privacy protection.

As digital cameras capable of taking high-resolution images become widespread, there is an increasing risk thatIn this paper, we propose an automatic removal of ﬁngerprint information in images and a natural fake ﬁn-

ﬁngerprint information is included in images. When you
take images that include your hands, your ﬁngerprint in-

gerprint image generation method. It is possible to remove the ﬁngerprint information in an image by replacing
the original ﬁngerprint with a generated fake ﬁngerprint.
Our method can generate a natural ﬁngerprint image thatformation can be unintentionally included in the images.
Ogane et al.1)pointed that ﬁngerprint information can beobtained from a ﬁnger image taken with a digital singlelens reﬂex camera from a distance of 3 m. There is a riskthat your ﬁngerprint can be maliciously used when you
share images containing ﬁngerprints on SNS. To avoid
such a risk, there is a demand for a method that can

has ﬁngerprints’ ﬁne structure and retains the color appearance by considering the global color distribution of
the ﬁngertips in the original image. Furthermore, ourmethod can easily restore the ﬁngerprint information ofeasily remove ﬁngerprints in images. Orekondy et al.2)
proposed the method to redact privacy content including

the input image from the output image by using the key
information used in the ﬁngerprint replacement process.
Only persons who know the key information can restore it
by referring the least signiﬁcant bit (LSB) substitution,
which is the basic digital watermarking method. Our
method can be applied to criminal investigations if per-ﬁngerprints, but generated images are unnatural because
privacy regions are blacked out. Sharing unnatural images is not desirable for SNS users, so the required naturalness is that the appearance of images is not changed asmuch as possible. Applying a Gaussian ﬁltering method
is not secure because the original ﬁngerprint can be re-sons with speciﬁc authority manage this key. The police
can use the ﬁngerprint information in the image on SNS

stored by a deblurring method such as deconvolution3). Therefore, a method for completely removing personal

for the investigation when they access the key information. In this paper,

information and generating natural images is needed. Yu
et al.4)proposed a method which detects the privacy content in images and replaces such content with privacyprotected content generated by Generative Adversarial(i) We propose a novel framework to remove originalﬁngerprint information in images for privacy protection.

66

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）                               
 



 

              

Fig. 1 The overview of our ﬁngerprint replacement procedureand it takes a high cost to make such a dataset. Therefore, it is diﬃcult to directly generate color ﬁngerprint
images by GAN5)or VAE12).

(ii) We propose a novel algorithm that replaces ﬁngerprint information in color images with that of fakegenerated ﬁngerprints and synthesizes natural ﬁngerprint images retaining the color appearance.

We propose a unique method that is focused on removing ﬁngerprint information in color images for privacy
protection. Our method replaces the ﬁngerprints’ information in the images with that of fake generated ﬁngerprint images, and this enables us to remove the originalﬁngerprints’ information, retaining the color appearance.

(iii) We propose a novel method that allows only persons
who have the key information to easily restore original ﬁngerprint information from synthesized ﬁngerprint images using a binary random number table.

2.

Related Work3.

(a) Fingerprint Removal
Basic methods for removing general privacy content areblurring, pixelating, and masking6). Ribaric et al.7)stated

MethodWe input an image including ﬁngertips and generate anatural image by replacing the original ﬁngerprint with agenerated fake ﬁngerprint. Figure 1 shows an overview
of our ﬁngerprint replacement procedure. Firstly, it is determined whether the ﬁngerprint exists in the input image
for each ﬁnger. Next, for each ﬁnger of the removal target, a natural image is generated in which the ﬁngerprint
information is replaced with generated fake ﬁngerprint
information. Then, we saved the information necessarythat similar methods could be applied to ﬁngerprint deidentiﬁcation. Recently, inpainting or replacement withprivacy-protected content is adopted for removing general privacy objects4)8). However, a recent study about
removing privacy content, including ﬁngerprints in images, adopts the masking method2), so output images areunnatural. We are reluctant to use the inpainting method
for removing ﬁngerprints because it may be diﬃcult torepresent natural ﬁngerprint images due to the ﬁngerprints’ ﬁne structure. Therefore, generating ﬁngerprint
images is necessary for a replacement method to prepare
natural ﬁngerprints.
(b) Fingerprint Generation
Study about ﬁngerprint generation9), which aims

for restoring the original ﬁngerprint information.
3.1

Fingerprint detection

We localize the detection area by the bounding boxof the hands in the input image, and detect the featurepoints around the bounding box by OpenPose13). Our
experiment is based on the method by Ortega14). Next,

mainly at ﬁngerprint dataset augmentation, deals withgenerating monochrome ﬁngerprint images. We hesitate to directly replace ﬁngerprints in images with these

we determine whether the ﬁngerprint exists in the image.
Then, we judge the front or back of the hand by calculating the cross product of the vector from the palm tothe index ﬁnger V1 and the vector from the palm to thelittle ﬁnger V2 . The front or back of the hand can be

monochrome images for privacy protection because theoutput images will be unnatural. There are several methods using deep learning frameworks10)11)such as GAN5)

identiﬁed by the positive or negative of the z componentof this cross product. Next, we determine whether theﬁngerprint exists in the image if the input image includes
the front of the hand. For each ﬁnger, we calculate theinner product of the vector from the palm to the base of
the ﬁnger v1 and the vector from the ﬁrst joint of the ﬁn-

12)or Variational Auto-Encoder (VAE) . We can generatecolor ﬁngerprint images and remove the original ﬁngerprints in images if we use a color ﬁngertip image datasetwith the above deep learning framework. However, to ourbest knowledge, there is no color ﬁngertip image dataset

67

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022） 









 

     
 



 
 



 





Fig. 2 The process of making a mask image



 

 

 



 

 



 

           
 

Fig. 4 The ﬁngerprint replacement overview

3.3

The ﬁngerprint replacement process is applied to eachtarget ﬁnger for ﬁngerprint removal. For generating anatural image, we replace only the input ﬁngerprint information with the generated fake ﬁngerprint information, preserving the color appearance. Figure 4 shows
an overview of the ﬁngerprint replacement process. In
this process, we use the ﬁngertip image, its grayscale image, and the generated fake ﬁngerprint whose luminanceis inverted from the Finger-GAN10)output. Before theﬁngerprint replacement process, we divide these three images into small grids so that we can execute the replacement process locally and generate a natural image thatretains the color appearance. Moreover, we smooth thegenerated fake ﬁngerprint image with a Gaussian ﬁlter
because the correspondence in the following step (a) of
Figure 4 will be more eﬀective. In addition, the targetof the ﬁngerprint replacement process is only the area inthe ﬁngertip.
The ﬁngerprint replacement process is as follows.
Firstly, the pixels correspond according to the order ofthe luminance of the generated fake ﬁngerprint image andthe grayscale image inside the ﬁngertip area of the corresponding grid area (Figure 4 (a)). The pixel p having
the nth highest luminance in a speciﬁc grid area in thegenerated fake ﬁngerprint image corresponds to the pixel
q having the nth highest luminance in the corresponding grid area in the grayscale image. Secondly, based on
the correspondence between the pixels, the pixel p of thereplaced image is replaced by the pixel q of the original

Fig. 3 The result of determining ﬁngertip areas when
the keypoints cannot be detected accurately

ger to the ﬁngertip v1 . If this inner product is positive,
the ﬁnger is detected as a target for ﬁngerprint removal.
3.2

Fingerprint replacement process

Preparation for ﬁngerprint replacement

The generated fake ﬁngerprint used for the replacement
should be diﬀerent from the input ﬁngerprint. To preparesuch a ﬁngerprint image, we use Finger-GAN10), which
can generate various grayscale ﬁngerprint images. We
use the binarized image of the generated fake ﬁngerprint
image in the ﬁngerprint replacement process. We prepare
several generated fake ﬁngerprints and select one whoseﬁngerprint information is diﬀerent from that in the input
image because the selected fake ﬁngerprint should not be
identiﬁed as the same ﬁngerprint in the input image. If
a generated ﬁngerprint matches another ﬁngerprint thatexists in the real world, it does not matter as long as thegenerated ﬁngerprint does not match the original ﬁngerprint in the input image.
In addition, determining ﬁngertip areas is required in
the ﬁngerprint replacement. Figure 2 shows the processof making a mask image for determining ﬁngertip areas.
At ﬁrst, we rotate the input image so that the direction of
the ﬁnger is vertical, and crop the image. Next, we makea mask image by GrabCut15)and trim the mask image,referencing the white area. Finally, we rerotate the mask
image and we can determine the ﬁngertip area in the input image. That is, we use a coarse-to-ﬁne approach thatimage (Figure 4 (b)).
These two steps are performed for all pixels in the ﬁn-

ﬁngertip images are roughly cropped and ﬁngertip areas
are accurately determined. This coarse-to-ﬁne method
enables us to determine the ﬁngertip areas accurately, regardless of the accuracy of OpenPose13). Figure 3 shows

gertip area estimated by GrabCut, not for pixels in the
background area. This process means that the white pixels of the generated fake ﬁngerprint representing the ridgeof the ﬁngerprint correspond to the pixels having a highluminance in the grayscale image. On the other hand, thethat our method can determine ﬁngertip areas even if the
keypoints of hands cannot be detected accurately.

68

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）     




    

    
           Fig. 5 Fingertip mask image of the input image

  

black pixels of the generated fake ﬁngerprint representing
the groove of the ﬁngerprint correspond to the pixels having a low luminance in the grayscale image. After that,the ﬁngerprint information is replaced by exchanging the
pixels based on the correspondence. Furthermore, by locally performing this pixel exchanging process, it is possible to generate a natural image in which the global color
distribution of the ﬁngertips is maintained. Finally, bysynthesizing the replaced ﬁngertip image with the input
image, it is possible to generate an image that the ﬁngerprint information of the input image is replaced withthat of the generated fake ﬁngerprint image.
3.4

      



      


  
    

Fig. 6 Comparison of the simple LSB substitution-like
method and our saving methodminance in the output data represents the saved binary
mask of pixel shuﬄe area.
To solve this problem, we propose a novel LSB
substitution-like method using a random binary number
table. Instead of saving the binary representation information to be saved as it is, when the information to be
saved is 1, the value of the random binary number tableis saved as it is. On the other hand, when the information
to be saved is 0, the value obtained by inverting 0 and 1 of
the random binary number table is saved (Figure 6 (b)).
The random number table allows us to restore the saved
information in the proposed method. The saved information is proved to be 1 when the parity of the luminance inthe output data and the parity of binary numbers in the
random number table are matched. Otherwise, it meansthat the saved information is 0.
We consider that this proposed method is far more secure than the LSB substitution method because the saved
information cannot be restored without the random number table. Furthermore, when a random number table isgenerated using a pseudo-random seed, only a speciﬁc

Saving information for restoring the original ﬁngerprint

Since the output image is generated by shuﬄing the
input image pixels, the ﬁngerprint information of the input image can be restored by the reversed pixel shuﬄing
operation on the output image. The reversed pixel shufﬂing operation requires both the target area of pixel shufﬂing and the pixel movement information. An exampleof the pixel shuﬄing area in the ﬁngertip image is shownin Figure 5. White pixels in the ﬁngertip binary mask
image (Figure 5(b)) are the target of pixel shuﬄing in theﬁngerprint replacement process. By saving two types of
information, the ﬁngertip area and the pixel movementin the ﬁngerprint-replaced image, it is possible to restore
the original ﬁngerprint information.
In this paper, in order to save the information necessary
for restoring the input image, we referred to the LSB substitution method, which is the basic digital watermarking
method. The LSB substitution method replaces the leastsigniﬁcant bits with the saved information. That is, theluminance in an image is converted to an odd number
where the binary saved information is 1, while it is converted to an even number where the binary saved infor-

person who knows this seed can easily restore the information. The pseudo-random seed plays a role in the ﬁngerprint restoring process as the key information.
In this saving process, we save both the ﬁngertip binary
mask image like Figure 5 (c) and its position in the wholemask image like Figure 5 (b). Moreover, we also save thepixel movement information, which includes the grid size
and the change of pixel positions in each grid.
Figure 7 shows examples of pixel shuﬄing in a gridmation is 0. However, when considering the case where
the binary mask of pixel shuﬄe area like Figure 5 (c) is
saved by using the LSB substitution-like method, there
is a problem that anyone can specify the pixel shuﬄearea(Figure 6 (a)). This is because the parity of the lu-

69

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

gertip and the ﬁrst joint of the ﬁnger. This ratio is 51.2,

 





 











 













      

  which is calculated by a preliminary experiment. The estimated distance between ridges is calculated as follows,


Dr =





Dr is 5 pixels, so we set the grid size to 5 pixels.

            

In the ﬁngerprint replacement process, we select a generated fake ﬁngerprint from several prepared fake ﬁngerprint images. Fake ﬁngerprints should have a diﬀerent
distribution of minutiae (ﬁngerprint feature points suchas ridge endings and bifurcations) from that of the original ﬁngerprints. Therefore, we use the ﬁngerprint authentication model to select a fake ﬁngerprint. We construct
the ﬁngerprint authentication model using 80 ﬁngerprint
images in DB1 B in FVC200217), referring to FingerPrint   
Fig. 7 Method of saving the pixel movement information; (a)(b) a grid in the ﬁngerprint replacement
process before and after this process, respectively,and (c) the saved data of the gridand saved data. Figure 7(a) and Figure 7(b) represent
pixel shuﬄing in a grid and the grid size is 3. Figure 7(a)means a grid in the input image and Figure 7(b) means
the corresponding grid in the replaced image. For restoring the original information, we need where each pixel inthe replaced image existed in the input image. Therefore,
we save the pixel coordinate information in the input image like the saved data in Figure 7(c). The number of
elements in the saved data of each grid is the square of
the grid size (for example, the number of elements is 9
in Figure 7). The nth element in the saved data is the
position (y, x) of the nth brightest pixel in the grid. By
using the fact that the order of the luminances in eachgrid does not change, the amount of saved information is
reduced, and eﬃcient encoding is performed. Finally, we
save the binarized data of both the grid size and the saved
data of each grid as the pixel movement information.

4.

(1)

where the ﬁngertip size is Df , the ratio calculated in apreliminary experiment is α, and the estimated distance
between ridges is Dr . For example, when Df is 256 pixels,             DfαMatching18). The inputs of the model are two ﬁngerprint
images, and the model calculates the similarity score between the input images19). It determines whether a pairof ﬁngerprint images are identical based on the similarity
score. The deﬁnition of the similarity score is as follows,
Similarity score =m2
n1 n 2(2)where the number of minutiae in one input ﬁngerprint
image is n1 , that in the other input is n2 , and the number of matched minutiae in the two input ﬁngerprint images is m. This ﬁngerprint authentication model identiﬁes whether the two ﬁngerprints are matched by comparing the similarity score with the predetermined threshold.
To determine the threshold, we use FMR (False Matching Rate) and FNMR (False Non-Matching Rate) curves.
FMR is the rate at which two diﬀerent ﬁngerprints are

Experiment Details

We used a Canon EOS Kiss X5 (the image size is 5184
× 3456) to capture input images and the input images’
format is JPEG. For the learning of Finger-GAN10), we
used 1680 grayscale ﬁngerprint images obtained by cropping the center of the DB2 A in FVC200616).

recognized as the same, while FNMR is the rate at which
the same two ﬁngerprints are recognized as diﬀerent ﬁngerprints. We set the threshold to 0.38, where FMR and
FNMR are equal19).

The grid in the ﬁngerprint replacement process is required to be a minimum area that contains both ridges
and grooves of the ﬁngerprint. Therefore, we set the gridsize as the distance between ridges estimated from the
input image. In this estimation, we used the ratio of the
ﬁngertip size to the distance between ridges, where theﬁngertip size is deﬁned as the distance between the ﬁn-

In the ﬁngerprint replacement process, we select one
generated fake ﬁngerprint from the prepared fake ﬁngerprint images so that the similarity score between the original ﬁngerprint and the replaced ﬁngerprint is lower than
the threshold of the ﬁngerprint authentication model.
This means that the replaced ﬁngerprint is not identiﬁed
as the same original ﬁngerprint in the input image.

70

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

tracted from ﬁngertip images captured by cameras tends
to be larger than that of real ﬁngerprints captured byﬁngerprint sensors because the minutiae extracted from        
                    

images captured by cameras include much noise. That
is why there are many ﬁngerprint-replaced samples identiﬁed as the same as the original ﬁngerprint. Therefore,
it is important to select the proper fake ﬁngerprint images used in the ﬁngerprint replacement process. As longas we select a fake ﬁngerprint image properly by using
the ﬁngerprint authentication model, the ﬁngerprint information in the input image can be removed automatically, and we conclude that our method is eﬀective inprotecting the ﬁngerprint information.

     

5.2

Fig. 8 A comparison of the replaced and restored ﬁngerprints

5.
5.1

Restoring ﬁngerprint information in the
input image

Compared the original ﬁngertip in Figure 8(b) with therestored ﬁngertip in Figure 8(d), the restored image does
not seem to be diﬀerent from the original image. Thisis because the pixels inside the ﬁngertips in the output
ﬁngertip image can be automatically returned to the original position of the input image by the saved information.
The total saved information size is about 0.27MB when
we use the input image (4.4MB) in Figure 1. Next, we
calculate the similarity score between each ﬁngerprintreplaced image in the preceding subsection and the original ﬁngerprint in the input image. Figure 9 shows thatthe similarity score of all 400 samples is higher than thethreshold. This means that the ﬁngerprint of the restored
image is recognized as the same ﬁngerprint in the input
image. Note that the similarity scores are not equal to 1
because the luminance of the restored image is changedin the JPEG compression. The result indicates that the

Results and Discussion

Automatic ﬁngerprint information removal and ﬁngerprint image generation

The output ﬁngertip image in Figure 8 is the resultof the ﬁngerprint replacement process when the proposed
method is applied to the input image in Figure 1. We
found that the distribution of the minutiae in the original
image (Figure 8(b)) is entirely diﬀerent from that in the
output image(Figure 8(c)), as shown in the comparisonof the minutiae.
Next, we use the ﬁngerprint authentication model to
conﬁrm that the ﬁngerprints of the output image are not
identiﬁed as the original ﬁngerprints. The ﬁngerprint authentication model is the same as the model describedin the preceding section. We conﬁrm the existence of theproper fake ﬁngerprint images so that the replaced ﬁngerprint is not identiﬁed as the same as the original ﬁngerprint. We prepare 80 fake ﬁngerprint images generated
by Finger-GAN10)and apply the ﬁngerprint replacement

ﬁngerprint information in the input image can be restoredfrom the output image.
5.3

process to ﬁve ﬁnger images using all the 80 fake ﬁngerprint images. And then, we calculate the similarity score
between each replaced ﬁngertip image and the originalﬁngerprint in the input image. The blue histogram in
Figure 9 shows the distribution of the similarity score.
Figure 9 indicates that we can get a fake ﬁngerprint which

Impact of image processing on the proposed method

In a real situation, we sometimes upload images on SNS
after image processing such as ﬁltering. Our method isapplied to input images after such image processing, so weshould investigate the impact of image processing on our
method. We apply three types of image processing to the
input images and conduct the same experiment in Sec. 5.1
and Sec. 5.2. The ﬁrst method is Twitter’s image ﬁltering
named Kilda, which changes the color tone more vivid.
We use JPEG images which are uploaded to Twitter afterﬁltering and downloaded as inputs. The second method
is the noise reduction of the default Photo application inis not recognized as the original person, as we select a generated ﬁngerprint whose score is lower than the thresholdof the ﬁngerprint authentication model. However, if we
do not select the proper fake ﬁngerprint images regardlessof the similarity score, many samples’ similarity scores arehigher than the threshold. The number of minutiae ex-

71IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

To conﬁrm this, we conducted an additional experiment. Figure 11(c) shows the output ﬁngertip image re-   placed with a real ﬁngerprint whose minutiae are diﬀerent
from those of the input image. We found that a more natural image can be generated with a real ﬁngerprint thana generated ﬁngerprint because the ﬁngerprint density ofthe output image is almost the same as that of the input
image. Thus, our method can generate more natural ﬁngerprint images by fake ﬁngerprints with high ﬁngerprint
density.

   

6.

Fig. 9 The distribution of the similarity score in our experiments to check whether ﬁngerprint replacement and restoration are conducted properly

Our method can automatically remove ﬁngerprints inimages, so it can be applied to images on SNS for privacy
protection. Figure 12 shows the overview of a realistic
scenario for applying our method. At ﬁrst, when SNS
users upload JPEG images, the SNS uploader applies our
method to remove ﬁngerprints in the images. Next, theﬁngerprint-replaced images are converted to the JPEG
format and saved to the database of the SNS administrator because they can be displayed on SNS and it takes
less time for page rendering as compared with only uncompressed images being saved. Then, SNS users can access the shared images which do not include the originalﬁngerprints. For restoring the original ﬁngerprints, we require where each pixel in the ﬁngerprint-replaced image
existed in the input image. Thus, the SNS administrator
should also save this information, which is described inSec.3.4. When the police need the original ﬁngerprints
in images for investigations, they can only access the restored images with the permission of the administrator.
A limitation of our study is that saving information aboutpixel shuﬄing may be a burden to SNS administrators.

iPhone. The third method is a sharpening ﬁlter, which
enhances the edges of objects in images.
Figure 10 shows the qualitative and quantitative results of this experiment. In every type of image processing, the replaced ﬁngertip images maintain the color
appearance of the input image, and the restored ﬁngertip images do not look so diﬀerent from the inputs. In
addition, these histograms show that our method can protect the original ﬁngerprint information when we choose
the proper generated fake ﬁngerprint image whose similarity score is lower than the threshold. Moreover, allsamples of the restored ﬁngertip images are recognized as
the same ﬁngerprint in the input image. We demonstrate
the practicality of the proposed method by using input
images operated by several image ﬁlters that SNS users
are familiar with.
5.4

An Example of Application

The naturalness of the output image

Figure 8 shows that we can generate a natural image
that retains the global color distribution of the input ﬁngertips because the ﬁngerprint replacement process is locally performed. In addition, there are few artifacts on
the boundaries of the grid area because the grid size isvery small. However, comparing the original ﬁngertip
image in Figure 11(a) with the output image in Fig-

In a real situation, there is a potential risk that personal
information unintentionally included in shared images ismaliciously used. To give an example, a stalker was able
to locate an idol’s home through shared selﬁes on social
media which include eye reﬂection20). Our study is important to avoid such a potential risk in shared images.

ure 11(b), we found that the ﬁngerprint of the output
image is thicker than the original. This is because the7.

ﬁngerprint images used in the training of Finger-GAN10)

Conclusion

In this paper, we proposed an automatic ﬁngerprint
information replacement method in color images for privacy protection. We conﬁrmed that the output image obtained by applying the proposed method did not include
the original ﬁngerprint information in the input image byusing the ﬁngerprint authentication model. Therefore, we
conclude that the proposed method is eﬀective in privacy

include only the central part of the ﬁngertip, and the generated fake ﬁngerprint is stretched over the entire ﬁngertip in the ﬁngerprint replacement process. That is, theproblem of thicker ﬁngerprints stems not from the ﬁngerprint replacement algorithm, but from the ﬁngerprint
image dataset.

72

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）          

                         

 

 

Fig. 10 The results of the same experiment in Section 5.1 and Section 5.2 withprocessed images as inputs of the proposed method; (a) color correction
image ﬁltering in Twitter (b) noise reduction ﬁltering in iPhone, and (c)edge enhancement with image sharpening convolution ﬁlter




    




 



 


 



 










 

 

         

  

 

   

Fig. 12 Application of the proposed method for a realistic scenario

by applying the improved LSB substitution-like method
using a binary random number table. Our method is a big
step in the research direction of ﬁngerprint replacement
because it allows only persons who have the key information to restore the original ﬁngerprint information, unlike
simple Gaussian ﬁltering.

          
   

Acknowledgement
This research is supported by JST-Mirai Program
Grant Number JPMJMI19B2, JSPS KAKENHI Grant          

Number JP21H05054 and JP19H01129.

Fig. 11 A comparison of an original image and ﬁngerprint replacement results with fake and real ﬁngerprint

References
1) T. Ogane, I. Echizen: “BiometricJammer: Method to Prevent Unauthorized Capturing of Fingerprint in Considerationof Use-friendliness”, Proc. of Computer Security Symposium,
Vol. 2016, No. 2, pp. 355–362 (2016).

protection. Furthermore, the proposed method allows us
to easily restore the original ﬁngerprint information fromthe output image with the seed value of the random number table. We veriﬁed that the restored ﬁngerprint information matched the original ﬁngerprint information by
the ﬁngerprint authentication model and showed that it
is possible to actually restore the ﬁngerprint information

2) T. Orekondy, M. Fritz, B. Schiele: “Connecting Pixels to Privacy and Utility: Automatic Redaction of Private Informationin Images”, Proc. of Conference on Computer Vision and Pattern Recognition, pp. 8466–8475 (2018).
3) R.A. Hummel, B. Kimia, S.W. Zucker: “Deblurring Gaussian
Blur”, Computer Vision, Graphics, and Image Processing, Vol.
38, No. 1, pp. 66–80 (1987).

73

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Kazuya NAKAMURA

4) J. Yu, H. Xue, B. Liu, Y. Wang, S. Zhu, M. Ding: “GAN-BasedDiﬀerential Private Image Privacy Protection Framework for
the Internet of Multimedia Things”, Sensors, Vol. 21, No. 1, p.
58 (2020).

He received a B.E. degree in the Department of Pure Physics from Waseda University, Japan. He is currently a student in the
Graduate School of Advanced Science andEngineering at Waseda University. His research interests are image processing, computer vision, and privacy protection.

5) I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.
Warde-Farley, S. Ozair, A. Courville, Y. Bengio: “Generative
Adversarial Nets”, Proc. of the 27th International Conference
on Neural Information Processing Systems, Vol. 2, pp. 2672–
2680 (2014)6) J. R. Padilla-López, A. A. Chaaraoui, F. Gu, F. FlórezRevuelta: “Visual Privacy by Context: Proposal and Evaluation of a Level-Based Visualisation Scheme”, Sensors, Vol.
15, No. 6, pp. 12959–12982 (2015).

Shugo YAMAGUCHIHe received a B.E. degree in the Department
of Applied Physics and an M.E. degree inthe Department of Pure and Applied Physics
from Waseda University, Japan. He is currently a student in the Graduate School of
Advanced Science and Engineering at Waseda
University. His primary research interests arein texture synthesis, digital human.

7) S. Ribaric, A. Ariyaeeinia, N. Pavesic: “De-identiﬁcation forPrivacy Protection in Multimedia Content: A Survey”, Signal Processing: Image Communication, Vol. 47, pp. 131–151(2016).
8) R. Uittenbogaard, C. Sebastian, J. Vijverberg, B. Boom, D. M.
Gavrila, P. H. N. With: “Privacy Protection in Street-View
Panoramas Using Depth and Multi-View Imagery”, Proc. ofConference on Computer Vision and Pattern Recognition, pp.
10581-10590 (2019).

Hideki TSUNASHIMAHe received an M.E. degree in the Dept. ofElectrical Engineering and Electronics from
Kogakuin University, Japan. He is currently
a student in the Graduate School of Advanced
Science and Engineering at Waseda University. His primary research interests are in
deep generative models, virtual try-on, embodied AI.

9) R. Cappelli: “SFinGe: an Approach to Synthetic Fingerprint
Generation”, Proc. of International Workshop on BiometricTechnologies, pp. 147-154 (2004).
10) S. Minaee, A. Abdolrashidi: “Finger-GAN: Generating Realistic Fingerprint Images Using Connectivity Imposed GAN”,
arXiv preprint arXiv:1812.10482 (2018).
11) M. Attia, M. H. Attia, J. Iskander, K. Saleh, D. Nahavandi,
A. Abobakr, M. Hossny, S. Nahavandi: “Fingerprint Synthesis Via Latent Space Representation”, Proc. of International
Conference on Systems, Man and Cybernetics, pp. 1855-1861(2019).

Shigeo MORISHIMA (Member)
He was born on August 1959. He received his
B.S., M.S. and Ph.D. degrees, all in ElectricalEngineering from the University of Tokyo, in
1982, 1984, and 1987, respectively. He wasa visiting professor of University of Toronto
from 1994 to 1995 and an invited researcher ofAdvanced Telecommunication Research institute from 1999 to 2011. Currently, he is a professor of School of Advanced Science and Engineering, Waseda University. He was a General Chair of ACM VRST2018 and VR/ARadviser of SIGGRAPH ASIA 2018. He is a
General Chair of Paciﬁc Graphics 2022. Hereceived many awards and takes an administration board member of several societies.

12) D.P. Kingma, M. Welling: “Auto-Encoding Variational
Bayes”, Proc. of the 2nd International Conference on Learning
Representations (2014).
13) Z. Cao, G. Hidalgo, T. Simon, S. Wei, Y. Sheikh: “OpenPose:
Realtime Multi-Person 2D Pose Estimation Using Part Aﬃnity
Fields”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 43, No. 1, pp. 172–186 (2019).
14) M.
Ortega,Standalone
Hand
Keypoint
Detector,
https://github.com/ortegatron/hand standalone (2019).
15) C. Rother, V. Kolmogorov, A. Blake: ““GrabCut”: Interactive Foreground Extraction Using Iterated Graph Cuts”, ACM
Trans. on Graphics, Vol. 23, No. 3, pp. 309–314 (2004).
16) R. Cappelli, M. Ferrara, A. Franco, D. Maltoni: “Fingerprint
Veriﬁcation Competition 2006”, Biometric Technology Today,
Vol. 15, No. 7, pp. 7–9 (2007).
17) D. Maio, D. Maltoni, R. Cappelli, J.L. Wayman, A.K. Jain:“FVC2002: Second Fingerprint Veriﬁcation Competition”,
Proc. of the International Conference on Pattern Recognition,
Vol. 3, pp. 811–814 (2002).
18) V. K. Alilou, FingerPrint Matching: A Simple Approach,https://github.com/alilou63/ﬁngerprint (2021).
19) T. Li: “Fingerprint Identiﬁcation by Improved Method of
Minutiae Matching”, Electronic Thesis or Dissertation, Miami
University (2017).
20) Newsweek, Stalker Finds Japanese Idol’s Home from Reﬂections in Her Pupils, https://www.newsweek.com/stalker-ﬁndsidol-reﬂection-pupils-1464373 (2019).

(Received May 28, 2021)(Revised Feb. 1, 2022)

74

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）-- Special Issue on Journal Track Papers in IEVC2021Part II --Game Development Using the Discrimination System to Improve Typing Skills
Surena KAWAHARA†, Teruaki HIRANO†, Noriyoshi OKAMOTO† (Member), Daisuke TAKAHASHI†††Kanto Gakuin University, ††Bunka Gakuen University<Summary> Accurate typing in the home position is recommended to type efficiently on a PC. For this reason, a lot ofsoftware has been developed to learn typing in the home position. However, few of them can determine the actual fingering
simply by presenting the location of the fingers. Therefore, the system to automatically determine have been developed whether
the fingering is in the correct home position using a Leap Motion. Using this discrimination system, the Typing Learning Gameis constructed that can correct subjects to type in the home position. Thus, by handling the fingering data of these subjects,
efficient learning can be achieved. In the experiment, 15 subjects typed 1800 characters each, and the discrimination accuracyof the Fingering Discrimination System was 98.8%, and the fingering was improved by 25.6% from the results of the Typing
Learning Game. The results show that the proposed automatic Fingering Discrimination System can accurately determine thefingering of the subject.
Keywords: typing fingering, leap motion, typing learning systemcorrect fingering in the home position is recommended inmost case. By entering the same key with the same finger
each time, the user can memorize the key position by finger
acquisition. Therefore, it is important to learn the home
position. However, the problem is people who have become
accustomed to inputting with their self-taught fingering.
Some people type with just their index finger. Correcting a
habit by yourself is not an easy task because it requires astrong consciousness to correct it.
The purpose of this paper is to correct the fingering
according to the home position. Thereby, fingering learning
environment is proposed. The fingering learning
environment consists of two proposed methods, the Typing
Learning Game and the Fingering Discrimination System.
The Typing Learning Game has the effect of correcting
fingering by using a unique control method that utilizes theshooting game. The Fingering Discrimination System
automatically evaluates whether the typing is a correct
fingering or not. Conventional typing learning software used
only two pieces of information: typing speed and typing
accuracy. The proposed method improves the learning effectby newly handling fingering information that has not beenused in conventional software. Correcting fingering andmeasuring its effectiveness as an advanced typing learning
environment.
The structure of this paper is described as follows. In
Chapter 2, related research is referred to, respectively,
Section 2.1 for typing learning and Section 2.2 for fingering

1. Introduction
The rate of personal computer (referred to as PC) use
among youth is low level, partly due to the widespread useof smartphones. Few situations occur in society where PCs
are not used. For this reason, it is useful for youths tobecome accustomed to operating PCs before going out into
society.
The results of a survey of PC usage in Japan is comparedfor youth only and for all ages, and the results are shown in
Fig.11), 2). The survey for all ages does not distinguishbetween desktop PCs and laptops. This graph represents the
low rate of PC use among youth, despite the comparable rateof smartphone use.
In this paper, fingering is the key word. The meaning of
fingering is to specify the finger and hand positions thatshould be used when pressing a key. Choosing the suitablefingering will improve the speed and accuracy of typing.
The home position is one of the chosen fingering where thehand is placed in the home row of the middle of the
keyboard and the nearby keys are assigned to each finger
placed. In this paper, the fingering as per the home position
is considered as correct fingering and the other self-taught
fingering is considered as incorrect fingering.
One of the reasons why youths avoid PCs is thedifficulty of keyboard input. Unlike smartphones, keyboards
require a large number of keys, as well as the use of all
fingers. For beginners to learn the keyboard efficiently,

75

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

home position reference positions. Also, when a beginner istyping for the first time, the home position is the best way tokeep these two conditions. Therefore, even with these results,
the home position is still important for beginners.
Takaoka et al. developed a typing learning system for the
purpose of acquiring touch typing ability4). To make thesubject conscious of the home position, the keyboard on thescreen is presented with which finger to press. In addition,
the test is performed using the typing learning system withthe hands hidden by a handkerchief. By hiding their hands,
the subject is forced into a home position.
Takakura et al. proposed a typing learning support
method using passive tactile sensation5). By applying
vibratory stimuli to the finger that has the correct fingering
according to the character to be input, the subject is
encouraged to correct the fingering. The system was able tomemorize the subject's key layout.
Although the above related research was effective in
learning fingering, they used indirect correction methods
such as displaying a guide on the screen, and relied on thesubject's own consciousness to correct the fingering. To
correct the subject's habit of typing with their own fingering,
it is necessary to correct them directly without relying on thesubject. The method of giving stimuli to the input finger cancorrect the habit directly, but it requires wearing special
gloves on the hand. Therefore, in this paper, the Typing
Learning Game and Fingering Discrimination System is
proposed that can directly correct fingering without the
subject being conscious of it.

Utilization ratio[%]

100
80
60
40
20
0

Each device
youth

all ageFig.1 PC usage rates for youths and all ages

discrimination. Chapter 3 describes the proposed method in
detail. Section 3.1 details the Typing Learning Game, amethod for correcting fingering. Section 3.2 details theFingering Discrimination System, a method forautomatically discriminating fingering. Section 2.1
corresponds to Section 3.1, and Section 2.2 corresponds to
Section 3.2 in terms of the relationship between the related
research and the proposed method. In Chapter 4, theprocedure and results of an experiment is described tomeasure the effect of the proposed Typing Learning Gameon fingering correction and to determine the accuracy offingering discrimination of the Fingering Discrimination
System. In Chapter 5, the results of the experiment are
discussed. Chapter 6 gives a conclusion.

2. Related Research

2.2 Related research on fingering discrimination

2.1 Related research on typing learning

As a previous research of automatic fingering
discrimination, Feit et al. performed fingering discrimination
by motion tracking 52 markers attached to the hand with 12
high-speed infrared cameras (already referred to in Section
2.1). In addition, a spectacle-type eye-tracking device wasused to discriminate touch typing. In this research, fingering
was regarded as a strategy for how to efficiently input the
keyboard. The acquired fingering data was used to find out
what kind of strategy was superior.
Chigira et al. proposed a typing practice systemequipped with a fingering discrimination function by using
Leap Motion6). The acquired fingering data gives feedback
by detecting the weak key and asking a word containing the
weak key. The conditions that are judged to be weak keys
include not only typos, but also incorrect fingering thatcannot be obtained with ordinary typing software.

First, the studies that investigated the significance of thehome position are described below. Then, related studies
that devise ways to make the home position learnable aredescribed. In the last paragraph, related research iscompared with this paper, and the significance of the
proposed method is clarified.
Feit et al.’s research demonstrated that home position
typing is not necessarily superior3). That research was foundthat pressing the same key with the same finger and using
more fingers are more important factors that determine
typing speed and accuracy than whether typing in the home
position. Subjects who met these two conditions hadoutperformance typing ability regardless of their home
position. However, all the subjects who excel in their own
fingering had the correct fingering for F and J, which are the76

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

These keys are the reference positions for the home position.
By using these key assignments, the left and right index
fingers are naturally placed on the F and J keys whencontrolling. Also shoot a bullet with the space key.
Nevertheless, with only the above-mentioned movement
control, only 3 keys can be corrected. The highlight function
corrects the fingering of other keys. The inverted triangle
with the character at the top of the screen drawn in the centeris the enemy. The inverted triangle with the letters drawn inthe center at the top of the screen is the enemy. As an
example, let us assume that Enemy L and K are in the screen.
These enemy with character will not be destroyed when hit.
However, when the L key is pressed, a circle is drawn
around the enemy L and highlighted. If the enemy ishighlighted, it can be destroyed by hitting it with a bullet.
On the other hand, when Enemy L is highlighted, Enemy K
cannot be destroyed. Press the K key to switch the highlight
display from Enemy L to Enemy K. The last pressed key isretained as the highlight display key. As a supplement,characters other than the movement keys F and J appear as
enemies.
Consider the case where the subject operates this game.
As the basis of a shooting game, it is necessary to always

The related research using motion markers describedabove is capable of discriminating with very good accuracy.
However, it is not easy because of its high cost andcomplexity, such as the use of multiple expensive
high-speed cameras. In this paper, simple and inexpensive
system for learning is proposed. In addition, similar related
research using Leap Motion differ from this paper in thatthey do not have a calibration function to obtain the key
positions. By performing the calibration by the subject
himself, it is possible to take into account the accuracy
difference derived from the shape of individual fingers.

3. Proposed Method
The proposed Typing Learning Game is a learning
software that has a home position correction effect. Next, theproposedFingering
Discrimination
Systemcanautomatically discriminate whether the subject's fingering iscorrect or incorrect. The correlation diagram between thesubjects and the proposed method is shown in Fig.2. This
environment corrects the fingering for the subject andevaluates the fingering. The subject will receive feedback on
the evaluation results. The details of the proposed Typing
Learning Game are described in Section 3.1, and the detailsof the proposed Fingering Discrimination System aredescribed in Section 3.2.
3.1 Typing Learning Game with fingering correction
In the Typing Learning Game proposed, the home
position is corrected naturally without the subject being
conscious of it. For that purpose, the control method of theshooting game is utilized. Correct the fingering by typing
the player's movement control.
A captured image of the actual play screen is shown in
Fig.3. The filled triangle at the bottom of the screen is theplayer. Assign the player's movement control to the F key
for the left direction and the J key for the right direction.

Fig.3 Play screen

Fig.2 Fingering learning environment

77

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

resolution of 1.2 mm on average was obtained. This issufficient accuracy when the key pitch of the keyboard isconsidered.
The fingering discrimination method is described below.
First, the system recognizes that a key has been pressed. The
fingertip closest to the pressed key is the finger that pressed
the key. Then, the system obtains the finger from the sensor.
If the relationship between the key and the finger is the sameas the home position, the fingering is correct; otherwise, thefingering is incorrect. For example, if the A key is pressedby the little finger of the left hand, discrimination by thesystem is a correct fingering, and if it is pressed by any otherfinger, discrimination is an incorrect fingering.

move laterally with the F and J keys to align the vertical axis
with the enemy and avoid the bullets of the enemy. On topof that, in this game, highlighting must be done. The index
finger needs to be placed on the F and J keys as much as
possible, which causes constraints on pressing thehighlighting keys with other fingers. Due to this restriction,
the highlight display key is also pressed with the correct
fingering. Therefore, it is possible for the subject to naturally
correct the home position by these control methods.
3.2 Fingering Discrimination System
The Fingering Discrimination System proposed in thispaper can discriminate fingering in a simpler way. The
system discriminates between correct and incorrect fingering
as per the subject's home position. Improve learning is
aimed by making it possible to newly collect fingering data.
In typing learning, feedback such as the amount oflearning and the score to provide subjects is important7). The
system proposed in this paper shows the subject which key
fingering is most likely to be mistaken.
Use Leap Motion is a USB device that can acquire
fingertip position information and finger type. This sensor isa high-performance motion sensor specialized for hand
tracking. A schematic diagram of this system is shown in
Fig.4. Using the stand in this way, the sensor is hung fromabove. The sensor gives a bird's-eye view of the keyboard
and the hands that cover it.
To use this system, calibration to obtain the key position
coordinates is necessary in the preliminary stage. Have thesubject input one character at a time and obtain the fingertip
coordinates as the key position. The fingertip coordinates are
obtained from the sensor. Calibration allows us to tolerate
differences in keyboard dimensions such as key pitch tosome extent.
There is a verification to find the actual spatial resolutionof Leap Motion8). As a result of verification, a spatial

Fig.4

4. Experiment of Proposed Methods
4.1 Experimental procedure
The accuracy experiment of the Fingering
Discrimination System and the effect measurement of the
Typing Learning Game were carried out at the same time in
one flow. The number of subjects is 15 (A to O).
As a beginning, subjects were asked to enter 300 Roman
characters from A to Z, one by one, not as words. At thattime, the Fingering Discrimination System measures the
subject's fingering while typing. Next, a Typing LearningGame is played to correct the subject's fingering. After
playing, the 300-character test is performed again. 3 sets ofthis procedure will be performed at weekly intervals, and thepatient will be monitored. The 15 subjects inputting 1800
characters ((300 + 300)×3). The flowchart for one set isshown in Fig.5. Each set took about 20 minutes, and there
were no restrictions on the subjects.

Fig.5 Flow of a procedure

Outline of Fingering Discrimination System

78

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig.6 Transition of correct fingering rate for each set (legend α is conventional method)

typing with their own fingering. The mean age of subjects
E-O is 35.0 ± 30.0. These subjects tend to input slower thanthe others. However, subject J has a reasonably fast input
speed due to his unique fingering.

The result of fingering discrimination by the system inthe 300-character test is recorded and the actual fingeringobtained with the naked eye is compared. The
discrimination accuracy of the Fingering Discrimination
System will be determined and discussed.
To measure the effect of a Typing Learning Game using
the result of automatic fingering discrimination by aFingering Discrimination System. By comparing thefingering before and after playing the game, the effect of thisgame on correcting fingering will be determined. Althoughtyping speed is an important factor in evaluating typing
ability, this paper does not consider speed because accuracy
is focused on of input.
Madison et al. demanded that the larger the key pitch of
the keyboard, the higher the input speed and accuracy9). Our
proposed method use a 109-key keyboard with a key pitchof 19 mm.
The attributes of the subjects are described below. 4
subjects, A to D, are working adults who are skilled typists.
The skilled subjects typed efficiently, with both speed andaccuracy of input. On the other hand, the subjects from E to
O either do not use the PC very often or have a habit of

4.2 Experiments to compare conventional methods
The proposed typing learning game will improvelearning by correcting fingering. However, it is expectedthat conventional typing learning systems also have acorrective effect by the subjects themselves. The experiment
was conducted by replacing the Typing Learning Game ofthe proposed method with a conventional one in the flow
shown in Fig.5. As a representative, use the “Home Position
Basic” of “Free Typing Practice Software” by P-Ken10). Thisis a web browser application that can be played on theInternet without installation. The flow of the game is thesame as that of normal typing software, where the user
enters the characters displayed on the screen one by one.
The correct fingering for each letter is displayed on thescreen to encourage learning of the home position. The
number of subjects is two, different from the subjects
playing proposed method, and they are beginners in typing.

79

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

between values of correct fingering rate in “1b” and that in“3a” was 15.3%±1.7, which was 12.3% less than the
proposed method's 25.6%±15.5. The seven subjects who
showed a positive fingering rate lower than α in the "1b"phase were preceded in the "3b" phase. This difference canbe attributed to the direct fingering correction. In
conventional typing learning software, a guide to correct
fingering is displayed on the screen, but subjects who areaccustomed to their own fingering ignore it and type. Suchsubjects to correct their habits spontaneously is difficult, andtherefore external correction is necessary.
The discrimination accuracy of the Fingering
Discrimination System was 98.8% of one character. It isthought that the calibration of acquiring the key position
coordinates one character at a time enables highly accurate
discrimination. Since it is necessary to acquire all thecharacter keys, the system cannot be used immediately, but
in this paper, accuracy is emphasized. During calibration,
the position coordinates acquired on the sensor's camera
image can be mapped on the keyboard. Therefore, if anabnormal value is found, it is corrected on the spot. On theother hand, a failure in discrimination occurs when a correct
fingering is identified as an error, or vice versa. This is dueto the overlapping of fingers. There were also cases where
Leap Motion lost tracking of the hand.

4.3 Experimental results
The accuracy of the Fingering Discrimination System asa result of the experiment, the discrimination accuracy per
character was as high as 98.8% out of the total number of
inputs of 27,000 characters. See reference for details of thedata11). The effect measurement results of the Typing
Learning Game for each subject shown in Fig.6 are based on
the fingering data collected by this system. In addition, thelegend α of the triangular markers described below is thedata of the conventional method. The legend α is the averageof two subjects who played conventional typing learning
software, treated as one person.

5. Discussion
The correct fingering rate on the vertical axis in Fig.6 isthe probability of typing in the home position per character,the higher the rate, the better. The numbers on the horizontal
axis represent the number of sets, and the before and after
representing whether the test results are before or after
gameplay. The capital letters in the legend represent eachsubject.
In “1b”, the result of the test without playing the gameeven once, there is a large difference between those who aregood at typing and those who are not. However, in "3a", theresult after playing the game three times, the difference issmaller, indicating that the less skilled players have
improved. When the target correct fingering rate was set at79.3%, which was the average correct fingering rate of theadvanced players in the preliminary experiment, 13 out of
15 players exceeded the target rate in "3b".
Subjects A to D have a high percentage of correct
fingerings at “1b”, but the overall results show a slightblurring. From this, that even advanced users do not always
type in the perfect home position can be seen.
Subjects A to D have a high correct fingering rate from
"1a", and are therefore not affected by the fingeringcorrection effect. On the other hand, subjects E to J arestrongly affected by the fingering correction effect. In
particular, the percentage of correct fingering tends toincrease from “1b” to “1a”, the first play. However, thegradient became more gradual with each play. Therefore, thecorrect fingering rate is expected to become saturated after acertain level of increase.
Similar experiments were conducted with theconventional typing learning software described in Section
4.2, and the results showed that P-Ken also measured an
improvement in fingering. However, the means difference

6. Conclusion
In this paper, the fingering is focused on, and aimed tocorrect to the correct home position. For this purpose, theTyping Learning Game was constructed that can correct
fingering. In the Typing Learning Game, the fingeringcorrection can be done by typing the controls of a shooting
game. However, only this typing learning game cannot
evaluate whether the fingering is correct. Therefore, theFingering Discrimination System is developed to measure
whether the fingering is correct or not. The Fingering
Discrimination System can automatically determine thefingertip position coordinates, the position of each key, andthe type of finger while typing. In the experiment, theeffectiveness of the Typing Learning Game and theaccuracy of the Fingering Discrimination System wasmeasured in a series of steps.
The effectiveness of the game was measured by theautomatic discrimination of the Fingering Discrimination
System, and the accuracy of the discrimination wasdetermined by comparing the fingering with the actual
fingering with the naked eye. As for the measurement of theeffect of the Typing Learning Game, the results of the80IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

experiment showed that the average fingering of all 15
subjects improved by 25.6% after playing 3 sets. The
accuracy of the T was as high as 98.8% per character out ofa total of 27,000 (300 characters ×6 sets ×15 subjects)
characters. From these experiments, it was found that the
beginners of typing receive a correction effect. As an
immediate task, the correction effect of this system on
typing beginners will be quantitatively evaluated by a t-test
to see if significant differences are found.
As future works, learning environment is aimed that cancorrect fingering more firmly by combining these two
methods. In addition, to discuss the meaningfulness ofcorrecting the home position, the system will be constructed
to automatically measure whether the subject is now touchtyping and aim to create an environment that can evaluate
overall typing ability.

“Mind the Gap: The Effect of Keyboard Key Gap and Pitch on Typing
Speed, Accuracy, and Usability, Part3,” Human Factors The Journal of
the Human Factors and Ergonomics Society, Vol.57, No.7, pp.
1188-1194 (2015).
10) ICT Proficiency Assessment, the P-Ken Assessment of ICT,
https://manabi-gakushu.benesse.ne.jp/gakushu/typing/homeposition.ht
ml (2021).
11) S. Kawahara, T. Hirano, D. Takahashi, K. Nakayashiki, N. Okamoto, R.
Tachino: “Development of Typing Learning Game That Can Correct
Fingering by Finger Position Descrimination,” IIEEJ Reserch
Proceedings, Vol.295, pp. 52-57, (in Japanese) (2021).
(Received May 31, 2021)(Revised Jan. 29, 2022)

Surena KAWAHARA
He received B.Eng. and M.Eng. degree in 2019and 2021, respectively, from Kanto Gakuin
University. His research interests include patternrecognition, signal and image processing.

References
1)

Cabinet Office, Government of Japan,
https://www8.cao.go.jp/youth/youth-harm/chousa/h29/net-jittai/pdf/sokuhou.pdf (2021).

2)

Teruaki HIRANO

Ministry of Internal Affairs and Communications,

He received Ph.D. degree in Informatics from the
Graduate school of Kogakuin University in 2005. He
became an Assistant Professor of Networking andMultimedia Course College of Science andEngineering Kanto Gakuin University in 2013. His
research interests include Image processing,Augmented reality and Virtual reality.

https://www.soumu.go.jp/johotsusintokei/statistics/data/180525_1.pdf(2021).
3) A. Feit, D. Weir, A. Oulasvirta: “How We Type Movement Strategies
and Performance in Everyday Typing,” Proc. of The 2016 CHI
Conference on Human Factors in Computing Systems (CHI’16), pp.
4262-4273 (2016).

Noriyoshi OKAMOTO

4) E. Takaoka, T. Hashimoto: “Development of Web-Based Touch-Typing(Member)

He received a Doctor of Engineering degree in 1985
from Kogakuin University. He became a Professor of
Networking and Multimedia Course College of
Science and Engineering Kanto Gakuin University in
1999. His research interests include biometrics,
signal and image processing and pattern recognition.

Education System and Analysis of Students’ Learning Activities,” IPSJSIG Technical Report, Vol.2010-CE-106, No.2, pp.1-10(in Japanese)(2010).
5) R. Takakura, K. Hakka, B. Shizuki: “Exploration of Passive Haptics
Based Learning Support Method for Touch Typing,” Proc. of the 31st
Australian Conference on Human-Computer-Interaction (OZCHI’19),

Daisuke TAKAHASHI

pp. 529-533 (2019).

He received a Doctor of Engineering degree from
Kanto Gakuin University. He became an AssistantProfessor of Bunka Gakuen University in 2021. His
research interests include biometrics, Image
processing and pattern recognition.

6) Y. Chigira, M. Nishi: “Development of Typing Practice System(Fingers) by Identifying a Finger That Presses a Key with a Leap
Motion Controller as a Function of Focused Typing Practice with AI,”
JSTE Journal, Vol.61, No.3, pp. 223-230, (in Japanese) (2019).
7) E. Minoura, S. Takeoka, C. Liao: “Effects of Presentation of Learning
History Information on Sustainable Learning,” JSET Journal, Vol.43,
Suppl, pp. 37-40, (in Japanese) (2019).
8) F. Weichert, D. Bachmann, B. Rudak, D. Fisseler: “Analysis of the
Accuracy and Robustness of the Leap Motion Controller,” Sensors(Basel) 2013, Vol.13, No.5, pp. 6380-6393(2013).
9) H. Madison, A. Pereira, M. Korshoj, L. Taylor, A. Barr, D. Rampel:

81

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）-- Special Issue on Journal Track Papers in IEVC2021Part II --High-Speed and Accurate Authenticity Judgment Using Physically Unclonable
Function of Inkjet Printed CodeKazuaki SUGAI†, Kohei SHIRAI†, Kitahiro KANEDA† (Member), Keiichi IWAMURA† (Member)†Dept. of Electrical Engineering, Tokyo University of Science,<Summary> The distribution of easily imitated counterfeit products, such as food packaging, brand tags, and pharmaceutical labels, has
become a serious economic and safety concern. To address this issue, we propose a system for authenticity judgment of genuine andcounterfeit products with high speed and accuracy, focusing on the physically unclonable function of an inkjet-printed code and a locallylikely arrangement hashing (LLAH) system that performs high-speed image retrieval. In this study, we verified that the proposed system
has high discriminability and stability, based on highly accurate results obtained from a dataset of up to 4,000 sheets. In addition, theeffectiveness of the system was also confirmed by validating it on multiple printers and comparing it with Oriented FAST and Rotated
BRIEF (ORB), a typical feature matching method, in terms of discriminability and speed.
Keywords: inkjet print, physically unclonable function, authenticity judgment, image processing

1. Introduction
In recent years, manufacturing technology for various
products has advanced at a considerably fast pace, and the trade
has expanded along with it. While this has increased
convenience in daily life, determining the authenticity of
information and distinguishing between genuine and counterfeit
products has become increasingly difficult for the average
consumer. According to an OECD/EIPO survey, the amount ofcounterfeit and pirated goods traded worldwide is approaching$500 billion, which is equivalent to approximately 2.5% of theworld's total trade1). Counterfeit goods range from branded
goods and airline boarding passes to pharmaceuticals and foodproducts. According to a WHO report, a wide range ofcounterfeit products, with an increased Internet penetration rate,
provide a dizzying array of both branded and generic drugs2). In
addition, counterfeit medicines have a tremendous impact onsociety, not only in terms of economic damage but also as athreat to the health and safety of consumers, infringement ofintellectual property rights and trademark rights, and as asource of income for illegal organizations.
In the field of security printing, holograms and security inks
are used, and artifact metrics using transmitted light images havebeen used in Japan for some time; systems that use lightscattering to determine authenticity have also been studied 3)–8).
This study aims to construct a system that enables producers
and consumers to judge product authenticity by cross-verifying
printed codes on labels and tags at the manufacturing, production,82

and delivery stages.
At the microscale, ink grains can be observed in inkjet
printouts. The shapes and positions of these grains are slightly
different for each print, and the combination of these grains canbe regarded as unique features. Herein, the features (shape andposition of the ink grains) to determine authenticity.
As reported in Reference7)-8), this system uses theproperties of physical unclonable function (PUF), but insteadof using optical properties such as the material properties of
the ink, it uses individual differences that exist during printing.
The advantage of this system is that it does not require special
optical analysis for verification or electrical analysis, such as
the widely introduced RFID, and the equipment required forthe determination is easy to use.
Then, we confirmed the accuracy of the proposed system bycomparing it with a printed image that was physically
reproduced by the same printer (not a malicious clone) in an
ideal environment.

2. Physically Unclonable Function of Inkjet
Printed CodeIn this section, we introduce the physically unclonable
function of inkjet-printed matter, which is used for authenticity
determination.
The shape and positional relationship of ink on paper canrepresent a unique feature of materials printed by an inkjet printer.
The following factors can cause differences in the shapes andpositions of the ink,

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 3 Part of the QR code to be extracted and digital
microscope used for capturing

Fig.1 Three codes and an enlarged photo of the upper left corners

Real

Fake

Fig. 2 Overview of authenticity judgment system

1.
2.
3.
4.

Slight airflow on the paper surface.
The direction in which the header moves when the ink isfired.
Air resistance of the ink in flight.
Differences in the fiber quality of the adherend.

3. Authenticity Judgment System
3.1 Overview of authenticity judgment system
An overview of this system is presented in Fig. 2, which canbe divided into the following three main stages:
1. Image capture by a microscope.
2. Image pre-processing.
3. High speed authenticity judgment using LLAH.

These properties can be used for authenticity determination,
as the uniqueness of these object fingerprints (physicallyunclonable function properties) for each print hinders malicious
reproduction. These properties are independent of the printertype and apply to all printers.
In this study, QR codes, typically used for airline tickets
and pharmaceutical labels, were used as an embedding
medium to obtain characteristics unique to inkjet printing.
Specifically, a single color (grayscale information) with adensity of 200 (Max 255) is embedded in the white area in theupper left corner of the QR code and printed.
As an example, Fig. 1 shows an enlarged image of the area
where a single color is embedded in the position detection partof the upper left corner of three different printed QR codes. Eachcode has a single color embedded in the white area in the upperleft corner. The same URL can then be read from each QR code.
Fig.1 shows that the shapes and positions of printed inks differ
greatly among the three types of QR code in which grayscale
information is embedded, making it difficult to accurately
reproduce the same ink arrangement on a microscopic level.
Therefore, the features of printed materials can be used for
determining authenticity.

3.2 Image capture by microscopeFirst, a magnified capture of the upper left corner of theprinted QR code, in which grayscale information wasembedded, is captured using a digital microscope. Figure 3
shows detailed capturing of an actual digital microscopeusedure3.3 Image preprocessing
Using the captured images, we preprocessed the images toextract the feature points of the LLAH system to be used in
the authenticity assessment. Image preprocessing is
performed as follows. In all the steps, OpenCV 3.30 was used.
To extract appropriate feature points from microscopic
images, an appropriate binary image was created by passing
the image through multiple filters and extracting the hue ofthe ink by dividing the grayscale portion into three colors:cyan, magenta, and yellow. Figure 4 shows the extracted ink
colors before and after image preprocessing (cyan image isshown as an example).

83

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a) Before
preprocessing

Fig.5 Calculation of affine invariants(b) After
preprocessing(e.g., Cyan)

Fig.6 Configuration of the database9)
Fig. 4 Comparison before and after preprocessing

1.
2.
3.

4.
5.
6.
7.
8.
9.

combinations (the triangle combination in this case) was
calculated as one feature. When registering the extracted featuresin the database, for fast retrieval, the index (Hindex) and quotient(Q) were calculated from the extracted features so that they
could be retrieved using Eq. (2).

Smoothing by Gaussian filter.
Adjust contrast.
Extract the three ink hues by setting threshold values.
Hue: Cyan: 90 to 150, Magenta: 150 to 180, Yellow: 20
to 40, Saturation All: 90 to 255, Value All: 120 to 255
Grayscale conversion.
Adaptive binarization.
Smoothing by Gaussian filter.
Re-binarization.
Invert bits to create an image for input into LLAH.
Shrink the image again to remove the noise.

𝑟

For the features, we use affine invariants9)-11) as geometricinvariants. Because we are computing features for very fine parts
as geometric invariants that are robust to changes in the positionof feature points, as opposed to changes in noise and camera
shake, for use in smartphones in future systems. When there are
four points A, B, C, and D on the same plane as shown in Fig. 5,
the affine invariant X for point A is calculated using Eq. (1).
𝑃 𝐴, 𝐶, 𝐷
𝑃 𝐴, 𝐵, 𝐶

𝑄𝐻

𝐻

,

2

where r is the discrete value of the i-th affine invariant, di isthe number of discrete value levels, and Hsize is the size of thehash table. In this study, we set d = 10, m = 6 and Hsize = 1086.
This means that the same index, Hindex, and quotient, Q, can beobtained from the same feature vector in the two matching
images. This value is a discretization of the ratio of 15 triangles,
which is very discriminative and almost unique. Therefore,
during retrieval, for features with the same Hindex (whencollisions occur with the same index), quotient Q is used forcomparison instead of the feature value. Therefore, as shown in
Fig. 6, the image ID, feature point ID, and quotient Q are
registered in the database. The image ID and feature point ID are
the identification numbers of the image and feature point,respectively.

3.4 LLAH extraction of feature points andcalculation of feature values

𝑋

𝑑

1

3.5 Outputting the result of the authenticity assessment

where P (A, B, C) is the area of the triangle formed by thethree points A, B, and C. This feature is computed from the pointclosest to the point and evaluated discretely. To improve
discrimination, for each point of one image, the combination of
six adjacent points is calculated based on the x-coordinate and ycoordinate, and the combination of four points is then calculated
from the six selected points. The area ratio of the resulting 15

Matching is performed by voting on registered images using
a voting table. First, as in registration, the index of the databaseis calculated for each feature point, and the obtained index isused to refer to the list shown in Fig.6, For each item in the list,
we check if the quotient matches, and if it does, we increase theimage ID item in the voting table.

84

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Finally, the image ID with the highest number of increments
is output as the retrieval result and the number of increments as
the number of matches. The retrieval result is used forauthenticity judgment as follows:
1. Positive judgment: If output image ID matches the database
and the correct result is output, the number of matches
exceeds the threshold value.
2. False judgment: If output image ID does not match the
database and the wrong result is output
3. Undecided: If there is no corresponding image in the
database

invariant like LLAH and faster than other SIFT and SURF
image-matching algorithms.
For the ORB, the original images were matched with eachother using Hamming distance and cross-checking, and thefeatures with a distance of 300 or less were counted as thematched features; the total number was used as the number ofmatches. Subsequently, as in the LLAH system, a threshold of 5
was set to determine the authenticity for comparison.
4.2 Equipment
The following instruments were used for the verification
Printer:
IP8730 (print resolution 9,600 dpi) by Canon Japan
MG3530 (print resolution 4,800 dpi) by Canon JapanPro C7100 (print resolution 4,800 dpi) by RICOH Japan
2. Microscope:
Auto focus microscope VIEWTY (Maximum
magnification 250×) by 3R SOLUTION Japan
3. XY Stage: Manual XY stage by CHUO PRECISION
INDUSTRIAL Japan
4. Computer:
OptiPlex 7050 (Intel Core i7-7700 @ 60 GHz, 64 GBRAM) by Dell America

1.

4. Verification
4.1 Overview
For verification, QR codes were printed by Canon IP8730 as
captured in the environment described in Section 3.2. The
captured images were categorized as follows:
1. Registration data: 1000 images were taken for registration.
2. Collation data A: The same 1000 QR codes were recaptured and used to match the registration images.
3. Collation data B: Dataset taken using the same procedure
as dataset A to improve reliability.
4. Fake data A: Data taken from the second 1000 QR codes
printed by the same inkjet printer without registration.
5. Fake data B: Data taken from the second 1000 QR codes
printed by the same inkjet printer without registration to
improve reliability.
The resolution of the image was 1920 pixels (width) and
2160 pixels (height). The magnification of the microscope
was fixed at 40×, and the position of the microscope wasadjusted using the XY stage described in section 3.2. A totalof 5000 images were collected. To show that the images will not
be affected by their environment they were taken on differentdays and under the same scale, lighting, and brightness
conditions.
To verify the accuracy of system, we changed the size of thedatasets registered in the database. Specifically, verifications
were conducted for four dataset sizes: 100, 250, 500, and 1000
registrations, collation A B, and fake A B data.
In addition to prove that there is no printer dependency, we
performed the same verification using the data of 100 prints eachfor both collation and fake on a home printer (Canon MP3530)
and an industrial printer (Ricoh Pro C7100).
In addition, to demonstrate the superiority of LLAH in termsof accuracy and speed, it was tested on the matching and fakedata of each of the 100 sheets to compare it with Oriented FAST
and Rotated BRIEF (ORB)12),13), which are scale- and rotation-

5. Results
Table 1 shows the results of the authenticity evaluation foreach registration dataset. As can be seen from Table 1, the results
are highly accurate, with 100% positive results for both collationand fake data, Table 2 shows the average number of matches,
Minimum number of matches for collation data and maximum
number of matches for fake data and average processing time foreach registration dataset size separately for collation and fakedata. Furthermore, Fig. 7 shows a box-and-whisker diagramof the number of matches for the collation data (Fake data isomitted because the maximum value is 4). As can be seen from
Table 2, the number of matches of fake data increases slightly as
the registration dataset size increases: however, even with 1000
sheets, the number of matches between the collation data andfake data is average approximately 34.5 (minimum 3.25) times
farther apart from Table 2, confirming that the feature points andfeatures have very high discriminability.
In addition, the LLAH system can perform authenticity
judgment at a very high speed of 13.69 ms even if the registration
dataset size increases to 1000, without any change in the average
processing time of the index-value-based matching system. The
required storage capacity is 48.0 MB for a dataset size of 1000,
which is approximately 53.3 times less than the 2.56 GB (2,560
MB) required when the original images are stored.

85

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 1 Judgment results for each dataset size of the authenticity judgment system
Dataset size 100

Dataset size 250

Positive

False

judgment

judgment

Collation A

100

0Collation B

1000Fake B

0

Fake A0

Undecided

Positive

False

judgment

judgment

0

250

0

0

0

250

0

0

0

100

0

0

250

0

100

0

0

250

Dataset size 500

Undecided

Dataset size 1000

Positive

False

Undecided

Positive

False

Undecided

judgment

judgment

judgment

judgment

Collation A

500

0Collation B

500

0

0

1000

0

0

0

1000

0

0

Fake A

0

0

500

0

0

1000

Fake B

0

0

500

0

0

1000

Table 2 Comparison of number of matches, and processing time for each dataset size
DatasetAverage number of

Average number of

Minimum number ofMaximum number

Averagematches (Collation)

matches (Fake)

matches (Collation)of matches (Fake)processing
time [ms]

100

44.38

1.03

14

3

13.24

250

44.28

1.21

14

3

13.05

500

44.01

1.28

13

3

13.67

1000

49.06

1.42

13

4

13.69

Table 3 Comparison of accuracy, number of matches, and processing time of averages for each dataset size
Canon PIXUS iP8730 Data A
Positive

False

judgment

judgment

Collation

100

0

Fake

0

0

Canon PIXUS iP8730 Data B
Undecided

Positive

False

judgment

judgment

0

100

0

0

100

0

0

100

Canon PIXUS MG353
Positive

False

judgment

judgment

Collation

100

0

Fake

0

0

Undecided

RICOH Pro C7100
Undecided

Positive

False

judgment

judgment

Undecided

0

100

0

0

100

0

0

100

Table 4 Comparison of LLAH and ORB
Verification with LLAH
Positive

False

judgment

judgment

Collation

100

0

Fake

0

0

Verification with ORB
Undecided

Positive

False

judgment

judgment

0

94

0

6

100

0

0

100

86

Undecided

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Canon iP8730

Canon MG3530

RICOH Pro C7100

Fig. 8 Comparison of three types of inkjet printersFig. 7 Comparison of a number of matches for each dataset size

Next, for printer dependency, results of verification using theimages are shown in Fig. 8 and Table 3. The figure shows that
irregular scattering of ink grains, which is necessary for thisverification, can be observed in all inkjet printers. The table also
shows that all the printers were able to determine the authenticityof the collated and fake data at 100%. This confirms that thissystem can determine the authenticity independent of the printer.
Furthermore, the results of the comparison between LLAH
and ORB are shown in Table 4. This table shows that LLAH
was able to determine the authenticity of 100% of the images,
while ORB was not able to match some of the images with 94%
of the matching images. We can see that the average number ofmatching points for LLAH is 44, while the maximum numberof matching points for ORB is approximately 10. In addition, theaverage speed of LLAH was 13.24 ms, while that of ORB was
89.30 ms. These results confirm the superiority of the LLAH interms of both accuracy and speed.

Fig. 9 Example of printing with a laser printerwhether the system can determine the authenticity of images
captured by laser printers (Fig. 9), widely used in offices.
In addition, as a long-term future work, this system could
be adapted for food and pharmaceuticals that are stored in agood environment with a short cycle time but would also need
to be tested for longer storage periods and in poor
environments.

Acknowledgments
The authors would like to thank the reviewers for theirconstructive comments to improve our paper. This work was
supported by the JSPS KAKENHI Grant Numbers 21K11931.

References
1)

OECD and European Union Intellectual Property Offices: “Trends in Tradein Counterfeit and Pirated Goods, Illicit Trade”, OECD Publishing Paris

6. Conclusion[PDF file]. Retrieved fromIn this study, we proposed a system that can perform
authenticity judgment for inkjet-printed code with high speed
and accuracy. We obtained sufficient results a 100% positive
judgment rate with an average processing time of 13.69 ms withup to 4000 datasets (total of collation and fake data).
For future work, it will be desirable to move from a
microscopic environment to a more practical environment to amore practical environment that combines a standard camera
and the microlens of a smartphone. In addition, to maintain thecurrent high level of discrimination even as the size of theregistration dataset increases, it is necessary to consider
improvements such as setting limits on the sampling and listingof feature points and features. Furthermore, although an inkjet
printer was used in this verification, it is necessary to verify

https://www.oecd-ilibrary.org/trade/trends-in-trade-in-counterfeit-andpirated-goods_g2g9f533-en (2019).
2)

The World Health Organization (WHO): “Growing Threat from Counterfeit
Medicines. Bulletin of the World Health Organization”, Vol. 88, pp. 247–248(2010).

3)

K. -T. P. Lim, H. Liu,Y. Liu, J. -K.W. Yang; “Holographic Colour Prints forEnhanced Optical Security by Combined Phase and Amplitude Control”,
Nature Communications, Vol. 10, p. 25 (2019).

4)

Z. Song, T. Lin, L. Lin, S. Lin, F. Fu, X. Wang, L. Guo: “Invisible Security
Ink Based on Water-Soluble Graphitic Carbon Nitride Quantum Dots”,
Angewandte Chemie, Vol. 55, pp. 2773–2777 (2016).

5)

H. Matsumoto, T. Matsumoto: “Artifact-Metric Systems. Technical Reportof IEICE”, ISEC 2000-59, pp. 7–14 (2000).

6)T. Matsumoto, N. Iwashita: “Financial Operations and Artifact Metrics”,
Financial Studies, Vol. 23, pp. 153–168 (2004). (In Japanese).

87

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

7)

R. Arppe-Tabbara, M. Tabbara, T. J. Sorensen: “Versatile and Validated

Kazuaki SUGAI

Optical Authentication System Based on Physical Unclonable

He received the B.S. degrees in ElectronicalEngineering from Tokyo University of Science in
2021. Since 2021, he has been a Master course
student in Electronical Engineering of Tokyo
University of Science. His research interests are
document understanding, information security andcyber real media processing.

Functions”, ACS Applied Materials & Interfaces, Vol. 11, pp. 64756482 (2019).
8)

Y. Liu, F Han, F. Li, Y. Zhao, M. Chen, Z. Xu, X. Zheng, H. Hu, J. Yao,
T. Guo, W. Lin, Y. Zheng, B. You, P. Liu, Y. Li, L. Qian: “Inkjet-printed
Unclonable Quantum Dot Fluorescent Anti-Counterfeiting Labels with

Kohei SHIRAIArtificial Intelligence Authentication”, Nature Communications, pp.1-

He received the B.S. and M.S. degrees inElectronical Engineering from Tokyo University of
Science in 2019, 2021 respectively. His researchinterests are document understanding, information
security and cyber real media processing.

7 (2019).
9)

T. Nakai, K. Kise, M. Iwamura: “Use of Affine Invariants in Locally Likely
Arrangement Hashing for Camera-Based Document Image Retrieval”,
Document Analysis Systems, Vol. 7, pp. 541–552 (2006).

10) M. Iwamura, T. Nakai, M. Kise: “Improvement of Retrieval Speed andKitahiro KANEDA (Member)

Required Amount of Memory for Geometric Hashing by Combining

He received the B.S. and M.S. degrees in
Mechanical Engineering from Waseda University in1984 and 1989, respectively. He also received the
M.S. degree in Electrical Engineering from Duke
University in 1995 and the Ph.D. degree in ElectricalEngineering from Tokyo University of Science in2010. He is now with NAGASE & CO., LTD., andvisiting professor of Osaka Prefecture University.
His research interests are document understanding,information security and cyber real media
processing.

Local Invariants”, Proc. BMVC 2007 (2007).
11) M. Iwamura, T. Nakai, K. Kise: “Real-Time Document Image Retrieval
for a 10 Million Pages Database with a Memory Efficient and Stability
Improved LLAH”, Proc. International Conference on Document Analysis
and Recognition, pp. 1054–1058 (2011).
12) S. A. K.Tareen, Z. Saleem: “A Comparative Analysis of SIFT, SURF,
KAZE, ORB, and BRISK”, Proc. of International Conference on

Keiichi IWAMURA(Member)
He received the B.S. and M.S. degrees inInformation Engineering from Kyushu University in1980 and 1982, respectively. During 1982-2006, hebelonged to Canon Inc. He received the Ph.D in
Tokyo University. He is now Professor of Tokyo
University of Science. His research interests include
Coding theory, Information security, Digitalwatermarking. He is Fellow of Information
Processing Society of Japan, Chair of TechnicalCommittee on Enriched Multimedia of the Instituteof Electronics, Information and Communication
Engineers in Japan.

Computing, Mathematics and Engineering Technologies (2018).
13) H.-J. Chien, C.-C. Chuang, C.-Y. Chen, R. Klette: “When to Use What
Feature SIFT, SURF, ORB, or A-KAZE Features for Monocular Visual
Odometry”, Proc. of International Conference on Image and Vision
Computing New Zealand (2016)(Received May 24, 2021)(Revised January 21, 2022)

88

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Upon the Special Issue on
CG & Image Processing Technologies
Supporting and Expanding Human Creativities
Editor: Shinya KITAOKA(DWANGO Co., Ltd.)The AI technology is expected to become a key technology for solving social problems
and SDGs (Sustainable Development Goals), such as declining birthrate, aging population,shortage of labor, depopulation area, and so on, to which mature society, especially Japanis facing. The application scope of the technology in the field of image processing is
expanding beyond the image generation and object recognition to include areas related tocreativity, such as attribute conditioned image generation, super-resolution, image
colorization, and line art coloring.
On the other hand, deep learning techniques don’t always have good nature atinterpretability and explainability of results required for creativity processing. Also, there
are still issues to be addressed in AI technology, such as the ineffectiveness to the humancomputer interface field by a simple application. So, technologies to support and expand
the human creativity are gathering much concern, and quite promising in wide variety of
applications.
Based on the above observation, Trans. on IE and VC has planned the special issue on“CG & Image Processing Technologies Supporting and Expanding Human Creativities”
targeting the June 2022 issue.
In this special issue, 2 papers that were accepted within the publication schedule ofJune issue, are contained. The topics are “Face Reconstruction Algorithm based on
Lightweight

Convolutional

NeuralNetworks

andChannel-wise

Attention”

and“Investigation of New Design Method Rooted in Local History and Culture throughApplication of Photogrammetry: A Case Study on Application of Maruko-bune, aTraditional Boat Unique to Lake Biwa in Japan, to Architectural Design”.

We believe thisspecial issue theme will be common to various study items and the results of these papers
will stimulate the eagerness of the related activities.
Last but not least, I would like to thank all the reviewers and editors for theircontribution to improving the quality of papers. I would also like to express my deepest
gratitude to the members of the editorial committee of IIEEJ and the staff at the IIEEJ
office for various kinds of support.

89

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper -- Special Issue on CG & Image Processing Technologies Supporting and Expanding Human Creativities -Face Reconstruction Algorithm based on Lightweight Convolutional Neural
Networks and Channel-wise Attention
Haoqi GAO† (Student Member) , Koichi OGAWARA†† The Faculty of System Engineering, Wakayama University<Summary> 3D face reconstruction and face alignment are two highly relevant topics in face research.
However, for these tasks, computational complexity is another consideration besides the training accuracyof the model. Our goal is to regress the 3D facial geometry and dense correspondence information from thegiven 2D image. Thus, in this paper, we ﬁt the 3D morphable model based on a lightweight convolution neural network of the ShuﬄeNetV2 Plus series network and channel-wise attention model, which can improve
the representation ability of the network and the performance of the 3D face reconstruction task without
increasing the number of network parameters. Evaluations on test datasets show that our approach achieves
signiﬁcant performance improvements on both 3D face reconstruction and dense face alignment tasks. Alignment performances evaluate on AFLW2000-3D, and our method obtains a lower mean Normalized Mean
Error (NME(%)) of 3.694.
Keywords: 3D face reconstruction, Attention model, ShuﬄeNet

1.

methods are applied to solve 3D face reconstruction problems, most previous 3DMM-based networks have complex

Introduction

structures and large model parameter spaces, making itdiﬃcult to achieve convergence in network training.

Since 2D images do not provide any depth information,
3D face reconstruction from 2D images has been a fundamental task and challenge in computer vision and graph-

Our goal is to regress the 3D geometry of the face andits dense counterpart information.
Inspired by the eﬀectiveness of the Lightweight net-

ics in recent years. Recovering 3D face geometry canaddress numerous challenges (e.g., large poses and occlu-

work, which exploits pointwise group convolution andchannel shuﬄe. Our network structure combines threemajor achievements: 3DMM2),3), ShuﬄeNetV2 Plus se-

sions). Traditional approaches employ a set of 3D baseshapes to capture morphological models of face shapevariations1). Blanz et al.2),3) suggested that the geometricstructure and texture of a face can be approximate by alinear combination of orthogonal basis vectors obtainedby Principal Components Analysis (PCA) and proposeda typical statistical 3D face model called 3D Morphable
Model (3DMM). Based on 3DMM, most of the previous
methods are mainly based on optimization algorithms tories of units9),10) and Squeeze-and-excitation (SE) attention mechanism11) to improve the representation abilityof the network. For 3D face reconstruction, the proposed network signiﬁcantly reduces the computational
cost while maintaining the model accuracy.
Additionally, we report the experimental result of ourswith Selective kernel network (SK) attention12), Coordi-

obtain the coeﬃcients of 3DMM. But such approaches areusually time-consuming due to the high optimality com-

nated attention (CA)13), and no attention mechanisms
respectively. It demonstrates the eﬀectiveness of the SE
attention mechanism embedded in our network.
Our method performs better than 3DDFA5) andDAMDNet1) with a reduction in Normalized Mean Error (NME(%)) of 1.725 and 0.202 on the AFLW2000-3D
dataset respectively. Experimental results show that ouralgorithm improves the performance of 3D reconstruc-

plexity and suﬀer from local optimal solutions and poor
initialization. With the development of deep learning,
using convolutional neural networks (CNNs) can solve
numerous computer vision problems. There have been
several works to use CNNs for estimating 3DMM coeﬃcients4)–8) to recover the corresponding 3D information from 2D face images, which signiﬁcantly improves
the reconstruction quality and eﬃciency. Although CNN

tion.

90

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

2.

in a fully weakly-supervised fashion. Furthermore, unsupervised approaches explored, Tewari et al.31) proposed

Related Workan encoder network with an expert-designed generative

We reviewed several previous works on the area of facereconstruction in this section.
2.1

model that can be trained end-to-end in an unsupervisedmanner. Nevertheless, unsupervised methods do not perform well in large poses and heavily occluded faces.

3D face reconstruction

Traditionally, a sparse set of 2D facial ﬁducial points
represents face shape5). Cootes et al.14),15) suggested that3.

Method Overview

Real-world tasks have motivated much work to design lightweight architectures to achieve better accuracy, which includes Xception32), MobileNet33),34), ShufﬂeNetV29),10), and GhostNet35). Group-wise Convolutionsubspace analysis can be used to model shape variations.
Unlike 2D Shape Models, a 3DMM model separates rigid(pose) and nonrigid (shape and expression) transformations, which allows the model to cover a range of shapevariations and preserve shape simultaneously5). 3DMM
based methods are popular for 3D face reconstruction
tasks, and a large amount of work proposes to improveand Depthwise Convolutions (DWConv) are both crucial
in these works10).
Model’s architecture contains convolutional layers with
H-swish activation34), a stack of reconstructed Shuﬄenet
block units that are structured in four stages, ﬁnally,

the performance of 3DMM based modeling. Most previous approaches regress the 3DMM coeﬃcients by solving

time-consuming and rely heavily on the accuracy of land-with FC layers. The pipeline of our method is shown in
Fig. 1, where blocks in gray are 3 ʷ 3 ShuﬄeNet unit,
larger green blocks are ShuﬄeNet Xception unit, which
is deeper than 3 ʷ 3 ShuﬄeNet unit, yellow and orange
are also ShuﬄeNet unit with the sizes of kernels as 5marks or other feature points25).

and 7, respectively. H-swish[x]=x ReLU6(x+3)
was utilized
6

a nonlinear optimization problem to establish the point
correspondence between a 2D facial image and a canonical 3D face model, including facial landmarks16)–21) andlocal features22)–24). However, such methods are usually

CNN-based approaches have achieved remarkable suc-as a drop-in replacement for ReLU. ReLU6 is a modiﬁ-

cess in many areas. In contrast to nonlinear optimiza-cation of ReLU where we constrain the activation to ation, CNNs can be used as regressors to estimate 3DMM
coeﬃcients directly, which can signiﬁcantly improve themaximum size of 6. Attention mechanisms can automatically learn the importance of each feature channel andreconstruction quality and eﬃciency. Alp. Guler et al.26)
designed a fully convolutional network to estimate thedense correspondence between a given 3D template andan input image. Jourabloo et al.4) introduced cascaded

enhance useful features and suppress non-informative features by the learned importance metric, which uses widely
in many applications11)–13). The addition of SE atten-

CNNs to regress the 3DMM parameters, which takes lots

tional mechanisms can enhance the representation powerof our network structure. The computation distributionof time due to the multiscale and iterations. More re-in the ShuﬄenetV2 network is small on DWConv, andsearchers have proposed to obtain the reconstructed 3D
face directly bypassing 3DMM coeﬃcient regression. For
example, Jackson et al.27) devised a 3D binary volumetric as a new representation of the 3D structure. Deng etal.28) proposed a multi-task that the reconstructed branch

the main computation is on 1 × 1 convolution. Therefore,
extending the convolution kernel of DWConv can improvethe eﬀect without increasing the computational weights.
So we reconstructed the Shuﬄe Xception unit with a size
is 5.
The issue with achieving high model capacity and eﬃciency is how to maintain a large number and equallyof the 3D vertex representation incorporated with the existing box and 2D landmark regression branches during
joint training. Feng et al.29) suggested UV position map
representation, which records the 3D shape of a face in
UV space. Although their approach is no longer limited to the space of 3DMMs, it needs complex network
structures and lots of time to predict voxel or mesh information25). Some approaches proposed to reconstructmultiple faces simultaneously. Like Zhang et al.30) sug-wide channels with neither dense convolution nor too
many groups9),10). Following the insight from some literature32)–34), an eﬃcient ShuﬄeNet unit included DWConv
with Batch Normalization layer. Figure 2 shows the operation of the reconstructed ShuﬄeNet unit in our network structure. The high eﬃciency in the shuﬄe of eachbuilding block applied ’channel split’, ’concat’, ’channel

gested a single-shot multi-face reconstruction framework

shuﬄe’ operations. This structure enables using more

91

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6WDJH

6(

6KXIIOH
8QLW 

6KXIIOH;FHSWLRQ 

6KXIIOH
8QLW 

6KXIIOH
8QLW &RQ9%1
+6ZLVK

6WDJH

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6WDJH

Fig. 12WKHU

The ﬂowchart of our network based on ShuﬄeNet units9),10)

feature channels for information communication, whichis regarded as a kind of feature reuse. It also improves&KDQQHO6SOLW&RQY%1+6

network eﬃciency by reducing the degree of network fragmentation and element-wise operations. SE layer induces
the model to pay more attention to the contribution offeature areas of the human face and reduces the inﬂuence&RQY%15HOX

NVL]H NVL]H':&RQY%1
VWULGH 

NVL]H NVL]H':&RQY%1&RQY%1+6

NVL]H NVL]H':&RQY%1
VWULGH &RQY%1+6&RQY%15HOX
6(/D\HU*OREDO$YHUDJH
3RROLQJ)XOO\&RQQHFWHG5HOX)XOO\&RQQHFWHG5HOX
6LJPRLG

6(/D\HUof other unrelated features.
3.1

)&

)&
+6ZLVK

6($YJ3RRO&RQ9%1
+6ZLVK

6(

6KXIIOH
8QLW 

6(

6KXIIOH;FHSWLRQ 

6(

6KXIIOH
8QLW 

6(

6KXIIOH
8QLW 

6WDJH

&RQFDW&RQFDW

3D Morphable Model&KDQQHO6KXIIOHBlanz et al.2),3) design 3D Morphable Model (3DMM)
to recover the 3D facial geometry. It is composed of a parameterized generative 3D shape, a parameterized albedo
model, together with an associated probability density on
the model coeﬃcients4). The 3DMM renders a 3D face

&KDQQHO6KXIIOH

D 6KXIIOH8QLW VWULGH 
ZLWK6(OD\HU

E 6KXIIOH8QLW VWULGH 
ZLWK6(OD\HU&KDQQHO6SOLW ':&RQY%1
 ':&RQY%1

 ':&RQY%1
VWULGH &RQY%15HOX

shape of S with a linear combination over a set of Princi-&RQY%15HOX

pal Opponent Analysis (PCA) basis functions. It is oneof the most widely used methods for describing the space ':&RQY%1

of 3D faces nowadays. The 3DMM model is expressed as ':&RQY%1&RQY%15HOX&RQY%15HOX

6(/D\HU ':&RQY%1&RQY%1+6&RQY%15HOX

follows:&RQY%15HOX ':&RQY%1*OREDO$YHUDJH
3RROLQJ)XOO\&RQQHFWHG5HOX)XOO\&RQQHFWHG5HOX
6LJPRLG

6(/D\HU

S = S̄ + Aid αid + Aexp αexp ,(1)&RQFDW&RQFDW&KDQQHO6KXIIOH

where S̄ represents the mean shape. The identity basis&KDQQHO6KXIIOH

Aid and the expression basis Aexp come from the Basel
Face Model (BFM)36) and the Face Warehouse model37)
respectively. Zhu et al.5) used 40 bases from BFM to generate the face shape component and 10 bases from Face
Warehouse to generate the face expression component.

F 6KXIIOH;FHSWLRQ8QLW VWULGH 
ZLWK6(OD\HU

Fig. 2G 6KXIIOH;FHSWLRQ8QLW VWULGH 
ZLWK6(OD\HU

Shuﬄe unit with SE layer and reconstructed
Shuﬄe Xception unit with SE layer14),15)

and expression.

αid and αexp are the corresponding coeﬃcient of identity

In the 3DMM ﬁtting process, we use Weak Perspective

92

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Projection to project 3DMM onto the 2D face plane1). It
can be expressed as follows:

region, and C is a constant that smoothly links piecewisedeﬁned linear and Non-linear parts1),37).

Vp = f ∗ Pr ∗ R ∗ S + t ,Here, we compare with networks proposed earlier (e.g.,
3DDFA48) and DAMDNet1)), using their same loss exper-(2)

imental function Lwpdc vdc and Lwpdc wing setting during
the training, respectively. By comparing the experimental results in Table 1 and Fig. 3, it shows the accuracy

where Vp stores the coordinates of the 3D vertices projected onto the 2D plane, f is the scale
Pr is the factor, 
1 0 0, R is aorthographic projection matrix Pr =0 1 0
rotation matrix consisting of 9 parameters and t is thetranslation vector. Our goal is to predict the coeﬃcients
p = [f, R, t, αid , αexp ]T for rendering a 3D face, which includes a 40d identity parameter vector, a 10d expression
parameter vector, and other 12d vector.
3.2of the results.
Lwpdc vdc = λ1 ∗ Evdc + Ewpdc
Lwpdc wing = λ2 ∗ Ewing + Ewpdc

In Fig. 3, we give the results of diﬀerent loss function
with diﬀerent λ1 and λ2 settings. As shown in the ﬁgure,our network optimizes with the Lwpdc wing loss function
and sets parameter λ2 = 0.7.

Loss function

Previous literatures on 3D reconstruction1),5),26) havedescribed some widespread losses.
Weighted Parameter Distance Cost (WPDC)5): minimize the distance between the ground truth pg and themodel predicted parameter p0 . The researchers claimed
that the parameters in the 3DMM contribute to the ac-

4.

0 T

g

The 300W-LP dataset5) contains 61,225 synthetic face
images across large poses (1,786 from IBUG40), 5,207
from AFW41), 16,556 from LFPW42), and 37,676 fromHELEN43)) along with their corresponding 3DMM annotation coeﬃcient values. These images are synthesized0

Ewpdc = (p − p ) W (p − p )(3)from 300W40) through a morphable model-based 3D proﬁling algorithm proposed in the paper5)and are of cover-W = diag(w1 , ..., w62 ) ,wi = ||V (p0 (i)) − V (pg )||/



wi ,

Experiments

4.1 Datasets(a) 300W-LP

curacy of ﬁtting with diﬀerent impacts.
g

(8)

(4)

where the diagonal matrix W contains the weights of eachparameter, wi indicates the importance of the i-th coefﬁcients, computed from how much error it introduces tothe locations of 2D landmarks after projection.
Vertex Distance Cost (VDC)5): minimize the vertex

age across large pose ranges from -90 to 90 degrees.
(b) AFLWThe ALFLW dataset39) contains 21,080 faces in the wild,
which is a large-scale face database including multi-poses
and multi-views, and each face annotates with 21 featurepoints. At test time, we divide it into 3 subsets based

distances between the ﬁtted and the ground truth 3D
face.

with 11,596, 5457, and 4027 samples, respectively.

Evdc = ||V (pg ) − V (p0 )||2 ,on absolute yaw angle: [0◦ , 30◦ ], [30◦ , 60◦ ] and [60◦ , 90◦ ](c) AFLW2000-3D(5)

The AFLW2000-3D dataset is a sample of 2,000 faces selected from the AFLW dataset39). Zhu et al.5) introduced
this dataset and annotated its corresponding 3DMM co-where V (·) is the face construction and weak perspective
projection as Eq.(2).
For 3D face vertices reconstructed with the estimated
3D parameters, Wing Loss1),38)is another loss function
Ewing which deﬁne as:⎧⎨ω ln(1 + |ΔT (P )|)/ if |ΔT (P )| < 
Ewing =⎩|ΔT (P )| − C otherwise ,eﬃcients and the corresponding 68 3D facial landmarks.
For evaluation, we also split it into 3 subsets according to their absolute yaw angles: [0◦ , 30◦ ], [30◦ , 60◦ ] and[60◦ , 90◦ ] with 1,312, 383 and 365 samples, respectively.
(d) Menpo-3D
The Menpo-3D44) dataset contains 8,955 challenging
frames varying in illuminations, poses, and occlusions.

(6)
g

0

ΔT (P ) = V (p ) − V (p ); C = ω − ω ln(1 + ω/) , (7)4.2 Evaluation metric
A commonly used metric for 3D face alignment tasks
is the Normalized Mean Error (NME), which deﬁne as

where ω = 10 sets the range of the Non-linear part to (ʵω, ω),  = 2 constrains the curvature of the Non-linear

93

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

NME errors by diagonal length of the bounding box on the AFLW and AFLW2000-3D datasets
AFLW DataSet(21 pts)
AFLW 2000-3D Dataset(68 pts)
Method[0◦ ,30◦ ] [30◦ ,60◦ ] [60◦ ,90◦ ] Mean
Std[0◦ ,30◦ ] [30◦ ,60◦ ] [60◦ ,90◦ ] Mean
Std
RCPR45)
5.43
6.58
11.530
7.85
3.24
4.26
5.96
13.18
7.8
4.74
ESR46)
5.66
7.12
11.94
8.24
3.29
4.6
6.7
12.67
7.99
4.19
SDM47)
4.75
5.55
9.34
6.55
2.45
3.67
4.94
9.76
6.12
3.21
48)
3DDFA
5.00
5.06
6.74
5.60
0.99
3.78
4.54
7.93
5.42
2.21
3DDFA+SDM48)
4.75
4.83
6.38
5.32
0.92
3.43
4.24
7.17
4.94
1.97
3DFAN49)
3.38
4.46
5.59
4.48
1.11
SRN50)
2.97
3.85
5.09
3.97
1.07
DAMDNet1)
4.359
5.209
6.028
5.199 0.835
2.907
3.830
4.953
3.897
1.02
Weakly-Supervised30)
2.60
3.48
4.78
3.62
1.10
28)
RetinaFace
2.57
3.32
4.56
3.48
1.01
Our((wpdc vdc 1)
4.871
6.019
6.857
5.916 0.997
3.302
4.333
5.662
4.432 1.183
Our(wpdc wing 0.7)
4.217
5.004
5.77
4.997 0.777
2.795
3.593
4.695
3.694 0.954
Table 1the average of visible landmark error normalized by thebounding box size1),5) instead of the common inter-pupil.
The formula for NME can be written as Eq.(9).
N M E(ki , ki∗ ) =



10( 

M

||ki − ki∗ ||21 √,
M i=1 wbbox × hbbox












































(9)






















:HLJKW/ZSGFBZLQJ/ZSGFBYGF

Herein, M is the number of facial landmarks. ki isthe estimated landmarks points, ki∗ is their corresponding
ground truth. wbbox and hbbox deﬁne as the width andFig. 3height of the ground-truth bounding box.
4.3





Diﬀernent loss optimization results for diﬀerent
weights

and RetinaFace28), which are used 256d 3DMM coeﬃ-

Training details

cients and 1k 3D vertices regression for 3D face reconstruction, respectively. In contrast, our method only used

Our proposed method is implemented on an Nvidia
GTX 2080 Ti GPU using Pytorch. By applying random
perturbations to the pitch, yaw, and roll angles of the300W-LP dataset to obtain 687,854 training images48).
The datasets were cropped around the facial region andpublic 62d 3DMM coeﬃcients. Compared with WeaklySupervised30), the NME values of our algorithm decrease
under large yaw angles. Our method is not designed
for such tasks speciﬁcally, it still outperforms the mostsimilar approach DAMDNet1) and 3DDFA48), which arejointly detecting and reconstructing 3D face shape30).
Figure 3 shows the mean NME(%) values of optimization with diﬀerent loss functions Lwpdc vdc and Lwpdc wing

resized to 120 × 120. We train the network using SGD
optimizer with a learning rate of 0.02, a momentum of0.9, and a weight decay of 5e-4. For a total of 60 training
epochs with a batch size of 256, we adjust the learning
rate to 0.004, 0.0008, and 0.00016 after 30, 40, and 50
epochs, respectively.

Table 1 lists NME value by diagonal length of thebounding box with the best results highlighted. We eval-in diﬀerent λ1 , λ2 for our model.
Lower NME is much better. The results show that optimizing the model with Lwpdc wing (blue) will be better
than Lwpdc vdc (green). Also, we can see that for the
Lwpdc vdc loss function, the minimum value of NME(%)

uate our model for the 3D dense face alignment task on
the AFLW2000-3D and AFLW datasets, which are di-obtain when the λ1 is equal to 1, and for the Lwpdc wingloss function, the model performs best when the λ2 isvided into three groups by comparing it with several baseline methods(e.g., 3DDFA48), 3DFAN49), DAMDNet1),
SRN50), Weakly-Supervised30) and RetinaFace28)). The
results in Table 1 show that in comparison to these baseline methods, our method can achieve a comparable result

equal to 0.7, which can signiﬁcantly improve the face

4.4

Experimental resultsalignment accuracy in a full range of poses.
In real-life scenarios, faces often have multiple challenging elements, such as makeup faces occluded simultaneously (Makeup occ) and large-scales expressions face due
to blurring (Exp blur).

on facial landmark localization. Weakly-Supervised30),

94

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 2

NME errors and comparisons of model size and complexity on AFLW and AFLW2000-3D dataset

GFLOPs
MethodMobileNet33)
GhostNet35)
DAMDNet1)
ShuﬄeNet10)
With SK12)
With CA13)
Our

12.82M
3.98M
2.76M
4.41M
7.29M
4.65M
5.58M

([SBEOXU([SB3RVH
([SBLOOAFLW DataSet(21 pts)
[30◦ ,60◦ ] [60◦ ,90◦ ] Mean
5.044
5.874
5.076
5.285
6.149
5.302
5.209
6.028
5.199
5.058
6.127
5.13
5.187
6.17
5.256
5.054
5.919
5.067
5.004
5.77
4.997

Std
0.782
0.839
0.835
0.963
0.881
0.845
0.777

[0◦ ,30◦ ]
2.853
3.039
2.907
2.797
2.995
2.862
2.795

AFLW 2000-3D Dataset(68 pts)
[30◦ ,60◦ ] [60◦ ,90◦ ] Mean
Std
3.757
4.921
3.844
1.037
3.9
5.2
4.047
1.088
3.830
4.953
3.897
1.02
3.629
4.834
3.753
1.025
3.9
5.26
4.052
1.14
3.693
4.812
3.789
0.979
3.593
4.695
3.694 0.954

tures proposed in the last two years mainly include MobileNet33),34), ShuﬄeNet9),10), GhostNet35), Xception32)

2FFBLOO
2FFBLOO

and so on. For the operational eﬃciency of networks,
the commonly used evaluation factors are GFLOP andparameters. These two factors are used to measure thecomplexity of the model to judge the algorithm performance. The parameter value is related to the model size,
GFLOP value is related to the model speed.
In Table 2, we learn with Lwpdc wing loss function and'$0'>@'$0'>@2ULJLQDO

2FFB([S2FFBH[S[0◦ ,30◦ ]
4.31
4.472
4.359
4.204
4.413
4.229
4.217

'')$>@'')$>@0DNHXSBRFF/DUJHB2FF

0.70
0.05
0.12
0.12
0.17
0.12
0.12

Paramscompare each of these models with our model. In termsof accuracy, the result of our experiments performs bet-

2XUV2XUV'00

ter. Also, we list the results of diﬀerent attention mechanisms embedded in our network model. Here, we compare'')$>@'')$>@the original ShuﬄeNet network without attention mechanism, with SK attention mechanism12), and with CA
attention mechanism13). Although the GFLOPs and parameters of ShuﬄeNet are much higher than GhostNet ʟs
GFLOPs and DAMDNet ʟs parameters. However, the increase in model size and speed is acceptable compared to'$0'>@the increased accuracy. In comparison with CA and SK
attention mechanisms, the SE module can improve theprecision of our network. In summary, our framework can

2XUVsigniﬁcantly improve the accuracy of the network without
adding too many network parameters and GFLOPs.

Fig. 4

Figure 5 shows more examples of random selection
from diﬀerent challenging situations. Top row are the input images which come from Menpo-3D44), AFLW20003D5) and 300-test-3D40). The second and third rows are
the 3D landmarks (68 keypoints) plotted for diﬀerent display views. Our algorithm not only predicts the key

Comparison results with 3DDFA48) and DAMDNet1)
methods

Figure 4 shows some comparative visualization results with latest method. When the test image undermultiple challenging conditions (e.g., Makeup occ), ourpoints but also estimates the 3D face structure. The
fourth and ﬁfth rows represent the images of the predicted 3D pose and depth estimates, respectively. The
last two rows are the 3D face model with the texture
image of the face and the reconstructed 3D face projection on the input images. As can be seen, our method

method can accurately get the location points of occlusion
or makeup parts and provide complete 3D face structure
when compared to 3DDFA48), and DAMDNet1).
The core of lightweight networking is to lighten the
network in size and speed while maintaining as much accuracy as possible. Several lightweight network architec-

is robust to occlusions, illumination, and large pose andexpression variations.

95

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 5

Face alignment and 3D face reconstruction visualization results

sentational power of the network. Extensive experiments

5.

Conclusions

on two challenging datasets (AFLW and AFLW2000-3D)
demonstrate that our method obtains the mean NME(%)

In this paper, we focus on solving the 3D facial reconstruction problem. Brieﬂy, we took both the face imageof 4.997 and 3.694, respectively, which is a signiﬁcant improvement over existing similar methods. In addition, we
visualize the 3D face reconstruction results of our modelon Menpo-3D datasets. Our algorithm could substantially improve the quality of the 3D face reconstruction.

and 3DMM coeﬃcients as inputs and learned the model
to evaluate the intrinsic consistency between the predicted 3DMM coeﬃcients and the corresponding face image, oﬀering supervision for 3D face model learning. Our
method is based on lightweight convolutional neural net-

References

works to address relatively complex structures of previous
networks. In addition, our network incorporates a chan-

1) L. Jiang, X. J. Wu, J. Kittler: “Dual Attention MobDenseNet(DAMDNet) for Robust 3D Face Alignment”, Proc.
of IEEE/CVF International Conference on Computer Vision

neled attention mechanism that can improve the repre-

96

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

2) V. Blanz, T. Vetter: “A Morphable Model for the Synthesisof 3D Faces”, Proc. of 26th Annual Conference on Computer
Graphics and Interactive Techniques, pp. 187–194 (1999).

W. Christmas, M. Rätsch, J. Kittler: “A Multiresolution 3D
Morphable Face Model and Fitting Framework”, Proc. of 11th
Joint Conference on Computer Vision, Imaging and Computer
Graphics Theory and Applications, pp. 79–86 (2016).

3) V. Blanz, A. Mehl, T. Vetter, H. P. Seidel: “A Statistical
Method for Robust 3D Surface Reconstruction from SparseData”, Proc. of 2nd International Symposium on 3D Data Processing, Visualization and Transmission, pp. 293–300 (2004).

20) C. Cao, Q. Hou, K. Zhou: “Displaced Dynamic Expression
Regression for Real-time Facial Tracking and Animation”,
ACM Transactions on Graphics (TOG), Vol.33, No.4, pp. 1–10
(2014).

4) A. Jourabloo, X. Liu: “Large-pose Face Alignment via CNNbased Dense 3D Model Fitting”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 4188–
4196 (2016).

21) L. A. Jeni, J. F. Cohn, T. Kanade: “Dense 3D Face Alignment from 2D Videos in Real-time”, Proc. of 11th IEEE International Conference and Workshops on Automatic Face andGesture Recognition (FG), Vol. 1, pp. 1–8 (2015).

5) X. Zhu, X. Liu, Z. Lei, S. Z. Li: “Face Alignment in Full Pose
Range: A 3D Total Solution”, IEEE Trans. on Pattern Analysis
and Machine Intelligence, Vol.41, No.1, pp. 78–92 (2017).

22) C. M. Grewe, S. Zachow: “Fully Automated and Highly Accurate Dense Correspondence for Facial Surfaces”, Proc. of European Conference on Computer Vision (ECCV), pp. 552–568(2016).

Workshop (ICCVW), pp.504–513 (2019).

6) E. Richardson, M. Sela, R. Kimmel: “3D Face Reconstruction
by Learning from Synthetic Data”, Proc. of Fourth International Conference on 3D Vision (3DV), pp. 460–469 (2016).

23) P. Huber, Z. H. Feng, W. Christmas, J. Kittler, M. Rtsch:“Fitting 3D Morphable Face Models using Local Features”,
Proc. of IEEE International Conference on Image Processing(ICIP), pp. 1195–1199 (2015).

7) E. Richardson, M. Sela, R. Or-El, R. Kimmel: “Learning Detailed Face Reconstruction from a Single Image”, Proc. ofIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1259–1268 (2017).
8) F. Liu, D. Zeng, Q. Zhao, X. Liu: “Joint Face Alignment and3D Face Reconstruction”, Proc. of European Conference on
Computer Vision(ECCV), pp. 545–560 (2016).

24) S. Romdhani, T. Vetter: “Estimating 3D Shape and Texture
using Pixel Intensity, Edges, Specular Highlights, Texture Constraints and a Prior”, Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR),
Vol. 2, pp. 986–993 (2005).

9) X. Zhang, X. Zhou, M. Lin, J. Sun: “Shuﬄenet: An Extremely
Eﬃcient Convolutional Neural Network for Mobile Devices”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 6848–6856 (2018).

25) X. Tu, J. Zhao, M. Xie, Z. Jiang, A. Balamurugan, Y. Luo ... J.
Feng: “3D Face Reconstruction from a Single Image Assistedby 2D Face Images in the Wild”, IEEE Trans. on Multimedia,
Vol. 23, pp. 1160–1172 (2020).

10) N. Ma, X. Zhang, H. T. Zheng, J. Sun: “Shuﬄenet v2: Practical Guidelines for Eﬃcient Cnn Architecture Design”, Proc. ofEuropean Conference on Computer Vision(ECCV), pp. 116–
131 (2018).

26) R. Alp Guler, G. Trigeorgis, E. Antonakos, P. Snape, S.
Zafeiriou, I. Kokkinos: “Densereg: Fully Convolutional Dense
Shape Regression in the Wild”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 6799–
6808 (2017).

11) J. Hu, L. Shen, G. Sun: “Squeeze-and-Excitation Networks”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 7132–7141 (2018).
12) X. Li, W. Wang, X. Hu, J. Yang: “Selective Kernel Networks”,
Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 510–519 (2019).

27) A. S. Jackson, A. Bulat, V. Argyriou, G. Tzimiropoulos:“Large Pose 3D Face Reconstruction from a Single Image via
Direct Volumetric CNN Regression”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 1031–1039(2017).

13) Q. Hou, D. Zhou, J. Feng: “Coordinate Attention for Eﬃcient Mobile Network Design”, Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 13713–13722 (2021).

28) J. Deng, J. Guo, E. Ververas, I. Kotsia, S. Zafeiriou: “Retinaface: Single-shot Multi-level Face Localisation in the Wild”,
Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5203–5212 (2020).

14) T. F. Cootes, G. J. Edwards, C. J. Taylor: “Active Appearance Models”, IEEE Trans. on Pattern Analysis and MachineIntelligence, Vol.23, No.6, pp. 681–685 (2001).

29) Y. Feng, F. Wu, X. Shao, Y. Wang, X. Zhou: “Joint 3D Face
Reconstruction and Dense Alignment with Position Map Regression Network”, Proc. of European Conference on Computer
Vision (ECCV), pp. 534–551 (2018).

15) T. F. Cootes, C. J. Taylor, D. H. Cooper, J. Graham: “Active
Shape Models-Their Training and Application”, Computer Vision and Image Understanding, Vol.61, No.1, pp. 38-59 (1995).

30) J. Zhang, L. Lin, J. Zhu, S. C. Hoi: “Weakly-Supervised MultiFace 3D Reconstruction”, In: Computing Research Repository(CoRR), Vol. abs/2101.02000 (2021).

16) Y. J. Lee, S. J. Lee, K. R. Park, J. Jo, J. Kim: “Single
View-based 3D Face Reconstruction Robust to Self-occlusion”,
EURASIP Journal on Advances in Signal Processing, Vol.2012,
No.1, pp. 1–20 (2012).

31) A. Tewari, M. Zollhofer, H. Kim, P. Garrido, F. Bernard, P.
Perez, C. Theobalt: “Mofa: Model-based Deep Convolutional
Face Autoencoder for Unsupervised Monocular Reconstruction”, Proc. of IEEE International Conference on Computer
Vision Workshops (ICCVW), pp. 1274–1283 (2017).

17) X. Zhu, Z. Lei, J. Yan, D. Yi, S. Z. Li: “High-ﬁdelity Pose andExpression Normalization for Face Recognition in the Wild”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 787–796 (2015).

32) F. Chollet: “Xception: Deep Learning with Depthwise Separable Convolutions”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1251–1258 (2017).

18) J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, M.
Nießner: Face2face: “Face2face: Real-time Face Capture andReenactment of RGB Videos”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 2387–
2395 (2016).

33) M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L. C. Chen:“Mobilenetv2: Inverted Residuals and Linear Bottlenecks”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 4510–4520 (2018).
34) A. Howard, M. Sandler, G. Chu, L. C. Chen, B. Chen, M.

19) P. Huber, G. Hu, R. Tena, P. Mortazavian, W. P. Koppen,

97

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, H.
Adam: “Searching for Mobilenetv3”, Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314–
1324 (2019).

49) A. Bulat, G. Tzimiropoulos: “How Far are We from Solving the
2D & 3D Face Alignment Problem?(and a Dataset of 230,000
3D Facial Landmarks)”, Proc. of IEEE International Conference on Computer Vision(ICCV), pp.1021–1030 (2017).

35) K. Han, Y. Wang, Q.Tian, J. Guo, C. Xu, C. Xu: “Ghostnet:
More Features from Cheap Operations”, Proc. of IEEE/CVF
Conference on Computer Vision and Pattern Recognition(CVPR), pp. 1580–1589 (2020).

50) X. Luo, P. Li, F. Chen, Q. Zhao: “Improving Large Pose Face
Alignment by Regressing 2D and 3D Landmarks Simultaneously and Visibility Reﬁnement”, Chinese Conference on Biometric Recognition, pp.349–357 (2018).

36) P. Paysan, R. Knothe, B. Amberg, S. Romdhani, T. Vetter:“A 3D Face Model for Pose and Illumination Invariant Face
Recognition”, Proc. of sixth IEEE International Conference on
Advanced Video and Signal based Surveillance, pp. 296–301(2009).

(Received September 30, 2021)(Revised February 1, 2022)
Haoqi GAO (Student Member)
She received the M.E. degree from Beihang
University of Software in 2016. From 2019
she stared a Ph.D. in system engineering at
Wakayama University. Her research interests include face detection, 3D face reconstruction, medical image processing, and deeplearning.

37) C. Cao, Y. Weng, S. Zhou, Y. Tong, K. Zhou: “Facewarehouse: A 3D Facial Expression Database for Visual Computing”, IEEE Trans. on Visualization and Computer Graphics,
Vol.20, No.3, pp. 413–425 (2013).
38) Z. H. Feng, J. Kittler, M. Awais, P. Huber, X. J. Wu: “Wing
Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2235–2245(2018).

Koichi OGAWARA

39) M. Koestinger, P. Wohlhart, P. M. Roth, H. Bischof: “Annotated facial landmarks in the wild: A Large-scale, Real-world
Database for Facial Landmark Localization”, Proc. of IEEE
International Conference on Computer Vision Workshops (ICCVW), pp. 2144-2151 (2011).

He received his Ph.D. degree in Information
and Communication Engineering from the
University of Tokyo, Japan, in 2002. From2006 to 2011, he was a guest Associate Professor at Kyusyu University. Currently, he is an
Associate Professor at Wakayama University.
His research interests include robotics, computer vision, and machine learning. He won
the Best Vision Paper Award at the IEEE International Conference on Robotics and Automation 2007.

40) C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, M. Pantic: “300
Faces in the Wild Challenge: The First Facial Landmark Localization Challenge”, Proc. of IEEE International Conference on
Computer Vision Workshops (ICCVW), pp. 397–403 (2013).
41) X. Zhu, D. Ramanan: “Face Detection, Pose Estimation, andLandmark Localization in the Wild”, Proc. of IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), pp.
2879–2886 (2012).
42) P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman, N. Kumar:“Localizing Parts of Faces using a Consensus of Exemplars”,
IEEE Trans. on Pattern Analysis and Machine Intelligence,
Vol.35, No.12, pp. 2930–2940 (2013).
43) E. Zhou, H. Fan, Z. Cao, Y. Jiang, Q. Yin: “Extensive Facial Landmark localization with Coarse to Fine Convolutional
Network Cascade”, Proc. of IEEE international conference on
computer vision workshops (ICCVW), pp. 386–391 (2013).
44) S. Zafeiriou, G. Trigeorgis, G. Chrysos, J. Deng, J. Shen: “The
Menpo Facial Landmark Localisation Challenge: A Step towards the Solution”, Proc. of IEEE Conference on Computer
Vision and Pattern Recognition Workshops (CVPRW), pp.
170–179 (2017).
45) X. P. Burgos-Artizzu, P. Perona, P. Dollr: “Robust Face Landmark Estimation under Occlusion”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 1513–1520(2013).
46) X. Cao, Y. Wei, F. Wen, J. Sun: “Face Alignment by Explicit
Shape Regression”, International Journal of Computer Vision,
Vol.107, No.2, pp.177–190 (2014).
47) X. Xiong, F. De la Torre: “Supervised Descent Method and Its
Applications to Face Alignment”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 532–
539 (2013).
48) X. Zhu, Z. Lei, X. Liu, H. Shi, S. Z. Li: “Face Alignment acrossLarge Poses: A 3D Solution”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.146–
155 (2016).

98

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）-- Special Issue on
CG & Image Processing Technologies Supporting and Expanding Human Creativities --Investigation of New Design Method Rooted in Local History and Culture throughApplication of Photogrammetry: A Case Study on Application of Maruko-bune,a Traditional Boat Unique to Lake Biwa in Japan, to Architectural Design
Akari YOSHIDA†, Toshitomo SUZUKI†† (Member), Hiroyuki TAGAWA†††Daiwa House Industry, ††Department of Architecture, Mukogawa Women’s University<Summary> This study aims to investigate new design method rooted in the history and culture of a region throughphotogrammetry and the reconstruction of shapes as an expansion of the possibilities of using photogrammetric technology, which
has advanced rapidly in recent years. Specifically, as an example of something rooted in local history and culture, we focused on
Maruko-bune, which were developed as the boats unique to Lake Biwa in Japan but are no longer built or used. We measured the
Maruko-bune using photogrammetry and constructed its 3D model. We then attempted to design architecture by decomposing andreconstructing the shape of the 3D model. As a result, we found that the photogrammetry and 3D modeling technologies havereached the point where the shape of the Maruko-bune can be measured and reproduced by photogrammetry and 3D modelingusing software and a smartphone camera that can easily be used without special calibration by architects and designers who are notexperts in photogrammetry, and its curves and surfaces can be used for architectural design.
Keywords: photogrammetry, Maruko-bune, mesh data, 3D model, architectural design

1. Introduction
The purpose of this research is to investigate a new design
method rooted in the history and culture of a region byphotogrammetry and the reconstruction of shapes as an
expansion of the possibilities of using photogrammetrictechnology, which has advanced rapidly in recent years. As an
example, we will attempt to design new architecture by using
photogrammetry and constructing the shape of a traditional
Japanese boat.
Boats have been used as a means of transportation sinceancient times and have been built with the best technology andin accordance with the culture of each era and region. Looking
at these boats, we can see that they are not only practical but
also beautiful, rooted in the history and culture of their regions.
Today’s ships and boats are built more for practicality andconstruction efficiency and less in relation to local materials
and technology. Along with this, as with much contemporary
architecture, their relevance to local history and culture has
diminished.
In this study, the Maruko-bune, which was developed as aboat peculiar to Lake Biwa in Japan but is no longer built andused, and whose exact 3D shape has never been surveyed, ismeasured by photogrammetry, and a 3D model that captures
its characteristics is constructed. Then, we try to design abuilding by disassembling and reconstructing the reproduced

99

shape. The architectural design is based on the assumptionthat a theater will be built on the shores of Lake Biwa in Shiga
Prefecture.

2. Related Works
We have been studying the use of 3D laser scanners tomeasure and reveal the shape of handmade pantiles used on ahouse built around 1900. However, photogrammetry is moresuitable for surveying larger scale objects such as ships andboats, which can be directly used to shape the appearance ofbuildings.
The technology of photogrammetry has improved inrecent years and is used in a variety of fields, including not
only engineering such as architecture and naval engineering
but also geography and archaeology. There have been manystudies on photogrammetry in the field of architecture, suchas those by Kersten, Acevedo Pardo, and Lindstaedt1), Shults2),
and Pepe and Costantino3). The application ofphotogrammetry to ships has been studied such as by Ahmed,
Jamail, and Yaakob4) and by Burdziakowski and Pawel5). The
previous studies of photogrammetry in engineering havefocused on accurately measuring and reproducing the shapeof the object.
However, although data that is too precise or too complex
can be useful when it is used to recreate a shape as it is, it ismore of a hindrance when it is used to design a new piece ofIIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

architecture or something else. Alby, Grussenmeyer, andPerrin6) pointed out through their research on photogrammetry
and 3D modeling of architectural works that compromises arenecessary between photogrammetric measurements andarchitectural representations, which seems to overlap withwhat we want to argue. In any case, we do not know of any
studies like ours where the shapes of artifacts rooted in local
history and culture are not directly reproduced and utilized as
3D models, but rather the shapes are decomposed andreconstructed to be applied to new designs that are different
from the originals. Also, we cannot find any studies that apply
the result of photogrammetry of a ship or boat rooted in local
history and culture to the design of architecture on the land,
which is different from that of a ship or boat.
Most architects and industrial designers are not experts inphotogrammetry. If the goal is to make the photogrammetricsurvey as accurate as possible and faithfully reproduce theshape, it would be better to ask a photogrammetric expert.
However, an enormous cost is likely to be incurred byconducting accurate photogrammetry on large artifacts thatcannot be easily moved from their locations, such asarchitecture and the Maruko-bune displayed indoors in thisstudy. Nevertheless, if the goal is to restore or maintain anexisting historical building, it is possible to adopt the idea thatthe top priority is to grasp its shape as accurately as possible.
However, when deconstructing and reconstructing the shapeto create a new design, photogrammetry is one piece of
information that architects and designers cite to create a new
design. Overly complex data makes it difficult to process andin fact becomes an obstacle to the use of data in design.
Therefore, rather than incurring costs by improving theaccuracy of the photogrammetry itself, it is more critical forarchitects and designers to be able to do the photogrammetry
themselves as cheaply and easily as possible.
Architects and designers use a camera frequently in their
daily design work. On the other hand, they can hardly be
assumed to have a dedicated camera for photogrammetry. For
better photogrammetric accuracy, it is certainly desirable tocalibrate the camera to reduce the effects of lens deformation,
as already pointed out in several studies2)3). Requiring a
special, complex camera calibration for photogrammetry
would mean that the architects and designers would have torevert to the settings for other works. This could easily
become an obstacle to widespread adoption ofphotogrammetry among architects and designers. The setting
process also needs to be simplified to promote the use ofphotogrammetry.
In this study, we will explore the extent to which the100

results of photogrammetry can be used in architectural design
when photogrammetry is performed using Agisoft Metashape,
which is software that can be easily used by architects anddesigners who are not experts in photogrammetry, and asmartphone camera, which is a device that they use on a daily
basis, without any special calibration.

3. Features of Maruko-bune
Maruko-bune were wooden sailing boats with uniquestructures that were the mainstay of Lake Biwa transportationfrom the beginning of the modern era to before World War II
at the latest. They were used for not only transporting goods
and people but also fishing. A small number of engineequipped Maruko-bune was still built after World War II.
Maruko-bune are unique to Lake Biwa and havecompletely different structures from typical Japanese boats
used on the open seas during the same period, such as
Kitamae-bune (Fig. 1).
A Maruko-bune has four main characteristics.
(1) The hull is longer and deeper than that of a Kitamaebune.
(2) The bottom of the boat is a half arc while a Kitamaebune had a three-story shelf structure and an inverted
triangle shape.
(3) Half logs called "omogi" are used for the port sidepanels.
(4) Many strips of steel plates are used in the bow section.
The main characteristic of the Maruko-bune is expressedin its cross section. The roundness is created by three parts:
the shiki at the bottom, the frikake at the lower side, and theomogi at the upper side. For the shiki, lumbered cedar boards
are glued together using special boat’s nails called sewing
nails to form a gentle curve. The omogi on the port side is acedar log cut in half lengthwise, with the outside face of thelog facing outward. Between the shiki and the omogi, ahorizontal piece of frikake is attached diagonally.
The angle at which the frikake is attached is an importantvalue that determines the overall balance of the boat. Mr.

Maruko-bune

Kitamae-bune

Fig. 1 Central sections of a Maruko-bune and Kitamae-bune drawn by theauthors on the basis of photographs taken in the Kita-Omi Museumof Maruko-bune

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Sanshiro Matsui, a Maruko-bune carpenter, kept this value asa board chart for over half a century7). The name "Marukobune" is said to come from the maru (roundness) of its shape,
and this roundness is best shown in the cross section. Marukobune is also said to have been developed as a special type ofJapanese boat because the closed nature of Lake Biwa
prevented the exchange of shipbuilding technology with otherregions. Because the maruta (logs) are split in two andattached to the sides, the logs appear to stick out when viewed
from the outside (Fig. 2).
Due to this external characteristic, Maruko-bune was also
called "Maruta-bune" or "Maru-bune.” Maruko-bune played
an important role in the transportation of goods, numbered
more than 1,000 at any one time until the end of the 18th
century, and ranged from large ones carrying 400 koku tosmall ones carrying 6 koku (1 koku = about 180 liters). Of
these, the Maruko-bune with a loading capacity of about
hyaku (100) koku were the most popular, and thus weresometimes called "Hyakkoku-bune."

Structure

Appearance

Fig. 2 Structure and appearance of a Maruko-bune drawn by the authors on
the basis of photographs taken in the Kita-Omi Museum of Maruko-bune

4. Photogrammetry
In the past, boat carpenters did not draw any designs whenbuilding Maruko-bune; they made the boats by measuring thedimensions of already existing Maruko-bune8). Although
there are drawings (Fig. 3) that the boat carpenters wereforced to draw, these drawings alone do not provide enoughinformation to specify the shape of the Maruko-bune andcreate its 3D model.
Therefore, we attempted to reconstruct the shape of theexisting Maruko-bune by photogrammetry. The preserved
Maruko-bune is located at the Lake Biwa Museum (Kusatsu
City, Fig. 4) and the Kita-Omi Museum of Maruko-bune(Nagahama City, Fig. 5). However, it was difficult to measure
the shape of the Maruko-bune in the Lake Biwa Museum byphotogrammetry due to its installation condition. Therefore,
we decided to conduct a photogrammetric survey of the
Maruko-bune in the Kita-Omi Museum of Maruko-bune.
The Maruko-bune (Fig. 5) in the Kita-Omi Museum of
Maruko-bune was built in 1931 and was actually used until
around 1965. It was brought to its current location in
conjunction with the construction of the museum. It has anengine room because it was equipped with an engine (Figs. 6
and 7) instead of sails.
The smartphone used for photography was Apple's iPhone
XS. As already mentioned, which is important in this research
is not so much the accuracy of the photogrammetry, but howeasily architects and designers, who are not experts inphotogrammetry, can use the photogrammetry technology on

101

Fig. 3 General structural drawings of a Maruko-bune before WorldWar II 8)

Fig. 4 Maruko-bune in the Lake Biwa Museum(Photographed in November 2019)

Fig. 5 Maruko-bune in the Kita-Omi Museum of Maruko-bune(Photographed in November 2019)IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 1 Summary of EXIF data for 105 photos used forphotogrammetry

Fig. 6 The reconstruction result of most of the port side of the Marukobune in the Kita-Omi Museum of Maruko-bune

Fig. 7

Dimensions

4032 x 3024

Device makeDevice model
F numberExposure timeApple
iPhone XS
f/1.8
1/64.4mean
maximum
minimummeanmaximum
minimum

Focal length

1/40
1/122
125
320
40
4 mm

Focal length in 35mm

26 mm

ISO speed

The reconstruction result of the part from the rudder to the
starboard side in the rearmost part of the Maruko-bune in the
Kita-Omi Museum of Maruko-bune

a widely available device. Therefore, no special calibration
was done for the photography.
Instead, we tried to take photos from as many angles as
possible in a short period of time, within the constraints of thesize of the exhibition room. Architects and designers caneasily take a lot of photos in this way. As shown in the left
side of Fig. 5, there are stairs and other obstacles on the
starboard side. The photos on the port side (opposite side of
Fig.5) were used for photogrammetry, except for the rearmost
part, because the shape of the Maruko-bune is considered tobe symmetrical. The rearmost part was photographed fromboth the port and starboard sides because of the thin rudder
protruding and the lack of obstructions on both sides. A
summary of the EXIF data of the 105 photos taken during a
6-minute period and used for the photogrammetry is given in
Table 1.
We then imported the photos into Agisoft Metashape
Professional and tried to reconstruct the boat’s shape. The
camera positions obtained as a result of the software
calculations are shown in Fig. 8, and the photogrammetricresults are shown in Figs. 6 and 7. Since there were not many
obstacles around the port side and rearmost part of the
Maruko-bune and the space was bright, the overall shape andsurface irregularities were clearly visible. However, we could
not automatically connect the part from the rudder to the
starboard side in the rearmost part (Fig. 7 and lower in Fig. 8)
with most of the port side (Fig. 6 and upper in Fig. 8) of the
Maruko-bune. This may have been influenced by the
protruding rudder.

102

Fig. 8 Camera positions obtained from photogrammetry of the
Maruko-bune in the Kita-Omi Museum of Maruko-bune

Fig. 9 Mesh data of pieced-together the Maruko-bune

5. Construction of 3D Model
The mesh data obtained in Figs. 6 and 7 were imported in
VectorWorks, a computer-aided design (CAD) software, foruse in architectural design. Next, the port side data shown in
Fig. 6 was inverted to create the starboard side data, and theport side data, starboard side data, and stern data shown in Fig.
7 were integrated (Fig. 9). We then compared this mesh datawith the drawing on display at the Kita-Omi Museum of
Maruko-bune (Fig. 10).
From Fig. 10, we can see that the front part of the meshdata is a little bulkier than the drawing, and the rear part of themesh data is slightly missing from the side view. As alreadyIIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

mentioned, the Maruko-bune were not built in accordancewith the drawings, and the shapes of the boats and thedrawings may not perfectly match. Therefore, this degree ofdifference may occur, and reproducibility is impossible toevaluate. The overall shape of the mesh data and drawing are
similar. The effect of the deformation of the camera lens canhardly be observed in Figs. 8 and 9. Therefore, it is consideredthat a certain level of accuracy can be obtained without
calibration. Although the position of the rudder at the stern of
the ship is shifted significantly, this is not a problem because
the rudder is not fixed.
The mesh data created in Fig. 9 is a polyhedron, not a
curved surface. Also, the amount of data is too large to beprocessed and used for architectural design by CAD software.
Therefore, the mesh data cannot be used for design as it is.
Therefore, we decided to create a 3D model of the curved
surface of the Maruko-bune on the basis of this mesh data.
First, we created multiple longitudinal cross sections byreferring to the mesh data and attempted to create a 3D modelusing the Loft Surface tool of VectorWorks (Fig. 11).
However, the longitudinal cross-section could not accurately
represent the omogi, a feature of the Maruko-bune.
Therefore, we then attempted to create a 3D model bycreating shortitudinal cross sections. Because the total lengthof this boat is 17.2 m, we decided to divide the length into 16
equal parts and create cross-sections every 1,075 mm. The
cross-sections were created separately for omogi, whichcharacterizes the shape of the Maruko-bune, and for the restof the main body (Fig. 12). In addition, cross sections ofomogi were added where the shape of the section changed
abruptly.
Then we created the 3D model by using the cross-section
divided into omogi and the main body. The 3D model could
not be created smoothly with the Loft Surface tool of
VectorWorks because the curved surface was twisted andunevenness was created due to the error of each cross section.
Therefore, we used the Loft tool on Rhinoceros to generatesmooth curved surfaces without torsion. Because thearchitectural design in our study was carried out using
VectorWorks, the data of the curved surface generated by
Rhinoceros was imported back into VectorWorks (Figs. 13
and 14).
Then, the omogi data and the data of the main body werecombined to complete the 3D model of the Maruko-bune (Fig.
15). In this way, we were able to create a 3D model of the
Maruko-bune on CAD. We compared this 3D model with thedrawing in Fig. 10 (Fig. 16).
In the 3D model in Fig. 16, the rear part of the boat is103

Fig. 10 Comparison of mesh data and drawings(Red: mesh data, black line: drawing)

Fig. 11 3D model created with longitudinal cross-sectional viewFig. 12 Cross-sectional lines of the main body every 1,075 mm

Fig. 13 3D model of omogi created with Rhinoceros

Fig. 14 3D model of the main body created with Rhinoceros

Fig. 15 3D model of a Maruko-bune merged in VectorWorksslightly missing from the plan and side drawings, but theshape of the model and the drawing are closer than in Fig. 10,
except for the rudder and engine room, which were omitted increating the 3D model. As already mentioned, this Marukobune was not built on the basis of drawings. However, eventaking this into account, the mesh data obtained from thephotogrammetry was replaced by a 3D model, which isconsidered to more faithfully reproduce the boat’s shape.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig.17 Diagram of the study with images of a round boat and waves
Fig. 16 Comparison of 3D model and drawings(Red: 3D model, black: drawing).

6. Application to Architectural Design
As already mentioned in the introduction, this section
assumes the design of a theater to be built on the shore of Lake
Biwa in Shiga Prefecture and examines how the shape of the
Maruko-bune obtained by photogrammetry and its 3Dmodeling can be used for its architectural design.

Study model 1

Study model 2

Fig. 18 Study models made by using aluminum foil

6.1 Investigation using study models
Prior to using the 3D model shown in Figs. 15 and 16 forthe design of the theater, study models were first created withpaper clay from the image of a Maruko-bune and waves (Fig.
17).
On the basis of this created paper clay study model (Fig.
17), we made study models with aluminum foil to express
their curves and curved surfaces (Fig. 18).
Study models 1 and 2 are composites of the decomposed
shapes of the boat. Comparing study models 1 and 2, we canfeel the shape of the boat in study model 1, but we can feel
both the image of the waves and the shape of the boat more in
study model 2. Therefore, we decided to adopt the idea ofstudy model 2 and proceed with the design. Study model 3(Fig. 19) is based on study model 2, in which more details
were made clear and the image was given shape. On the basisof study model 3, the 3D model was created in VectorWorks.
Figure 20 shows the perspective view of this 3D model fromthe eye level.
On the basis of the shape of study model 3 and its 3Dmodel, architectural design will be done by reconstructing the
3D model of the shape of the Maruko-bune (Figs. 15 and 16).
The design will make use of the curves and curved surfaces ofthe Maruko-bune (Fig. 21).
6.2 Use of curves and curved surface of the Marukobune
Since roundness is a characteristic of Maruko-bune, its
curves were incorporated. The curves of the plan as seen from

104

Fig. 19 Study model 3 with aluminum foil and wire meshFig. 20 Perspective view from eye level of the 3D model created on thebasis of study model 3.

Fig. 21 Diagram of the reconstruction of the shape of a Maruko-bune

above of the 3D models of omogi and main body wereextracted. The curves were then incorporated into the design
by enlarging and reducing them to fit the shape of 3D modelof study model 3 (Fig. 22).
Also, the Maruko-bune were the boats that had been
widened by increasing the number of shiki, thus increasing
their capacity. Taking this into account, the curved surface ofIIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 22 Plan incorporating the curve of the boat
(Blue line: omogi, red line: main body)0

5 10 15m

Fig. 24 First floor plan of the theater

Fig. 23 Curved surface of the forward part of the boat cut and opened
60° at the top

the forward part of the boat, which had been widened by theshiki, was incorporated into the shape of the fly tower, which
has a larger volume. In fact, we first cut the curved surface ofthe main body of the Maruko-bune in half. After that, the tipof the boat was opened 60° at the top, and the two halves wereconnected by another curved surface (Fig. 23).
Figures 24 through 26 shows the floor plan, elevation,section, and hand-drawn perspectives of the architecturaldesign whose preliminary design was completed utilizing the
explained design method. We were able to design anarchitecture with an impressive curved surface that inherited
the shape of the Maruko-bune.

0

Fig. 25 South elevation and X-X’ section of Fig. 24

10

20

30m

7. Conclusion
In this study, we examined architectural design methodsby photogrammetry, 3D model construction, andreconstruction of the shape of a Maruko-bune, a boat that wasdeveloped unique to Lake Biwa in Japan but is no longer built
or used today, to investigate a new design method rooted in
the history and culture of the region by the photogrammetry.
As a result, we found that the photogrammetry and 3Dmodeling technologies have reached the point where theshape of the Maruko-bune can be measured and reproducedby photogrammetry and 3D modeling using software and asmartphone camera that can be easily used without special
calibration by architects and designers who are not experts inphotogrammetry, and its curves and surfaces can be used forarchitectural design.
This design method may require some adjustments

105

Fig. 26 Hand-drawn perspectives

depending on the photogrammetric and design objects, andthe design concept. In any case, we have shown that thehistory and culture of a region like the Maruko-bune can bepassed on to the future in a new form by applying this method.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

However, we could not conduct a detailed study on theappropriateness of changing the scale of curves and surfaces
obtained by photogrammetry in this architectural design. Thisis an issue to be addressed in the future.

Akari YOSHIDA
She received her B.Arch. and M. Arch degrees
from Mukogawa Women’s University, Hyogo,
Japan, in 2020 and 2022, respectively. Since
April 2022, she has worked at Daiwa House
Industry Co., Ltd.

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Numbers JP17K00741, JP20K12520.

References
1)

T. Kersten, C. Acevedo Pardo, M. Lindstaedt: “3D Acquisition,
Modelling and Visualization of North German Castles by Digital
Architectural Photogrammetry,” Proc. of ISPRS XXth Congress. Vol.
XXXV, Commission V, Part B2, pp.126–132 (2004).

2)

R. Shults: “New opportunities of low-cost photogrammetry for Culture
Heritagepreservation,”

The

International

ArchivesofthePhotogrammetry, Remote Sensing and Spatial Information Sciences, Vol.
XLII-5/W1, No.42, pp.481–486 (2017).
3)

M. Pepe, D. Costantino: “UAV Photogrammetry and 3D Modelling ofComplex Architecture for Maintenance Purposes: the Case Study of theMasonry Bridge on the Sele River, Italy”, Periodica Polytechnica Civil
Engineering, (2020).

4)

Y. M. Ahmed, A. B. Jamail, O. Yaakob: “Boat Survey Using
Photogrammetry Method,” International Review of MechanicalEngineering (IREME), Vol.6, No.7, pp. 1643–1647 (2012).

5)

P. Burdziakowski, P. Tysiac: “Combined Close Range Photogrammetry
and Terrestrial Laser Scanning for Ship Hull Modelling,” Geosciences,
Vol.9, No.5, p.242 (2019).

6)

E. Alby, P. Grussenmeyer, J. Perrin: “Visualization of Architectural
Works by Photogrammetry: a Compromise Between Measurement andRepresentation,” Workshop on Vision Techniques Applied to the
Rehabilitation of City Centres, 2004, Lisbonne, Portugal. pp.1–11 (2004).

7) K. Makino: “Marukobune,” the traditional sailing boats in the Lake Biwa
region, Japan., p.77, Yuzankaku (2013).
8) M.

Yoda:

“Maruko-bune,”

Shiga

Prefectural

AzuchiCastleArchaeological Museum, Nagahama Castle Historical Museum, eds.:
Biwako no Fune ga Musubu Kizuna ―Maruki-bune, Maruko-bune kara
“Uminoko” made ― [Bonds Connected by Lake Biwa's Ships andBoats: From Maruki-bune and Maruko-bune to “Uminoko”], pp.47–49,
Sunrise Publishing (2012). (in Japanese)(Received Nov. 26, 2021)(Revised Jan. 31, 2022)

106

Toshitomo SUZUKI(Member)
He received his B.Eng., M.Eng. and Dr. Eng.
degrees from Kyoto University, Japan, in 1998,
2000 and 2003, respectively. He had worked at
the Architectural Research Association, Kyoto,
Japan, in 2003. He had been a Lecturer from 2004
to 2011, an Associate Professor from 2012 to2017, and has been a Professor since 2018, of the
Department of Architecture in Mukogawa
Women’s University, Japan. His researchinterests focus on architectural planning anddesign, and design science. He is a member of
AIJ, IEICE, IIEEJ, IPSJ and JSPE.

Hiroyuki TAGAWA
He received his M.S. degree and Ph.D. degree in
Structural Engineering & Mechanics from
University of Washington, Seattle, USA, in 2000
and 2005, respectively. He had been an associate
professor from 2014 to 2020, and has been aprofessor since 2021, of the Department ofArchitecture in Mukogawa Women’s University,
Japan. His research interests include architecturaland structural design, structural analysis
including finite element analysis, structuralmorphogenesis. He is a member of AIJ, IASS and
JSCES.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper
Generative Image Quality Improvement in Omnidirectional Free-Viewpoint
Images and Assessments
Qiaoge LI† , Oto TAKEUCHI† , Hidehiko SHISHIDO† , Yoshinari KAMEDA† (Member),
Hansung KIM†† , Itaru KITAHARA† (Member)† University of Tsukuba , †† University of Southampton<Summary> This paper proposes a method that improves the quality of omnidirectional free-viewpoint
images by generative adversarial networks. Omnidirectional images are a popular way of obtaining threedimensional (3D) visual information, while free-viewpoint images are essential to Virtual Reality (VR) and
Mixed Reality (MR) applications. Therefore, we generated free-viewpoint images with 3D information
estimated by the captured omnidirectional images. The quality of the generated images is deteriorated by
the 3D reconstruction error due to occlusion and miss-correspondences. In this work, we proposed a methodthat uses Generative Adversarial Networks (GAN) to solve this problem. We focused on the structural
information of various perspectives and applied a “divide and conquer” approach by separating the images
into perspectives before training and recombining them at a later stage. At the same time, we conducted
a comprehensive, multi-faceted evaluation of the proposed method to verify its eﬀectiveness in improving
image quality. Based on the actual information distribution in the equirectangular images, we analyze theadaptability of diﬀerent image quality evaluation methods. After careful assessment, we consider that the
proposed method can generate highly accurate, omnidirectional free-viewpoint images.
Keywords: visual reconstruction, free-viewpoint image, omnidirectional image, image quality improvement,
generative adversarial networks

1.

Introduction

Photography with omnidirectional (360 degrees) cameras is an eﬀective technique for observing the surrounding environment. This technique has received more attention in recent years due to its ability to achieve immersive
viewing combined with head-mounted displays. Google
Street View achieves multiple-directional observations byswitching the viewpoints as instructed and appropriately
selecting the omnidirectional images taken from multiple
perspectives.
By applying a three-dimensional (3D) estimation process such as Structure from Motion (SfM) to omnidirectional multi-view images, the position and rotation of
the omnidirectional camera and the 3D shape of the target space can be estimated. We previously proposed a
generation method of bullet-time video using omnidirectional cameras, which used estimated 3D information toswitch viewpoints freely while focusing on a point in thecaptured space1). In this method, omnidirectional observation is only available at the captured viewpoints, not107

at the non-captured positions. The smoothness of the
viewpoint motion decreases when the intervals widen between multi-viewpoint cameras. Another signiﬁcant issue
is that the viewpoints are entirely stationary and cannot
be moved from the capturing positions.
Free-viewpoint image generation for reproducing theappearance of any perspective has been greatly researched in computer vision over the past twenty
years2)–14). Nonetheless, this ﬁeld still has many unsolved
or partially solved issues. For example, image quality
is reduced by 3D reconstruction errors caused by correspondence search errors and occlusion artifacts. Suchquality reduction remains an important research issue.
Even though the accuracy of 3D reconstruction can be improved using devices that obtain depth information, such
as RGB-D cameras4),10),11), the simplicity of the capturing system is reduced, causing complications for practical applications. We proposed a solution to this problem
using an omnidirectional camera. In multiple omnidirectional images, there are many overlapping regions due tothe wide ﬁeld of view. Hence, the same areas of the 3D

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 1 Overview of image-quality improvement in OFV images

space are viewed from diﬀerent viewpoints, thus improving the accuracy of the correspondence search. Furthermore, the omnidirectional image also has high adaptivity
to the Virtual Reality (VR) content creation for the immersive experience.
Barnes et al.15)proposed a method to restore the deteriorated image quality by using image reconstruction
techniques in 2009. In recent years, deep learning approaches have also been presented16),17)for more natural
image quality improvements. However, these algorithms
assume that the shape, size, and location of the areathat needs to be restored are known. In our case, it isquite diﬃcult to recognize regions with low quality forfree-viewpoint image generation, which depends on thecapturing conditions. This situation complicates applying conventional image reconstruction techniques to solve
the problem of image quality deterioration for generated
free-viewpoint images.
Figure 1 shows the overview of image-quality improvement in OFV images. This paper makes two main contributions. The ﬁrst proposes a generation method ofhigh-quality Omnidirectional Free-Viewpoint (OFV) image with deep learning. OFV images are generated from
3D information and camera parameters obtained by SfM
and the captured omnidirectional images. Subsequently,

108

Generative Adversarial Networks (GAN) are adopted toeliminate the artifacts and the missing regions within the
OFV images. Using this approach, the image quality of
OFV images is improved. The GAN performance is signiﬁcantly improved by image division due to the reduced
appearance diversity and the greatly reduced number ofstructural features to be learned per model. The second contribution is image quality veriﬁcation by several
assessment metrics; conventional metrics which are commonly used for such task, and novel metrics, which have ahigh correspondence to human perception. Based on ourprevious work18), we conducted additional experiments in
diﬀerent scenes using more suitable evaluation metrics tovalidate our method comprehensively.

2.
2.1

Related Works

Representation of omnidirectional multiviewpoint images
In Google Street View (Google, 2007), omnidirectional
images enable observation of surrounding scenery from anarbitrary perspective. By switching the images of adjacent viewpoints speciﬁed by viewers, one can get a moredetailed overview of the situation while observing the surrounding scenery. An image blending process and image
shape transformation enable viewers to experience a feeling of moving through the scene. Many previous types

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）of research also used omnidirectional images for the lo-

distortion and generate an acceptable result, the number

calization of cameras and representation of the capturedof cameras must be increased.

19)

scene

. On this basis, we estimated the 3D shape of thecaptured space, the position, and the rotation of omnidirectional cameras by processing the 3D reconstructionof the captured scene using the multi-viewpoint omnidirectional images. We developed the bullet-time video
generation method using the estimated 3D information
to allow both the viewpoint and the perspective to be
switched while focusing on a point in the captured space1). Unfortunately, the omnidirectional image switching
method limits the viewer to the capturing positions but
not beyond them. Bertel et. al.20)brought out a technique to acquire omnidirectional free-viewpoint images
with 6-degree-of-freedom(6-DoF). However, the images
generated by their approach do not move eﬀectively inthe vertical direction. Further, their shooting techniqueseverely limits the range of free-moving viewpoints thatthe images can present, resulting in a relatively small
range that can be observed.

2.3

Image quality improvement

Research on image quality improvement has been actively conducted for about ten years15)–17),27). One way
to improve the appearance of an image is by ﬁnding corresponding image information using the continuity of theimage itself15). Even though this technique has also complemented free-viewpoint videos27), it cannot reconstructunobserved information in images. Various methods havebeen proposed to reconstruct information not contained
in images using convolutional neural networks and GANs.
However, all of these methods assume that no informationis contained within the missing regions and the shapes,
sizes, and positions of these regions are known16),17). We
proposed a method that reproduces an equivalent appearance to the captured image by compensating for the deterioration of the image quality due to viewpoint movement
by applying reconstruction using GAN28)
2.4

2.2

Free-viewpoint images

There has been a vast amount of research on
free-viewpoint images.
Model-Based Rendering2),4),6)–8),11)(MBR)
employed 3D Computer Graphics(CG) models, which are reconstructed from multi-view
images of the captured scenes to reproduce views
from arbitrary perspectives. Image-Based Rendering(IBR)3),5),9),10) directly synthesizes appearances from
captured multi-view images.
These methods havealso been applied in the most recent researches that
proposed methods for synthesizing multi-viewpoint 360
images21)–26).
In MBR, the quality of the generated free-viewpoint
images depends on the accuracy of the reconstructed 3D
CG model. For this reason, errors can be signiﬁcant when
performing a 3D reconstruction for a complex scene, andartifacts may appear in the generated view. In addition,
when using multiple cameras for observation, the inherent
occlusion makes it challenging to reconstruct accurate 3D
shapes and reduces the quality of the generated images8).
Since IBR does not explicitly reconstruct a 3D shape
but instead applies a simple shape, free-viewpoint images
can be generated without considering the complexity of
the captured space. However, when the captured space’s
shape signiﬁcantly diﬀers from the actual shape, the appearance of the generated view can be distorted considerably by excessive image ﬁtting errors. To minimize this109

Omnidirectional image quality improvement
The combination of omnidirectional photography andimage quality improvement is still a new emerging region in computer vision. To the best of our knowledge,
there is currently no direct application of image quality
improvement methods for panoramas. However, methods that have a similar eﬀect, such as super-resolution
and image inpainting, have been applied to equirectangular omnidirectional images with satisfying results29)–32). However, the super-resolution method cannot solve theproblem of missing information, while image inpainting
requires prior knowledge for the missing regions. Since
the missing information of free-viewpoint images is not
regularly distributed. The existing method cannot increase the quality of OFV images. We proposed a methodthat reproduces an equivalent appearance to the captured
images by compensating for the deterioration of the image quality due to the viewpoint movement by applying
reconstruction using GAN. Our approach ﬁlls the gaps inaddressing this particular task.
2.5 Image quality assessments
When assessing the quality of images, Peak Signal-toNoise Ratio (PSNR), Root Mean Squared Error(RMSE),
and Structural Similarity Index Measure (SSIM) are themost common evaluation metrics33). However, the evaluation results obtained by these approaches do not fully
match our subjective visual perception. PSNR relies on

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）(a)before interpolation(b) after interpolation

Fig. 2 Omnidirectional depth images

comparing the L1 distance of each pixel of two images tocount the overall mean square error, which is insensitive
to the overall structural diﬀerences between the two images. RMSE has a similar disadvantage to PSNR. SSIM
incorporates the statistics of geometric diﬀerences, but
its results are still aﬀected by the local perceptual ﬁeld
size, and the overall quality assessment is not stable. In
addition, geometric diﬀerences, luminance, and contrast
are given the same weight in the default algorithm, which
does not necessarily match the human perceptual quality. To solve this problem, researchers have proposed a
deep feature-based picture quality evaluation criterion.
One of the most commonly used is Learned Perceptual
Image Patch Similarity (LPIPS)34)and Fréchet Inception
Distance (FID)35). FID scores can represent the performance of GAN to a certain extent. LPIPS focuses on
measuring the similarity of images using deep-level features that assess their quality similarly to human visual
perception. Both metrics have been well tested and compared and signiﬁcantly correlate with the similarity of
pictures that can be perceived by humans.

3.

Generation Method for Omnidirectional
Free-Viewpoint Image

3.1

Image capturing and 3D estimation

Multiple omnidirectional images are acquired at diﬀerent viewpoints around a target space. As a result of continuous research of 3D information estimation for multiviewpoint images, some excellent SfM libraries36)–39)have
been developed. These libraries, however, are usuallybased on perspective projection images for incremental
SfM, which is diﬀerent from the projection geometry of
the omnidirectional images. Therefore, in our approach,each omnidirectional image is divided into multiple perspective images. We used perspective geometry to virtually set up cameras with partial overlap between diﬀerent
perspective images at identical positions and applied the110

SfM library to each perspective projection image generated from omnidirectional images. Thus, the camera parameters of the image and the sparse 3D point cloud are
estimated. The position and orientation of each omnidirectional camera can be calculated from the estimatedcamera parameters of the corresponding virtual cameras1). Based on the parameters and the sparse 3D point cloud,multi-view stereo processing40)obtained a dense 3D point
cloud.
3.2

Generation of omnidirectional depth image
We obtained sparse omnidirectional depth images(Fig. 2(a)) by calculating the distance from each viewpoint of the omnidirectional camera to the 3D point cloud
estimated in section 4.1. We calculated the color diﬀerences in the CIELAB color space between the projected
3D point cloud and the pixels of the captured image atthe viewpoint where the depth information is generated.
This color diﬀerence increases when the 3D informationof the point cloud is estimated incorrectly. To reduce
the amount of incorrectly estimated 3D information, athreshold of 20 was set on the color diﬀerences. The
depth value is not added to the calculation when the color
diﬀerence exceeds the threshold.
Due to our inability to estimate the depth values of
the unprojected pixels of the 3D point cloud (Fig. 2(a)),
many regions are missing in the depth image. These regions were interpolated using a cross-bilateral ﬁlter41),
which reﬁnes one of the images based on another image
that has less observation noise. In our example, the captured color image is used to ﬁlter the depth image. The
ﬁltering equation is as follows:
Dp =Σr∈N d(p, r)c(Ip , Ir )Dr,Σr∈N d(p, r)c(Ip , Ir )d(p, r) = exp[−(p − r)2
],
2σ12

(1)

(2)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

(a)

(b)

(c)

Fig. 3 Generation method of OFV image; (a) virtual viewpoint and two nearest omnidirectional cameras forgenerating OFV images, (b) projection of each pixel value (color information) of acquired omnidirectional camera with reference to depth information, and (c) process of reprojecting color information in
space onto omnidirectional image sphere to generate omnidirectional OFV imagec(Ip , Ir ) = exp[−(Ip − Ir )2
],
2σ22

(3)

where p represents the pixel on the RGB equirectangular image at the same location as the invalid pixel fromthe point cloud, r is the adjacent reference pixels of p.
D is the depth value, N is the set of reference pixel coordinates, I is the luminance value, σ1 and σ2 are constants. d(p, r) is the spatial similarity, which represents
the spatial weight, and c(Ip , Ir ) represents the weight ofcolor similarity. The invalid depth values are calculated
by weighting the spatial distance and the color similarity between the reprojected point cloud and the captured
images. Thus, as shown in Fig. 2(b), the depth map canbe interpolated while maintaining the captured image’s
contours.
3.3

Generation of
viewpoint image

omnidirectionalfree-

We can determine the relative positions of the cameras used for shooting in the scene and deﬁne them as
actual viewpoints. Next, we specify the location of the
viewpoint for which we want to generate the OFV andcalculate the distance from this novel viewpoint to all111

the actual viewpoints, selecting the spots representing
the two closest cameras for the next step of the synthesis (Fig. 3(a)). The color information contained in eachpixel of the omnidirectional images captured at the twopoints mentioned above is projected into 3D space to generate a dense 3D point cloud model (Fig. 3(b)). The OFV
image is generated by projecting these 3D point clouds
back onto the omnidirectional image sphere of the newviewpoint. When diﬀerent points are projected on thesame pixel of the free-viewpoint image, the point closer
to that viewpoint is adopted to eliminate the hidden surface (Fig. 3(c)). With this method, we can generate
free-viewpoint images at any point around where multiple omnidirectional images were captured.
However, signiﬁcant artifacts and blurring can be observed in the generated OFV images. Causes of these
artifacts include errors in 3D estimation and missing 3D
information due to occlusions in the captured space. The
quality of generated OVF needs to be improved.

4.

Image Quality Improvement

This section describes how to reduce artifacts and blurring within the OVF images using GAN. In this re-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

(a)

(b)

Fig. 4 Example of cube mapping division on omnidirectional image; (a) one of the captured omnidirectional
equirectangular images and (b) cube mapping images after image division

search, we adopted Pix2Pix28)as a way to implement a
GAN. Pix2Pix is a conditional GAN that learns the correspondence between two diﬀerent styles of images, suchas sketches and photographs or aerial photographs andmaps, and then converts one to the other. In this research, Pix2Pix is applied to image conversion between
free-viewpoint images and captured images to improve
the quality of the former.
Pix2Pix consists of two networks: an image generator
and a discriminator. A prepared pair of pre-conversion
and post-conversion images is used as training data. The
pre-conversion image is input to the image generator. The
generated images or the prepared post-conversion images
are input to the discriminator, which decides which image
is input. In the learning procedure, the image generator
tries to deceive the discriminator, while the discriminator tries to make accurate judgments. In addition to theconditional GAN (cGAN) and the Least Absolute Deviations (L1) loss functions28), that are used in Pix2Pix,
the perceptual loss function is used to calculate the total
loss function42)to learn the features of perception and to
make the result closer to the ground truth on the structural similarity. Thus, the following functions represent
the objective generator:
G = arg min max(LcGAN + λL1 + ηLpercep ),GD

L1 = Exi G(xi , z) − yi ,

(5)
(6)

the activation map of our model that is used to extract
the deep structural features and calculate the perceptual
loss function, here we used a pre-trained 19-layer VGG
network43), where relu22 represents the features obtainedby the 2nd convolution before the 2nd max-pooling layer.
λ and η are coeﬃcients that balance the diﬀerent loss
terms. After the entire learning process is complete, theOFV images are processed using the image generator.
As training data, the OFV image with a synthetic sizeat the viewpoint captured in section 4.3 is used as the pretransformation image, while the omnidirectional image
captured in section 4.1 is used as the post-transformation
image. After training the image generator with the training data, the OFV image at the virtual viewpoint is input
to the learned image generator to generate highly realistic
images with less degradation of image quality.
We focus on learning the projection geometry of theimages to achieve learning eﬃciency. However, omnidirectional images based on equirectangular projections canlead to signiﬁcant changes in appearance based on theirprojection characteristics leading to shifts in viewpoints,

Fig. 4, the cube mapping method is used to divide theomnidirectional equirectangular image into six perspec-

Lpercep = Exi V GG19relu22 (G(xi , z))−V GG19relu22 (yi ),the input image and the corresponding ground truth, respectively. z is the arbitrary noise, and V GG19relu22 iswhich in turn leads to increased diversity in appearance
among the learned samples and more complicated learning. Therefore, by dividing the omnidirectional equirectangular image into multi-view projection images, the effect of projection geometry on appearance diversity canbe reduced, and then GAN learning for image contents
can be well implemented. In this paper, as shown in(4)

LcGAN = Exi [logD(xi , yi )]+Exi ,z [log(1 − D(xi , G(xi , z))],D is the discriminator, G is the generator, xi and yi are(7)

where LcGAN represents cGAN loss function, L1 is theL1 loss function, Lpercep is the perceptual loss function,
112

tives. The image generators are constructed using theperspective projection images on each plane separately.
The OFV images generated in section 4.3 are divided by

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

using cube mapping, sequentially processed by the gener-

space but with diﬀerent camera alignment and layouts. A

ators that are trained in all six directions and recombined

ﬂow chart of our complete experiment at the ﬁrst scene

to improve the quality.

is shown in Fig. 5. Figure 6 shows the layout of our5.

Omnidirectional Image Quality Assessment

When assessing the quality of images, Peak Signal-toNoise Ratio (PSNR), Root Mean Squared Error (RMSE)
and Structural Similarity Index Measure (SSIM) are themost common evaluation metrics. Previous work also
pointed out that these metrics allow objective evaluationof image quality for omnidirectional images33). However,assessments on omnidirectional images is quite diﬀerent
from perspective ones. One of main reasons is the differences of information distribution between the image
formats. In equirectangular image, information are projected unevenly along the vertical axis, since its original
source of information is a sphere, the projected information distributed more densly in the equator than polar areas. This unevenness makes the direct applicationof traditional evaluation metrics (PSNR, RMSE, SSIM)on equirectangular images less rigorous. In contrast, theuse of criteria that are more relevant to human perception (FID, LPIPS) can more eﬀectively reﬂect the humaneye’s evaluation of image quality. Therefore, in our experiments, all four evaluation criteria mentioned above
were applied. Furthermore, the structure of the equirectangular image is distorted compared to the real scene it
presents, and this distortion may make the quality assessment based on the panoramic image not necessarily
consistent with the quality of the image observed in realapplications (e.g. VR, AR). Therefore, we perform two
evaluation processes for each set of images, one directlybased on the equirectangular images, and the other wecalculate the average of the evaluation results obtainedby applying all criteria to the perspective views in 6 directions (up, down, left, right, front and back) as a supplement to provide a quality evaluation that is informative
for real applications.

6.
6.1

Experiments

Experiment environment

We conducted a comprehensive experiment to evaluate
the impact of deep learning on the quality of OFV images and image division on learning eﬃciency. We tookomnidirectional shots at three indoors scenes (Universityof Tsukuba) to validate our proposed method. Noticethat the second and third scenes are captured in the same113

cameras in the experiment of the ﬁrst scene, note thatthe cameras are placed by hand, therefore the distance
between every 2 cameras may not be the same. We captured omnidirectional images at 54 viewpoints as shownand divided them into two parts: part 1 (30 images) for
generating the OFV images of the capturing position ofpart 2 (24 images), and images of part 2 as the groundtruth of the generated OFV images. Then images inpart 1 were used to generate dense 3D point cloud using
open-source software: VisualSFM. Simultaneously, thosecameras’ position and orientation were obtained from thebundle adjustment ﬁle from the SFM results. To acquire
the extrinsic camera parameters of part 2, we ﬁxed thecamera position in part 1 and added images of part 2 tothe incremental SFM process. Notice that images in part
2 are only added to SFM for obtaining the camera parameters but do not contribute to the 3D point cloud of
the captured scene. The 3D point cloud is projected onto
each omnidirectional image sphere to generate a sparse
depth image based on the estimated camera parameters.
A dense omnidirectional depth image is then generated
at each viewpoint by interpolating the gaps between the
projected points. With the captured omnidirectional image as texture and integrating it with the corresponding
depth image, an omnidirectional image at any viewpoint
can be synthesized. Thus, a dataset of actual captured
omnidirectional images and synthesized ones at the same
viewpoints is obtained. The following are the PC speciﬁcations for the data processing: CPU AMD Threadripper
3960X, RAM: 128GB, and GPU: NVIDIA RTX 2080ti.
Then we conducted the experiments on the image quality improvement on the generated OFV images. We
planned to use Pix2Pix for a style transfer between thegenerated OFV and the captured images. However, due
to the limitations of our GPU’s VRAM, it was impossible
to train a model with images at the original resolution.
Therefore, bilinear downsampling was performed while
generating the cube mapping images. The resolution ofeach perspective projection image was 512 × 512 pixels,
and the resolution of each recombined equirectangularimage was 2048 × 1024 pixels. We aligned the captured
images of part 1 with the same pose as the OFV images,
split them by cube mapping and considered them true
images. Then we generated OFV images at the same positions of the images in part 1 and split them by cubeIIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 5 Flow chart of our full experiment at Scene 1

mapping. These are the fake images for the discriminator to classify. Pix2Pix was employed to realize the styletransfer between the generated OFV images and the captured ones on both the cube mapping images and therecombined equirectangular ones. Our approach for generating OFV is based on an assumption that the shooting environment is static, which means there shouldn’t beanything in the scene change during the capturing time.
However, the tripod is inevitable for capturing omnidirectional images. Also the segment of the cube mapped
images that contain the tripod are not our main focus forgenerating the free-viewpoint images. Hence, we didn’t
improve their quality, just kept them as the originallygenerated. We set the learning steps at 1000 epochs, according to our experience for the satisfying results. The
OFV images were subsequently processed by the genera-

114

Fig. 6 Layout of the shooting points during experiments

tor of both approaches to obtain their results.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

6.2

maps in three experimental scenarios, respectively.

Results and assessments

Figure 7 shows the error maps of each generated OFV

Figure 9 provides a comparison of the results holis-

image results relative to the ground truth, where thetically for all three experiments. Figure 10 provides a

black area indicates no diﬀerence between the image tobe measured and the ground truth. The larger the proportion of black areas is, the closer the image is to theground truth. We grayscaled the error map and increased
the exposure rate to facilitate observation. Figure 7(a)shows the comparison result of the original OFV image.
Figure 7(b) and (c) represent the comparison result of
the OFV images with image quality improvement. The
training images for Fig. 7(b) are equirectangular ones,
while the images used in Fig. 7(c) are segmented withdetailed visual comparison of the image results. Figure10(a) presents the generated OFV images without image
quality improvement, Fig. 10(b) and (c) show the OFV
images with improved image quality, and Fig. 10(d) is
the ground truth. When comparing Fig. 10(a) with Fig.
10(b), the overall picture noise is reduced compared to thegenerated OFV images, even though the improvement of
the image details is small. When we compare Fig. 10(a)with Fig. 10(c), most artifacts and overshadowing areas
were corrected, and the image quality improved signiﬁ-

cube mapping. The black area in Fig. 7(c) has the highest proportion from these three ﬁgures, indicating thatthe images processed by our proposed method with image division are the closest to the actual image data.
Figure 8 shows several examples of the curve of loss
with increasing epochs during our picture quality improvement training using GAN. The Lvgg in the ﬁgures
represents the Lpercep in equation 4.7. Figure 8 (a),(b), (c) represent the training curves using cube mapping

cantly.
We evaluated the generated results quantitatively with
PSNR, SSIM, RMSE, LPIPS, and FID, respectively, fordiﬀerent scenes. When assessing the quality of cube mapping results, we calculated the average score of 6 perspectives (up, down, front, back, left, and right) with diﬀerent
evaluation metrics.
In Table 1, the image division method resulted in better quality than the original image in all scenes and metrics. However, the method without image division does
not perform consistently. In some cases, it performs evenbetter than the method with image division, while insome cases, the results are worse than original images.
The results clarify that image division is necessary whenusing the GAN method to improve the quality of the results consistently.

(a)

7.
(b)

One of the applications of our research is to generate
free-viewpoint tour videos for museums or world heritage
sites by walking through the scene with an omnidirectional camera in hand. Many of these environments are(c)Fig. 7 Error map of OFV results; (a) the original OFV
image, and (b), (c): OFV images with two diﬀerent quality improvement approaches(a)Discussion and Future Work

(b)

(c)

Fig. 8 Curve of loss during training; (a), (b), (c): the loss curves when training with cube map images
captured in the three experiment scenes respectively

115

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Fig. 9 Results of generated OFV images and image quality improvement with proposed method in diﬀerent
scenes(a) Original OFV images(b) Quality improvement(equirectangular)(c) Quality improvemnet(cube mapping)(d) Ground truth

Fig. 10 Detail comparison for OFV images

complex and have confounding factors. Our image quality improvement approach could eliminate such kinds ofartifacts. The limitations of our system are the processing time and overall complexity. For the step of 3D estimation, the processing time increases exponentially with
the number of images. Also, training separate models fordiﬀerent perspectives also costs plenty of time. For example, in our experiment, the training process cost about 6hours for the model of each perspective. In future work,
we plan to optimize the current procedure and the tools
we use and generate high-quality omnidirectional images
in real-time.

8.

Conclusions

We proposed a generation method of high-quality omnidirectional free-viewpoint images. We used the omnidirectional images captured by a camera to estimate the 3D
information of a captured scene (i.e., dense point cloud)and camera parameters, which then generated a depth

116

image at any given position. Subsequently, the depth information was back-projected by several captured images
to obtain an OFV image at that viewpoint. We employed
the deep learning method to eliminate the artifacts andthe missing regions within the OFV image (i.e., GAN).
The performance of GAN was signiﬁcantly improved byimage division due to the reduced diversity of appearance
and the much-reduced number of structural features to belearned per model. We veriﬁed the improvement of our
proposed method by several assessment metrics, including both conventional and novel metrics that have a highcorrespondence to human perception.
This work references the article “Image-quality Improvement of Omnidirectional Free-viewpoint Images byGenerative Adversarial Networks” published in VISAPP2020. Additional experiments were conducted bychanging the shooting scenes and devices.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
Table 1 Quantitative comparison on images of diﬀerent
scenes (“Equirectangular” and “Cube Mapping” labels: the format of images for evaluation)
Methods
EquirectangularOriginal
GAN without
image division
GAN with
Image division
Cube Mapping
Original
GAN without
image division
GAN with
image division

PSNR↑

RMSE↓

SSIM↑
Scene 1LPIPS↓

FID↓

19.87020.19620.88370.179161.5231

20.7632

0.16700.83320.168255.9696

19.8974

0.17320.88940.140650.5382

24.2378

0.11510.88170.1761113.4358

25.3958

0.11060.87830.1812114.9271

25.5883

0.11040.89010.106275.0969

Video Display in a Large-Scale Space”, Proc. of IEEE Virtual
Reality, Vol. 2003-Janua, No. 2, pp. 45–52 (2003).
3) W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, L. McMillan: “Image-Based Visual Hulls”, Proc. of the 27th Annual
Conference on Computer Graphics and Interactive Techniques- SIGGRAPH ’00, pp.369–374. (2000).
4) R. A. Newcombe, A. Fitzgibbon, S. Izadi, O. Hilliges, D.
Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton:“KinectFusion: Real-Time Dense Surface Mapping and Tracking”, Proc. of 2011 10th IEEE International Symposium on
Mixed and Augmented Reality, pp.127–136 (2011).
5) S. M. Seitz, C. R. Dyer: “View Morphing”, Proc. of the
23rd Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH ’96, pp.21–30 (1996).
6) T. Kanade, P. Rander, P. J. Narayanan: “Virtualized Reality:
Constructing Virtual Worlds from Real Scenes”, IEEE Multimedia, Vol. 4, No. 1, pp. 34–47 (1997)

Scene 2EquirectangularOriginal
GAN without
image division
GAN with
Image division
Cube Mapping
Original
GAN without
image division
GAN with
image division

27.38380.08170.91940.087744.462428.0804

0.0754

0.94290.114042.255428.65690.07070.92670.077938.6189

29.37400.06090.90770.105836.51130.47430.55830.93230.138749.219431.192

0.53370.93590.071032.8505

7) S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S.
M. Seitz, R. Szeliski: “Building Rome in a day”, Communications of the ACM, Vol. 54, No. 10, pp. 105–112 (2011).
8) T. Shin, N. Kasuya, I. Kitahara, Y. Kameda, Y. Ohta:“A Comparison between Two 3D Free-Viewpoint Generation
Methods Player-Billboard and 3D Reconstruction”, Proc. of
3DTV-CON 2010: The True Vision - Capture, Transmission
and Display of 3D Video, pp. 3–6 (2010).

Scene 3EquirectangularOriginal
GAN without
image division
GAN with
image division
Cube Mapping
Original
GAN without
image division
GAN with
image division

21.4703

0.16570.88910.137966.720924.99850.11030.89520.155137.2136

22.5691

0.14380.89190.120764.5273

28.8787

0.06730.89150.104151.6350

30.2735

0.0666

0.92330.131282.707829.35820.06070.91600.061046.19749) M. Tanimoto: “FTV: Free-Viewpoint Television”, Signal Processing: Image Communication, Vol. 27, No. 6, pp. 555–570(2012).
10) P. Hedman, T. Ritschel, G. Drettakis, G. Brostow: “Scalable
Inside-Out Image-Based Rendering”, ACM Trans. on Graphics, Vol. 35, No. 6, pp. 1–11 (2016).
11) S. Orts-Escolano, C. Rhemann, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson, S. Khamis, M. Dou,
V. Tankovich, C. Loop, Q. Cai, P. A. Chou, S. Mennicken,
J. Valentin, V. Pradeep, S. Wang, S. B. Kang, P. Kohli, Y.
Lutchyn, C. Keskin, S. Izadi.: “Holoportation”, Proc. of the
29th Annual Symposium on User Interface Software and Technology, pp. 741–754 (2016).

Acknowledgements
This work was partially supported by JSPS KAKENHI
Grant Number 17H01772 and by JST CREST Grant
Number JPMJCR14E2, JapanQL and OT designed this system with support fromIK, HS, YK, and HK. QL summarized and improved theprevious OT research results, improved the loss functionof GAN by adding a loss term to the perception, andperformed a comprehensive validation and evaluation of
the whole system. QL also added evaluation criteria andwrote the paper. KI and OT participated in its review
and revision.

References
1) O. Takeuchi, H. Shishido, Y. Kameda, H. Kim, I. Kitahara:“Generation Method for Immersive Bullet-Time Video Using
an Omnidirectional Camera in VR Platform”, Proc. of the 2018
Workshop on Audio-Visual Scene Understanding for Immersive Multimedia, Co-located with MM 2018( AVSU 2018), pp.
19–26 (2018).
2) I. Kitahara, Y. Ohta: “Scalable 3D Representation for 3D

11712) N. Inamoto, H. Saito: “Free Viewpoint Video Synthesis andPresentation from Multiple Sporting Videos”, Electronics andCommunications in Japan, Part III: Fundamental Electronic
Science (English translation of Denshi Tsushin Gakkai Ronbunshi), Vol. 90, No. 2, pp. 40–49 (2007).
13) O. Grau, A. Hilton, J. Kilner, G. Miller, T. Sargeant, J. Starck:“A Free-Viewpoint Video System for Visualization of Sport
Scenes”, SMPTE Motion Imaging Journal, Vol. 116, No. 5–6,
pp. 213–219 (2007).
14) M. P. Tehrani, T. Tezuka, K. Suzuki, K. Takahashi, T. Fujii:“Free-Viewpoint Image Synthesis Using Superpixel Segmentation”, APSIPA Trans. on Signal and Information Processing,Vol. 6, No. 2017, pp.1–12 (2017).
15) C. Barnes, E. Shechtman, A. Finkelstein, D. B. Goldman:“PatchMatch: A Randomized Correspondence Algorithm for
Structural Image Editing”, ACM Trans. on Graphics, Vol. 28,
No. 3, Article No. 24, pp.1–11 (2009).
16) D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, A. A. Efros:“Context Encoders: Feature Learning by Inpainting”, Proc.
of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2016-Decem, pp. 2536–2544(2016).
17) S. Iizuka, E. Simo-Serra, H. Ishikawa: “Globally and Locally
Consistent Image Completion”, ACM Trans. on Graphics, Vol.
36, No. 4, Article No. 107, pp.1–14 (2017).

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
18) O. Takeuchi, H. Shishido, Y. Kameda, H. Kim, I. Kitahara: “Image-Quality Improvement of Omnidirectional FreeViewpoint Images by Generative Adversarial Networks”, Proc.
of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020), Vol. 4, pp. 299–306 (2020).
19) D. Caruso, J. Engel, D. Cremers: “Large-Scale Direct SLAM
for Omnidirectional Cameras”, Proc. of IEEE International
Conference on Intelligent Robots and Systems, Vol. 2015Decem, pp. 141–148 (2015).
20) T. Bertel, M. Yuan, R. Lindroos, and C. Richardt: “OmniPhotos: Casual 360 VR Photography with Motion Parallax”, ACM Trans. on Graphics, Vol. 39, No. 6, Article No.266,
pp.1–12 (2020).
21) M. Broxton, J. Flynn, R. Overbeck, D. Erickson, P. Hedman,
M. Duvall, J. Dourgarian, J. Busch, M. Whalen, P. Debevec:“Immersive Light Field Video with a Layered Mesh Representation”, ACM Trans. on Graphics, Vol. 39, No. 4, pp. 1–15
(2020).
22) B. Attal, S. Ling, A. Gokaslan, C. Richardt, J. Tompkin:“MatryODShka: Real-Time 6DoF Video View Synthesis Using Multi-Sphere Images”, Proc. of Computer Vision – 16th
European Conference (ECCV 2020), pp.441–459 (2020).
23) K. E. Lin, Z. Xu, B. Mildenhall, P. P. Srinivasan, Y. HoldGeoﬀroy, S. DiVerdi, Q. Sun, K. Sunkavalli, R. Ramamoorthi:“Deep Multi Depth Panoramas for View Synthesis”, Lecture
Notes in Computer Science (including subseries Lecture Notes
in Artiﬁcial Intelligence and Lecture Notes in Bioinformatics),
Vol. 12358 LNCS, pp. 328–344 (2020).
24) A. P. Pozo, M. Toksvig, T. F. Schrager, J. Hsu, U. Mathur, A.
Sorkine-Hornung, R. Szeliski, B. Cabral: “An Integrated 6DoF
Video Camera and System Design”, ACM Trans. on Graphics,
Vol. 38, No. 6, Article No. 216, pp.1–16 (2019).

tion for Generating Omnidirectional Video without Invisible
Areas”, IPSJ Trans. on Computer Vision and Applications,Vol. 2, pp.200-213 (2010).
33) C. Li, M. Xu, S. Zhang, P. Le Callet: “State-of-the-Art in 360
Video/Image Processing: Perception, Assessment and Compression”, IEEE Journal of Selected Topics in Signal Processing, Vol. 14, No. 1, pp. 5–26 (2019).
34) R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang: “The
Unreasonable Eﬀectiveness of Deep Features as a Perceptual
Metric”, Proc. of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, No. 1, pp. 586–595(2018).
35) M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter: “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”, Advances in Neural Information Processing Systems, Vol. 2017-Decem, No. Nips, pp.
6627–6638 (2017).
36) A. Al-Saidi, N. J. Avis, I. J. Grimstead, O. F. Rana: “Distributed Collaborative Visualization Using Light Field Rendering”, Proc. of 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID 2009, pp.
609–614 (2009)37) C. Wu: “Towards Linear-Time Incremental Structure from Motion”, Proc. of IEEE 2013 International Conference on 3D Vision, pp.127–134. (2013).
38) C. Sweeney, T. Hollerer, M. Turk: “Theia: A Fast and Scalable
Structure-from-Motion Library”, Proc. of MM 2015 - the 2015
ACM Multimedia Conference, pp. 693–696 (2015).
39) J. L. Schönberger, J. M. Frahm: “Structure-from-Motion Revisited”, Proc. of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, Vol. 2016 Decem,
pp. 4104–4113 (2016).

25) A. Serrano, I. Kim, Z. Chen, S. DIVerdi, D. Gutierrez, A.
Hertzmann, B. Masia: “ Motion parallax for 360 RGBD
Video”, IEEE Trans. on Visualization and Computer Graphics,
Vol. 25, No. 5, pp. 1817–1827 (2019).

40) S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, R. Szeliski:“A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms”, Proc. of the IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, Vol.
1, pp. 519–526 (2006).

26) R. S. Overbeck, D. Erickson, D. Evangelakos, M. Pharr, P.
Debevec: “A System for Acquiring, Processing, and Rendering
Panoramic Light Field Stills for Virtual Reality”, ACM Trans.
on Graphics, Vol. 37, No. 6, pp. 1–15 (2019).

41) L. Chen, H. Lin, S. Li: “Depth Image Enhancement for Kinect
Using Region Growing and Bilateral Filter”, Proc. of International Conference on Pattern Recognition, No. January 2012,
pp. 3070–3073 (2012).

27) H. Shishido, K. Yamanaka, Y. Kameda, I. Kitahara: “PseudoDolly-In Video Generation Combining 3D Modeling and Image Reconstruction”, Adjunct Proc. of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMARAdjunct 2017, pp. 327–333 (2017).

42) X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, C. C. Loy, Y.
Qiao, X. Tang: “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks”, Proc. of the European Conference
on Computer Vision (ECCV) Workshops, pp. 63–79 (2018).

28) P. Isola, J. Y. Zhu, T. Zhou, A. A. Efros: “Image-to-Image
Translation with Conditional Adversarial Networks”, Proc. of 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Vol. 2017-Janua, pp. 5967–5976 (2017).
29) H. Nagahara, Y. Yagi, M. Yachida: “Super-Resolution from an
Omnidirectional Image Sequence”, Proc. of 2000 26th Annual
Conference of the IEEE Industrial Electronics Society, Vol. 4,
pp. 2559–2564 (2000).
30) C. Ozcinar, A. Rana, A. Smolic: “Super-Resolution of Omnidirectional Images Using Adversarial Learning”, Proc. of 2019
IEEE 21st International Workshop on Multimedia Signal Processing (MMSP) (2019).
31) E. Upenik, P. Akyazi, M. Tuzmen, T. Ebrahimi: “Inpainting
in Omnidirectional Images for Privacy Protection”, Proc. of2019 IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP 2019), pp. 2487-2491 (2019).
32) N. Kawai, K. Machikita, T. Sato, N. Yokoya: “Video Comple-

11843) K. Simonyan, A. Zisserman: “Very Deep Convolutional Networks for Large-Scale Image Recognition”, Proc. of 3rd International Conference on Learning Representations, ICLR 2015
- Conference Track Proceedings, pp. 1–14 (2015).

(Received May 31, 2021)(Revised December 18, 2021)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Qiaoge LI

Hansung KIM

He recieved his B.E. degree in Engineeringfrom Dalian University of Technology in 2016,and his M.S.E. degree in Mechanical Engineering from University of Washington in
2019. Since 2020, he became a Ph.D. student
in Empowerment Informatics at the University of Tsukuba. His research interests include novel view synthesis and virtual reality.

He received his Ph.D. degree from Yonsei University, South Korea, in 2005. He is currently
an associate professor in the School of Electronics and Computer Science at the University of Southampton, UK. His research interests include 3D computer vision, audio-visual
data processing, and virtual reality.

Oto

TAKEUCHI

He recieved B.E. in Science Engineering fromthe University of Tsukuba in 2018. In 2017,
he joined Computer Vision and Image Media Laboratory supervised by Itaru Kitahara.
He recieved his M.E. degree in Intelligent andMechanical Interaction Systems from University of Tsukuba. His research interest was
Computer Vision.

Hidehiko SHISHIDO
He received his M.E. and Ph.D. degrees inEngineering from the University of Tsukuba,
Japan, in 2013 and 2016, respectively. In
2016, he served as a researcher at the Japan
Institute of Sports Sciences. Since 2017, hehas been an Assistant Professor at the University of Tsukuba.

Yoshinari KAMEDA (Member)He received a B. Eng., M. Eng., and Ph. D. in
Information Science in 1991, 1993, and 1999
from Kyoto University. He joined Universityof Tsukuba in 2002 and now a full professor at
University of Tsukuba. His research interests
include computer vision, 3D data processing,
VR, AR, Mixed reality, and 3D user interface.

119

Itaru KITAHARA (Member)He received his B.E. and M.E. degrees in Science Engineering from University of Tsukuba,
Japan in 1994 and 1996, respectively. In
1996, he joined Sharp Corporation. 20002003, he was a research associate of Universityof Tsukuba. He received his Ph.D. in 2003.
2003-2005, he was a researcher at ATR. 20052019, he was an assistant professor and associate professor at the University of Tsukuba.
Since 2019, he has been a professor at theCenter for Computational Sciences, University of Tsukuba. His research interests include computer vision, mixed reality, and intelligent image media.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Computerized Classification Method for 1p/19q Codeletion in Low Grade Gliomas
from Brain MRI Images Using Three Dimensional Radiomics Features
Daiki TANAKA† (Student Member), Akiyoshi HIZUKURI†, Ryohei NAKAYAMA††Graduate School of Science and Engineering, Ritsumeikan University<Summary> The purpose of this study was to develop a computerized classification method for 1p/19q codeletion in low grade
gliomas (LGGs) from brain MRI (magnetic resonance imaging) images using three dimensional (3D) radiomics features. Our database consisted of brain T2 weighted MRI images (102 LGGs with 1p/19q codeletion and 57 LGGs without it) obtained from 159
patients. In the proposed method, 107 3D radiomics features were extracted from LGG region in T2 weighted MRI images. The
feature selection was performed with a least absolute shrinkage and selection operator to reduce redundancy among the extracted
3D radiomics features. A support vector machine (SVM) with the selected 3D radiomics features evaluated the likelihood of 1p/19q
codeletion in LGG. A three-fold cross validation method was employed to train and test the proposed method. The classification
accuracy, the sensitivity, the specificity, and the area under the receiver operating characteristic curve with the proposed method
were 80.5%, 83.3%, 75.4%, and 0.836, respectively, showing an improvement when compared with SVM using 2D radiomics
features (74.2%, 78.4%, 66.7%, and 0.783; p = 0.03). The proposed method with 3D radiomics features achieved high classification
accuracy for 1p/19q codeletion in LGG from brain MRI images and would be useful for determining the patient managements.
Keywords: low grade glioma, 1p/19q codeletion, brain MRI image, 3D radiomics features

1. Introduction
Glioma is a primary tumor in brain, accounting for about
40% of central nervous system tumors1). It is classified four
grades (grades I-IV) according to its aggressiveness2). Low
grade gliomas (LGGs) classifying into grades II and III includes three molecular subtypes depending on the presence ofmutations in the isocitrate dehydrogenase (IDH) gene and theco-deficiency of the short arm of chromosome 1 and the longarm of chromosome 19 (1p/19q codeletion)3). Appropriate
treatment and the prognosis depend on those molecular subtypes. Especially, LGGs with 1p/19q codeletion are longer
survival and positive response to chemotherapy4-6). Therefore,
World Health Organization recommends determining thetreatment plan of patient with LGG based on the diagnostic
result of 1p/19q codeletion by genetic analysis. However genetic analysis for 1p/19q codeletion, which is an invasive examination, causes a heavy burden on patient7).
To overcome this problem, some investigators have developed computerized classification methods for 1p/19q codeletion in LGGs from brain MRI images. Hirano et al. proposed
an estimation method for 1p/19q codeletion using a linear discriminant analysis with two dimensional (2D) radiomics features8). Zhou et al. developed a classification method for
1p/19q codeletion using a random forest with 2D radiomics

120

features9). Those 2D radiomics features were determined byapplying a large number of feature extractors to LGG in MRI
images10). Some studies have applied convolutional neuralnetworks (CNNs) to classify 1p/19q codeletion in LGGs frombrain MRI images. Gonzalez et al. developed a classification
method with GoogLeNet11). Akkus et al. also used multi-scale
CNNs that integrate feature maps of different sizes12). The
CNN approach achieved higher classification performance
than the traditional approach based on a classifier with handcraft features. However, those studies only analyzed a LGG ina slice image with a maximum diameter of the LGG. Therefore, the LGG has not been analyzed in the through-plane direction, which means that only part of the LGG has been analyzed.
So, to analyze the whole LGG three-dimensionally in volume data will worth checking to improve the classification accuracy. The purpose of this study is to develop a computerizedclassification method for 1p/19q codeletion in LGGs frombrain MRI images using three dimensional (3D) radiomics
features for the whole LGG.

2. Materials and Methods
2.1 Materials
Our database is consisted of brain T2 weighted MRI images

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

for 159 patients (mean age 42; age range 13-84 years) obtained from The Cancer Imaging Archive13). It includes 102
LGGs with 1p/19q codeletion and 57 LGGs without 1p/19q
codeletion. Figure 1 shows an example of LGGs with/without
1p/19q codeletion in brain T2 weighted MRI images. The sizeof the T2 weighted MRI image is 256×256 pixels, whereas
the number of slices is from 20 to 60. The spatial resolutions
are 0.78×0.78 mm and 0.94×0.94 mm. The slice thicknesses are also 3.0 and 7.5 mm. Those images are resized using a trilinear interpolation14) to be an isotropic voxel with apixel size of 1 mm. The trilinear interpolation can be seen asa linear interpolation of two bilinear interpolations at throughplane direction. In the Cancer Imaging Archive, all LGG regions are provided as mask images.
The proposed method was developed and evaluated using
python3.7.0 on a workstation (CPU: Intel Core i7-9700K processor, RAM: 48 GB, and GPU: NVIDIA GeForce RTX 2070
SUPER).
2.2 Extraction of 3D radiomics features
To classify 1p/19q codeletion in LGG, 107 3D radiomics
features based on seven categories were extracted from a
LGG region based on the provided mask image in T2
weighted MRI images using pyradiomics15). The seven categories were described below:(1) Morphological Characteristics (MC)16)：14 features(2) Intensity First Order Statistics (IFOS)17)：18 features(3) Gray Level Co-occurrence Matrix (GLCM)18)：
24 features(4) Gray Level Size Zone Matrix (GLSZM)19), 20)：
16 features(5) Gray Level Run Length Matrix (GLRLM)21), 22)：16 features(6) Neighboring Gray Tone Difference Matrix (NGTDM)20), 23)：5 features(7) Gray Level Dependence Matrix (GLDM)24), 25)：
14 features
Category (1) evaluates the shape information of the LGG,

(a) LGG with 1p/19q codeletion

while the six remaining categories (2) - (7) evaluates the intensity variations and the tissue textures. Pyradiomics has extended the analysis of the matrixes such as GLCM, GLSZM,
and GLRLM to 3D and can determine 3D radiomics featuresfrom the whole LGG region (see pyradiomics documentation26) for details). The IFOS is determined from the histogram
based on the signal intensities in the entire LGG region. The
GLCM represents the probabilities of occurrences of combination of the signal intensities at a particular distance and direction. In this study, the GLCMs for each of 13 angles in 3D(26-connectivity) were determined with the distance of 1 pixel.
The GLSZM and the GLRLM represent the size and thelength of connected area with the same voxel intensity, respectively. The NGTDM shows the difference between the signal
intensities and the average signal intensity within a particulardistance, whereas the GLDM represents the relationship between every signal intensity in the entire LGG region and all
intensities at a particular distance. Those different matrices
evaluate the uniformity and the regularity of the signal intensities. The GLSZMs, GLRLMs, GTDMs and GLDMs were
determined from 13 angles.
2.3 Feature selection
A least absolute shrinkage and selection operator(LASSO)27) was employed to perform feature selection by reduce redundancy among the extracted 3D radiomics features.
The LASSO which is one of the linear regression models is aregularization least-square method using the sum of absolutevalues of regression coefficients. The linear regression model
including the LASSO was defined by    .

(1)Here,  was the teacher signals (LGG with/without 1p/19q
codeletion), and  was the extracted 3D radiomics features.
 and  were the regression coefficients and the error term,
respectively. In the LASSO, the regression coefficients were
determined by ∶min‖  ‖ଶଶ  ‖‖ଵ ,ࢼ(2)where ‖‖ଵ was the ଵ norm constraint of .  was thetuning parameter that controls the strength of the constraint.
Some of the regression coefficients that satisfy Eq. (2) were
given as zero. Note that the 3D radiomics features had the coefficient of zero were assumed to be redundant. The 3D radiomics features given non-zero were determined as the features
for classifying 1p/19q codeletion in LGG. The parameter was determined by grid search. Here, the range of  was set
to from 0.01 to 0.02 in increments of 0.001.

(b) LGG without 1p/19q codeletion

Fig.1 Example of brain T2 weighted MRI images

121

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

2.4 Classification of 1p/19q codeletion in LGG
A support vector machine (SVM)28) was employed to distinguish LGGs with 1p/19q codeletion from LGGs without it.
The SVM determines the hyperplane that maximizes the margin (distance) from the hyperplane to the closest point acrossboth classes.
The 3D radiomics features selected with the LASSO was
inputted to the SVM. Here, those features were normalized.
The output of the SVM provided the likelihood of 1p/19q
codeletion in LGG. Gaussian kernel29) was empirically determined as a kernel function in the SVM. The parameters ‫ܥ‬and ߛ in the SVM were determined by grid search. The
range of ‫ ܥ‬that controls misclassification was set to from 1
to 1000, whereas that of ߛ that controls the complexity of thehyperplane was set to from 0.0001 to 0.1 in increments of 10
times.
2.5 Evaluation indices
The classification accuracy, the sensitivity30), the specificity were used to evaluate the classification performance ofthe proposed method. The area under the receiver operating
characteristic curve (AUC)31) was also used for evaluation ofclassification performance.
A ݇ -fold cross-validation method32) with ݇ = 3 wasused for training and testing of the proposed method. Thismethod divided our database randomly into three groups. One
group was used as a test dataset, whereas the remaining two
groups were used as a training dataset. This process was repeated three times until every group was used as test dataset.
To evaluate the usefulness of 3D analysis, the classification performance for the proposed method with 3D radiomics
features was compared to that for the classification methodwith 2D radiomics features. In the classification method with
2D radiomics features, 107 2D radiomics features, whichwere the same as the 3D radiomics features, were determinedfrom a LGG region. Feature selection and classification process were the same manner as the proposed method.
30)

3. Results
Figure 2 shows the selected 3D radiomics features by the
LASSO in each training dataset using the three-fold cross validation method. With the grid search, the parameter ߣ of the
LASSO in each dataset was determined to be 0.016 for dataset
1, 0.019 for dataset 2, and 0.019 for dataset 3, respectively.
The numbers of the selected 3D radiomics features in the
training datasets 1, 2, and 3 were 17, 20, and 19, respectively,
whereas 11 3D radiomics features were selected in common
for all training datasets. The 11 3D features were Elongation,

122

Maximum, Minimum, Cluster Shade, Inverse Difference Moment Normalized, Maximal Correlation Coefficient, LongRun Low Gray Level Emphasis, Strength, Coarseness, Dependence Variance, and Large Dependence High Gray Level
Emphasis.
Based on the highest the F1-score33) in each training dataset, the parameter ‫ ܥ‬of the SVM with the selected 3D radiomics features were determined to be 100 for dataset 1, 1,000
for dataset 2, and 1,000 for dataset 3, respectively. The parameter ߛ was also determined to be 0.001 in all training datasets.
Table 1 shows the classification performances for the proposed method with 3D radiomics features and the computerized method with 2D radiomics features. Seven 2D features(Maximum, Kurtosis, Cluster Shade, Long Run Low Gray
Level Emphasis, Coarseness, Dependence Variance, andLarge Dependence High Gray Level Emphasis) were selectedin common for all 2D training datasets. Of the 11 3D features
selected in all training datasets, Elongation, Minimum, Inverse Difference Moment Normalized, Maximal Correlation
Coefficient, and Strength were not included in the 2D features
selected in common. Those features evaluate mainly the local
homogeneity in signal intensities within LGG region. The
classification accuracy, the sensitivity, the specificity, and the
AUC for the proposed method were 80.5% (128/159), 83.3%(85/102), 75.4% (43/57), and 0.836, respectively, which weregreater than those for the computerized method with 2D radiomics features (74.2%, 78.4%, 66.7%, and 0.783; p = 0.03).

4. Discussion
In this study, 3D radiomics features for the whole LGG
were used to classify 1p/19q codeletion in LGG from brain T2
weighted MRI images. The 3D radiomics features that also
analyze in the through-plane direction can significantly improve the classification performance with the 2D radiomics
features.
Elongation, Maximum, Minimum, Cluster Shade, Inverse
Difference Moment Normalized, Maximal Correlation Coefficient, Long Run Low Gray Level Emphasis, Strength,
Coarseness, Dependence Variance, and Large DependenceHigh Gray Level Emphasis were selected in common for all
training datasets as shown in Fig.2. Most of those features are
related to the complexity and the momentum in signal intensities within LGG region. Therefore, we believe the distribution of the signal intensities within LGG region contributes toclassify 1p/19q codeletion. The regression coefficient for the
Dependence Variance tended to be high in each training dataset. This feature is related to the homogeneity in signal intensities. Therefore, the homogeneity in signal intensities

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Root Mean Squared
Total Energy
Gray Level Non Uniformity
Least Axis Length
Surface Volume Ratio
90 Percentile
Size Zone Non-Uniformity
Small Dependence
Low Gray Level Emphasis
Kurtosis
Size Zone Non-Uniformity
Normalized
Cluster Prominence
Inverse VarianceInterquartile Range
EnergyLarge Dependence
Low Gray Level Emphasis
Maximum 3D Diameter
Uniformity
Correlation
SkewnessElongation
MCC
Idmn
CoarsenessStrength
Long Run
Low Gray Level Emphasis
Minimum
Maximum
Large DependenceHigh Gray Level Emphasis
Cluster Shade
Dependence Variance
0

0.02

0.04

0.06

0.08
0.1
0.12
Regression Coefficients

dataset1

dataset2

0.14

0.16

dataset3

Fig.2 Regression coefficients of the LASSO for selected 3D radiomics features in each training dataset

123

0.18

0.2

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Table 1 Comparison of classification performances for the proposed method and the computerized method with 2D radiomics features

MethodComputerized method with
2D radiomics features

Proposed methodkClassification accuracy [%]Sensitivity [%]Specificity [%]AUC

1

77.4 (41/53)

82.4 (28/34)

68.4 (13/19)

0.797

2

67.9 (36/53)

76.5 (26/34)

52.6 (10/19)

0.714

3

77.4 (41/53)

76.5 (26/34)

78.9 (15/19)

0.844

Ave.

74.2 (118/159)

78.4 (80/102)

66.7 (38/57)

0.783*

1

77.4 (41/53)

85.3 (29/34)

63.2 (12/19)

0.807

2

81.1 (43/53)

82.4 (28/34)

78.9 (15/19)

0.879

3

83.0 (44/53)

82.4 (28/34)

84.2 (16/19)

0.882

Ave.

80.5 (128/159)

83.3 (85/102)

75.4 (43/57)

0.836*p = 0.03

might be effective for the classification of 1p/19q codeletion.
To evaluate the usefulness of the 3D radiomics features,the proposed method was compared to the computerized
method with 2D radiomics features corresponding to the selected 3D radiomics features. The classification accuracy, thesensitivity, the specificity, and the AUC for the computerized
method with 2D radiomics features corresponding to the selected 3D radiomics features were 67.9% (108/159), 75.5%(77/102), 54.4% (31/57), and 0.726 (p = 0.001), respectively,
which were lower than those for the proposed method with theselected 3D radiomics features. 2D radiomics features would
not have evaluated the image features and the texture features
accurately in the whole LGG region.
In the LGGs with 1p/19q codeletion which were correctlyclassified by the proposed method and were incorrectly classified by the computerized method with 2D radiomics features,the signal intensities within the LGG regions tended to be uniform in the slice plane, but tended to be non-uniform in thethrough-plane direction. On the other hand, in the LGGs without 1p/19q codeletion which were correctly classified by the
proposed method and were incorrectly classified by the computerized method with 2D radiomics features, the signal intensities within the LGG regions appeared non-uniform in theslice plane, but could be evaluated as uniform throughout the
LGGs. The 2D analysis of LGG in the slice image was inaccurate, and the 3D analysis proved to be important.
In previous studies, the CNN approach achieved higher
classification performance than the traditional approach basedon a classifier with handcraft features11), 12). We also compared
the proposed method with the 3D convolutional neural networks (3D CNNs)34). The architecture of the 3D CNNs was
ResNet-1835) extended to 3D. The classification accuracy, thesensitivity, the specificity, and the AUC for the proposed
method were higher than those for the 3D CNNs (68.6%,
72.5%, 61.4%, and 0.687; p =0.007). As our database was too

124

small to provide the enough training data for the 3D CNNs it
was impossible to learn the difference between LGG with
1p/19q codeletion and LGG without 1p/19q codeletion, in signal pattern. On the other hand, the SVM used in this study has
only two parameters which were cost (C) and gamma (ߛ).
Thus, it is easy to adjust those parameters compared with the
3D CNNs.
To investigate the usefulness of the feature selection basedon the LASSO, we compared the proposed method with theselected 3D radiomics features to the SVM with all 3D radiomics features (107 3D radiomics features). The classification
accuracy, the sensitivity, the specificity, and the AUC for the
proposed method were substantially higher than those for the
SVM with all 3D radiomics features (77.4%, 81.4%, 70.2%,
and 0.799; p = 0.04). Therefore, the feature selection by the
LASSO was shown to be useful in the classification of 1p/19q
codeletion in LGG.
There are some limitations in this study. One limitation isthat our database in this study was small. In the future study,
we need to expand our database, and then evaluate the proposed method for the classification of 1p/19q codeletion. The
second limitation is that the mask images provided from The
Cancer Imaging Archive were used as the LGG regions. It
would be boring for radiologists to manually trace the LGG
regions at clinical practice. Therefore, we have to develop asegmentation method for LGG.

5. Conclusion
In this study, 3D radiomics features for the whole LGG
were used to improve the classification accuracy for 1p/19q
codeletion in LGG from brain MRI images. The proposed
method exhibited a higher classification accuracy when compared to the computerized method with 2D radiomics features.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

Acknowledgements11) S. R. Gonzalez, I. Zemmoura, C. Tauber: “Deep Convolutional NeuralThis work was supported by JSPS department expense19K20719.

References
1)

MRI in Low Grade Gliomas”, Proc. of the 10th International Conference
on Pattern Recognition Systems (ICPRS) (2019).
12) Z. Akkus, I. Ali, J. Sedlar, J. P. Agrawal, I. F. Parney, C. Giannini, B. J.

K. U. Rathod, Y. D. Kapse: “Automated Brain Tumor Detection and

Erickson: “Predicting Deletion of Chromosomal Arms 1p/19q in Low-

Brain MRI Classification Using Artificial Neural Network – A Review”,

Grade Gliomas from MR Images Using Machine Intelligence”, JournalInternational Journal of Science and Research, Vol. 5, No.7, pp. 175–179

of Digital Imaging, Vol. 30, pp. 469–476 (2017).
13) B. Erickson, Z. Akkus, J. Sedlar, P. Korﬁatis: “Data From LGG-

(2016).
2)

Network to Predict 1p19q Co-deletion and IDH1 Mutation Status from

M. Nitta, T. Komori: “Outline and Problems of the WHO 2016 Classification of Tumors of the Central Nervous System”, Japanese Journal of

1p19qDeletion”, The Cancer Imaging Archive (2017).
14) P. Thévenaz, T. Blu, M. Unser: “Image Interpolation and Resampling”,
In: Handbook of Medical Image Processing and Analysis, edited by

Neurosurgery, Vol. 25, No. 11, pp. 782–791 (2017).
3) D. N. Louis, A. Perry, G. Reifenberger, A. V. Deimling, D. Figarella-Bankman IN. Second Edition. Academic Press, pp. 465–493 (2009).

Branger, W. K. Cavence, H. Ohgaki, O. D. Wiestler, P. Klcihues, D. W.

15) J. J. M. van Griethuysen, A. Fedorov, C. Parmar, A. Hosny, N. Aucoin, V.

Ellison: “The 2016 World Health Organization Classification of Tumors

Narayan, R. G. H. Beets-Tan, J. C. Fillion-Robin, S. Pieper, H. J. W. L.

of the Central Nervous System: A Summary”, Acta Neuropathologica,

Aerts: “Computational Radiomics System to Decode The Radiographic

Vol. 131, No. 6, pp. 803–820 (2016).

Phenotype”, Cancer Research, Vol.77, No. 21, pp. e104–e107 (2017).

4) K. Ichimura, H. Arita, Y. Narita: “Genetic Basis for the Development of16) W. E. Lorensen, H. E. Cline: “Marching Cubes: A High Resolution 3D

Adult Gliomas”, Japanese Journal of Neurosurgery, Vol. 23, No. 7, pp.

Surface Construction Algorithm”, Proc. of ACM SIGGRAPH Computer

532–540 (2014).

Graphics, Vol. 21, No. 4, pp. 163–169 (1987).

5) G. Cairncross, M. Wang, E. Shaw, R. Jenkins, D. Brachman, J. Buckner,17) N. Aggarwal, R. K. Agrawal: “First and Second Order Statistics Features

K. Fink, L. Souhami, N. Laperriere, W. Curran, M. Mehta: “Phase Ⅲ

for Classification of Magnetic Resonance Brain Images”, Journal of Sig-

Trial of Chemoradiotherapy for Anaplastic Oligodendroglioma: Long-

nal and Information Processing, Vol. 3, No. 2, pp. 146–153 (2012).

term Results of RTOG 9402”, Journal of Clinical Oncology, Vol. 31, No.

18) B. Song, G. Zhang, H. Lu, H. Wang, W. Zhu, P. J. Pickhardt, Z. Liang:“Volumetric Texture Features from Higher-order Images for Diagnosis of3, pp. 337–343 (2013).
6) M. W. Ruff, J. Uhm: “Anaplastic Glioma: Treatment Approaches in the
Era of Molecular Diagnostics”, Current Treatment Options in Oncology,

Colon Lesions via CT Colonography”, International Journal of Computer
Assisted Radiology and Surgery, Vol. 9, No. 6, pp. 1021–1031 (2014).
19) G. Thibault, B. Fertil C. Navarro, S. Pereira, P. Cau, N. Levy, J. Sequeira,Vol. 19, No. 12, p. 61 (2018).
7) S. Miki, K. Ichimura, Y. Narita: “How to Understand The Results of BasicJ. L. Mari: “Texture Indexes and Gray Level Size Zone Matrix Applica-

Glioma Genome Sequence Data”, Japanese Journal of Neurosurgery, Vol.

tion to Cell Nuclei Classification”, Pattern Recognition and Information

26, No. 11, pp. 806–816 (2017).

Processing (PRIP), pp. 140–145 (2009).

8) N. Hirano, Y. Uchiyama: “Radiomics for Estimating 1p/19q Codeletion20) M. Valliéres, C. R. Freeman, S. R. Skamene, I. EI. Naqa: “A Radiomics

in Brain Tumor Using Magnetic Resonance Imaging”, International Jour-

Model from Joint FDG-PET and MRI Texture Features for The Predic-

nal of Computer Assisted Radiology and Surgery, Vol. 15, No. 1, pp.

tion of Lung Metastases in Soft-tissue Sarcomas of The Extremities”,
Physics in Medicine & Biology, Vol. 60, No. 14, pp. 5471–5496 (2015).

S126–S127 (2020).
9) H. Zhou, D. Chang, H. -X. Bai, B. Xiao, C. Su, W. -L. Bi, P. -J. Zhang, J.

21) M. M. Galloway: “Texture Analysis Using Gray Level Run Lengths”,

T. Senders, M. Vallieres, V. K. Kavouridis, A. Boaro, O. Arnaout, L. Yang,

Computer Graphics and Image Processing, Vol. 4, No. 2, pp. 172–179

R. -Y. Huang: “Machine Learning Reveals Multimodal MRI Patterns Pre-

(1975).

dictive of Isocitrate Dehydrogenase and 1p/19q Status in Diffuse Low and22) T. -Y. Kim, H. -J. Choi, H. -G. Hwang, H. -K. Choi: “Three-dimensional

High-grade Gliomas”, Journal of Neuro-Oncology, Vol. 142, No. 2, pp.

Texture Analysis of Renal Cell Carcinoma Cell Nuclei for Computerized

299–307 (2019).

Automatic Grading”, Journal of Medical Systems, Vol. 34, No. 4, pp.

10) H. J. W. L. Aerts, E. R. Velazquez, R. T. H. Leijenaar, C. Parmar, P. Gross-709–716 (2010).

mann, S. Carvalho, J. Bussink, R. Monshouwer, B. Haibe-Kains, D.

23) M. Amadasun, R. King: “Textural Features Corresponding to Textural

Rietveld, F. Hoebers, M. M. Rietbergen, C. R. Leemans, A. Dekker, J.

Properties”, IEEE Trans. on Systems, Man, and Cybernetics, Vol. 19, No.

Quackenbush, R. J. Gillies, P. Lambin: “Decoding Tumour Phenotype by
Noninvasive Imaging Using A Quantitative Radiomics Approach”, Na-5, pp. 1264–1274 (1989).
24) C. Sun, W. G. Wee: “Neighboring Gray Level Dependence Matrix for
Texture Classification”, Computer Vision, Graphics, and Image Pro-

ture communications, Vol. 5, No. 1, pp. 1–9 (2014).

cessing, Vol. 23, No. 3, pp. 341–352 (1983).

125

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

25) F. Bianconi, M. L. Fravolini, I. Palumbo, G. Pascoletti, S. Nuvoli, M.
Rondini, A. Spanu, B. Palumbo: “Impact of Lesion Delineation and Intensity Quantisation on The Stability of Texture Features from Lung Nod-

Daiki TANAKA(Student Member)
He is a Master course student at the Graduate
School of Engineering, Ritsumeikan University.
He is involved in research on machine learning.

ules on CT: A Reproducible Study”, Diagnostics, Vol. 11, No. 7, p. 1224
(2021).
26) Pyradiomics, https://pyradiomics.readthedocs.io/en/latest/index.html,(2022).
27) R. Tibshirani: “Regression Shrinkage and Selection via The Lasso”, Journal of the Royal Statistical Society. Series B (Methodological), Vol. 58,
No. 1, pp. 267–288 (1996).
28) V. N. Vapnik: “Statistical Learning Theory”, John Wiley & Sons, (1998).
29) W. Wang, Z. Xu, W. Lu, X. Zhang: “Determination of the Spread Parameter in the Gaussian Kernel for Classification and Regression”, Neurocomputing, Vol. 55, No. 3–4, pp. 643-663 (2003).
30) A. Benjaminse, A. Gokeler, C. P. van der Schans: “Clinical Diagnosis of
an Anterior Cruciate Ligament Rupture: A Meta-analysis”, Journal of Orthopaedic and Sports Physical Therapy, Vol. 36, No. 5, pp. 267–288(2006).
31) T. Fawcett: “ROC Graphs: Notes and Practical Considerations for Data
Mining Researchers”, Technical Report HPL-2003-4, HP Labs (2003).
32) R. Kohavi: “A Study of Cross-validation and Bootstrap for Accuracy Estimation and Model Selection”, Proc. of the 14th International Joint Conference on Artificial Intelligence, Vol. 2, No. 12, pp. 1137–1143 (1995).
33) A. P. Zijenbos, B. M. Dawant, R. A. Margolin, A. C. Palmer: “Morphometric Analysis of White Matter Lesions in MR Images: Method and Validation”, IEEE Trans. on Medical Imaging, Vol. 13, No. 4, pp. 716–724(1994).
34) M. Decuyper, S. Bonte, K. Deblaere, R. V. Holen: “Automated MRI
Based Pipeline for Glioma Segmentation and Prediction of Grade, IDH
Mutation and 1p19q Co-deletion”, Computerized Medical Imaging andGraphics, Vol. 88, No. 101831, pp. 1–-9 (2021).
35) K. He, X. Zhang, S. Ren, J. Sun: “Deep Residual Learning for Image
Recognition”, Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2016).
(Received Nov. 19, 2021)(Revised Jan. 22, 2022)

126

Akiyoshi HIZUKURIHe received his Ph.D. degree from Mie University in 2014. During 2014-2018, he was working at Mizuho Research & Technologies, Ltd.
Since 2018, he has been working at Ritsumeikan
University. He is currently an assistant professor
at the Graduate School of Engineering,
Ritsumeikan University. His research interests include machine learning. He is a member of IEICE(Japan), IEEJ, JSMP, and JAMIT.

Ryohei NAKAYAMA
He received his Ph.D. degree from Mie University in 2005. During 2005-2015, he was an assistant professor at Mie University Hospital.
Since 2015, he has been working at Ritsumeikan
University. He is currently a professor at theGraduate School of Engineering, Ritsumeikan
University. During 2008-2009, he was a visiting
assistant professor at University of Chicago. His
research interests include medical image recognition, and machine learning. He is a member ofIEEE, SPIE, IEICE (Japan), IPSJ, JAMIT, ISJ,
and JSRT.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
IIEEJ

Paper
Improvements over Coordinate Regression Approachfor Large-Scale Face Alignment
Haoqi GAO† (Student Member) , Koichi OGAWARA†† The Faculty of System Engineering, Wakayama University<Summary> The facial alignment task has been well-studied extensively and achieved signiﬁcantprogress in recent years. However, previous works remain challenging due to the ambiguity of invisible
landmarks under extreme viewpoints (e.g., large pose and expression). This paper proposes a novel dense
network on top of the coordinate regression method to improve face coordinates localization in extreme
environments. The attention mechanism can better guide the model on which information to emphasize or
suppress. In our network, we embed a Convolutional Block Attention Module (CBAM) in three stages of thedensely connected convolutional network (DenseNet) respectively for face alignment tasks. Then we concatenate landmark location labels with bounding box location and head pose value as the guide information forour network. We demonstrate the performance of our network through the AFLW2000-3D, AFLW2000-3DReannotated, Menpo-3D test datasets. Comparative experiments reveal that our network with lower mean
NME (3.2%) outperforms the baseline DenseNet (3.66%), ShuﬄeNet (4.39%), and DenseNet+SE (3.93%)
on AFLW2000-3D-Reannotated. We conclude that our network obtains improved performances for facelandmarks prediction even in extreme conditions.
Keywords: face alignment, coordinate regression, CBAM, DenseNet

1.

Introduction

Facial alignment1)–3)aims to estimate the projectionsof facial key points (e.g., eyes-corners, nose tip, eyebrows, chin center, and mouth corners) onto the face images, which is a necessary pre-processing for many facial
tasks. Face alignment is considered an essential intermediate step for face analysis. Many relevant tasks including face recognition4),5), 3D reconstruction6),7), andface attribute estimation8),9)need to consider it. Neuralnetwork-based algorithms have made substantial progressin face alignment and achieved high accuracy rates.
Previous face alignment approaches are parametric ﬁtting10)–12), regression-based methods13)–17). Regressionbased algorithms can further divide into coordinate regression, which takes the landmark coordinates as theregression target, and heatmap regression, which outputs
the likelihood response for each landmark. An advantage of a model-ﬁtting-based approach is that it establishes point correspondences between 2D facial images
and typical 3D facial models, making it easier to handle facial alignment in complex scenes, especially for invisible points predicted in large-scale poses. However,127

the ﬁtting-based method is typically time-consuming, andthe network structure used is relatively complex, whichmakes the model’s parameter space large. Several studies
have shown that regression-based methods perform welland achieve high precision for frontal and near-frontal
face images. However, in real-world applications, human faces are often exposed to uncontrolled and unconstrained environments, which opens up many possibilities
for capturing face images and brings various challenges tothe existing regression-based model training process. For
instance, some landmarks become invisible under largescale pose situations, and even expert operators ﬁnd it
hard to calibrate landmarks positions accurately.
In this work, our goal is to improve the performanceof facial landmark estimation in extreme environments
based on regression-based methods. In this paper, We
recombine and integrate the features extracted from previous dense block layers18), comprehensively exploiting
both bottom-level high-resolution features and top-level
semantics powerful features. Also, inspired by the attention mechanism, we propose a novel dense network forface alignment by embedding the Convolutional Block
Attention Module (CBAM)19)into a densely connectedIIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

5H/8

0D[3RROLQJ&RQ9%DWFK1RUP

6WDJH&%$0
PRGH

7UDQVLWLRQ
/D\HU

&%$0
PRGH

&%$0
PRGH

%RWWOHQHFNOD\HU %RWWOHQHFNOD\HU 'HQVHEORFN'HQVHEORFN)*$YHUDJH
3RROLQJ)*$YHUDJH
3RROLQJ

6WDJH
7UDQVLWLRQ
/D\HU

&%$0
PRGH

&%$0
PRGH

)&
/D\HU

%RWWOHQHFNOD\HU 'HQVHEORFN
6WDJH
7UDQVLWLRQ
/D\HU

&%$0
PRGH

&%$0
PRGH

)

*$YHUDJH
3RROLQJ%RWWOHQHFNOD\HU 'HQVHEORFN

Fig. 1The proposed network architecture

Convolutional Net. Next, We propose a new loss that incorporates landmark localization loss, GIoU loss of facebounding box, and head pose regression loss to guideour landmark localization model. Moreover, We conduct comparative experiments of Normalized Mean Error (NME) metrics based on AFLW2000-3D-Reannotateddatasets, AFLW2000-3D-Test datasets, 300W-Testset3D, and Menpo-3D datasets. Our method obtains alower NME value in the experiment compared to otherclassical network structures like MobileNet20),21), ShufﬂeNet22),23). Furthermore, experimental results showthat our method outperforms DenseNet18)and DenseNet

facial bounding box locations, facial landmarks, and 3Dvertices force the network to learn exclusive facial features.
Heatmap-based regression methods generate heatmaps
as the output for each landmark. Newell et al.30)werepioneers in designing a stacked hourglass (SH) network
to generate heatmaps for human pose estimation. The
state-of-the-art performance on facial alignment held
for some time by using Hourglass models or other deformable Hourglass models31),32). Deng et al.33)adopted

with Squeeze-and-Excitation(SE)24).

tion, which can simultaneously obtain the 2D and 3D
coordinates of facial landmarks. MobileFAN34), using

2.

lightweight architectures MobileNetV2 as the backbone
to achieve better accuracy.
The method based on 3D model ﬁtting, which of-

Related Work

Face alignment task is a well-studied area in the ﬁeld ofcomputer vision. Based on existing face alignment methods, we categorize methods into three subcategories: coordinate regression-based methods, heatmap regressionbased methods, and 3D model ﬁtting-based methods. In
this section, we brieﬂy review some previous works of eachsubcategory.
The output of coordinate-based regression methods is avector of landmark coordinates. Sun et al.25)used a CNNbased coordinate regression method to regress facial landmarks location for the ﬁrst time. Zhu et al.26)was introduced the ODN network to overcome the occlusion problem. A multi-stage model27), using spatial transformer
networks, hourglass networks, and exemplar-based shape
constraints to improve face alignment accuracy. PFLD28)
employs a branch of the network to estimate geometric information for each face sample, subsequently regularizing
the landmark localization. Retinaface29)jointly learning

128

a Cascaded Multi-view Hourglass scheme for face alignment. Researchers proposed SRN2)for landmark estima-

fers more expressive and occluded information than its
2D counterpart, has received lots of attention from researchers in recent years. Earlier researchers1)explored
face alignment to large-pose face images by combining
cascaded CNN regressor and 3DMM model. PR-Net6)
proposes a 2D representation called UV Positional Map
that records the 3D shape of a face in UV space, thentrains a simple Convolutional Neural Network to regress
it from images. VRN35)performs a direct regression of
the volumetric representation of 3D facial geometry fromthe 2D image. Wei et al.36)propose a graph convolution
network to regress 3D face coordinates, which directly
performs feature learning on the 3D face mesh. Recent
researches for the challenge of wild face images include
2DASL7), and 3DDFA237), which show state-of-the-arts
for both 3D face reconstruction and dense face alignment

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）&)PD[,QSXW)&î+î:ter’s computational eﬀort and eﬀectively suppress overﬁt-U 

0D[3RROting. The SharedMLP is a two-level-fully connected layer.
Because for the attention model, we want to make the

5H/8

&
)DYJ

6LJPRLG$YJ3RRO

weights correlate with the input, thus avoiding the issues
with ﬁxed weights in the FC layer. A sigmoid activation
with smoothing advantages use to generate the ﬁnal channel attention feature map CA (F ) ∈ RC×1×1 . Diﬀerent&$ )6KDUHG0/3

˄&îî˅

&RQY
OD\HU

)

6LJPRLG&î+î:^0D[3RRO$YJ3RRO`

from channel attention, spatial attention focuses on where
the informative part is, which is complementary to channel attention19). In the spatial attention block, channel-

6$ ) 

î+î:based global max pooling and global average pooling areapplied to the input F 1 , then a convolutional layer with

)
&î+î:

Fig. 2a kernel size of 7 × 7 is applied. Finally, the attention
features SA (F ) ∈ R1×H×W are generated by a sigmoid

The structure of CBAM model19)

function.
Mathematical expressions for the process of the channeland spatial attention mechanisms are given in Eq. (1).

by a large margin.

3.

Method Overview

3.1 Network architecture
We employ a hierarchical architecture with several convolutional layers to extract features and make landmark
predictions. Considering the satisfactory performance ofDenseNet techniques18), which is a densely connected network that can perform feature multiplexing well. Attention mechanism improves the representation of interest19),24),38). In our approach, we embed the CBAM module19)into DenseNet to eﬃciently assist the information
ﬂow within the network by learning which information toemphasize or suppress.
The pipeline of the proposed method is shown in
Fig. 1. The detail of the architecture is depicted blockwise: the gray blocks are convolution blocks with thebatch norm. The number of bottleneck layers for thedense blocks is given by [6, 12, 48, 32]. The green, blue
and red blocks are CBAM models, Max pooling layers,
and transition layers, respectively. At the end of eachstage, we extract the features deﬁned as F1, F2, and F3,
respectively. The output channel dimensions of these features are 512, 1792, and 1920, respectively. Finally, we
concatenate these features through Global Average Pooling layers and pass them to an FC layer to obtain theﬁnal output.
An illustration of the CBAM model detail provides inFig. 2.
The channel attention block focuses on what is meaningful given an input image. The given feature map
F ∈ RC×H×W is passed through max pooling and average pooling and then through an MLP with a reduction
ratio of r = 8. The purpose of r is to reduce the parame129

F 1 = CA (F ) ⊗ F ; F 2 = SA (F 1 ) ⊗ F 1 ,(1)where ⊗ denotes element-wise multiplication. F 2 is theﬁnal reﬁned output. The transition layer consists of abatch norm layer, a ReLU layer, a 1 × 1 convolutional
layer, and a 2 × 2 average pooling layer. Its role is to
reduce the dimension of each output channel of the Dense
block18).
3.2 Loss function
Based on landmark loss function (L), we also add headpose (H) (yaw, pitch, and roll) loss that can guide the
distribution of facial rotation and add bounding box (B)loss to help improve the accuracy of the model further.
3.2.1 Landmark loss (L)Wing loss39)for landmark regression to improve theability to cope with minor and medium-range errors during deep network training. It is deﬁned as follows:|x| < ωω ln (1 + | x |),(2)
W (x) =
|x| − C
otherwiseωC = (ω − ω ln (1 + ( ))).
(3)
li = {(lx1 , ly1 ), ..., (lx68 , ly68 )}i represent the predicted
landmarks coordinate(face bounding, eye, eyebrows,
nose, and mouth) and li∗ = (lx∗1 , ly∗1 ), ..., (lx∗68 , ly∗68 ) i represent the landmarks coordinate of the ground-truth, and
x=|li − li∗ | and M is the number of images. According to
Wing loss39), we also set ω = 10,  = 2.
3.2.2 Head pose loss (H)For head pose, pi = {p1 , p2 , p3 }i represents the valueof the predicted Euler angle (the three unit vectors of

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

the face gaze direction, that is, pitch yaw and roll).
p∗i = {p∗1 , p∗2 , p∗3 }i represents the value of the ground-truth

Loss = λ1

LM SE =M1(pi − p∗i )2
M i=1

M


1 − GIoU (bi , b∗i )

4.

(4)
4.1

(5)

i=1

bi = {bx1 , by1 , bx2 , by2 }i represents the coordinates of the
predicted bounding box and b∗i = b∗x1 , b∗y1 , b∗x2 , b∗y2 i represents the coordinates of the ground-truth bounding box.
For each face image i, its multi-task loss deﬁnes as:
Algorithm 1(6)

Empirically, we set λ1 = 0.1, λ2 = 0.25.

3.2.3 Bounding box loss (B)Bounding box regression is one of the most fundamental
parts of many computer vision tasks such as object localization, object detection, object tracking. One improvement is the Intersection over Union (IoU) loss. However,when two objects do not overlap, the IoU value will be
zero and not reﬂect how far the two shapes are from eachother. For non-overlapping targets, if the IoU uses as aloss, its gradient will be zero and cannot be optimized.
Inspired by40), we incorporate Generalized Intersection
over Union (GIoU) loss into the bounding box loss. This
loss preserves the main properties of IoU while having a
gradient in all potential cases, including non-overlapping
situations. The calculation process of GIOU and the difference with the IOU algorithm in the Algorithm 1.

LGIoU =

W (x) + λ2 LGIoU + LM SE

i=1

Euler angle. Here, the loss we used for the pose estimatoris the mean squared error (MSE) loss function.

M

A Comparison for Iou and GIou algorithm

Experiments

Datasets“300W-LP1)” dataset contains 61,225 synthetic face images along with their corresponding 2D and 3D groundtruth landmark-based annotations. It is synthesized from
300W41)through a morphable 3D model proﬁling algorithm proposed in paper1)and coverage across large pose
ranges from -90 to 90 degrees.
“300-VW16)” is the large-scale face tracking datasetthat contains 114 videos. Among these, 64 videos wereused for testing and the rest 50 videos were used for training with 95,192 frames in total. For the test videos, wehave three categories (CatA, CatB, and CatC). And CatC
is considered the most challenging video due to its low
resolution and poor facial quality.
“300W-Testset-3D41)” contains two categories: Indoor
and Outdoor. The dataset of the 300-W Challenge includes a total of 600 images from the in-the-wild.
“AFLW2000-3D1)” contains 2,000 face samples chosenfrom the AFLW dataset42), introduced by Zhu et al. alongwith 300W-LP datasets with varying expressions and illumination conditions.
“AFLW2000-3D-Reannotated16)” is re-annotated tomake the ground truth more accurate because for faces
with diﬃcult poses, the method from paper1)fails to produce precise landmarks. It provides along with the LS3DW dataset. Following the literature of2), we also catego-

Input:
Predicted bounding box coordinate B p , andgroundtruth bounding box coordinate B g ;Output:GIoU and IoU;
1: For B p = (bx1 , by1 , bx2 , by2 ), calcuate b̄x1 , b̄y1 , b̄x2 , b̄y2 ;b̄x1 = min(bx1 , bx2 ), b̄x2 = max(bx1 , bx2 );b̄y1 = min(by1 , by2 ), b̄y2 = max(by1 , by2 );2: The area Ap = (b̄x2 − b̄x1 ) × (b̄y2 − b̄y1 );The area Ag = (b∗x2 − b∗x1 ) × (b∗y2 − b∗y1 );3: The intersection I = (b◦x1 , b◦y1 , b◦x2 , b◦y2 );b◦x1 = max(b̄x1 , b∗x1 ), b◦x2 = min(b̄x2 , b∗x2 );b◦y1 = max(b̄y1 , b∗y1 ), b◦y2 = min(b̄y2 , b∗y2 );4: The area AI = (b◦x2 − b◦x1 ) × (b◦y2 − b◦y1 )if b◦x2 > b◦x1 , b◦y2 > b◦y1 ; else AI = 0;
5: The smallest enclosing box B c = (bcx1 , bcy1 , bcx2 , bcy2 );bcx1 = min(b̄x1 , b∗x1 ), bcx2 = max(b̄x2 , b∗x2 );bcy1 = min(b̄y1 , b∗y1 ), bcy2 = max(b̄y2 , b∗y2 );6: The area Ac = (bcx2 − bcx1 ) × (bcy2 − bcy1 );I
7: IoU= AU where U = Ap + Ag − AI ;c
8: GIoU=IoU − AA−Uc ;rize the face images in AFLW2000-3D-Reannotated into
three groups of views [0◦ , 30◦ ], [30◦ , 60◦ ], [60◦ , 90◦ ]. It
includes 1312, 390, and 296 images in the three groups,
respectively.
“Menpo-3D43)” dataset contains a total of 8,955 challenging frames that vary in illuminations, poses, and occlusions.
4.2

Evaluation metrics

A metric used for facial landmark localization algorithm is the point-to-point Euclidean distance, normalized by the square root of the ground-truth bounding box
size instead of the common inter-pupil16). The formula for
NME can be written as Eq.(7).
GT E(li , li∗ ) =130

N

1  ||li − li∗ ||2
N i=1
di

(7)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
NME compared with diﬀerent network model
AFLW2000-3D-Reannotated NME(%)
Method[0◦ ,30◦ ] [30◦ ,60◦ ] [60◦ ,90◦ ] Mean Std
MobileNet20)
5.10
6.58
7.29
6.32 1.12
ShuﬄeNet22)
2.98
4.15
5.15
4.09 1.09
DenseNet+SE24)
3.01
4.21
4.58
3.93 0.82
18)
DenseNet
2.76
3.99
4.24
3.66 0.79
DenseNet+CBAM 2.37
3.33
3.89 3.20 0.77

NME compared with diﬀerent loss functions
AFLW2000-3D NME(%)
Method[0◦ ,30◦ ] [30◦ ,60◦ ] [60◦ ,90◦ ] Mean Std
RCPR44)
4.26
5.96
13.18
7.80 4.74
ESR45)
4.60
6.7
12.67
7.99 4.19
SDM46)
3.67
4.94
9.76
6.12 3.21
Yu et al.47)
3.62
6.06
9.56
6.41 2.99
3DDFA1)
3.78
4.54
7.93
5.42 2.21
3DDFA+SDM1) 3.43
4.24
7.17
4.94 1.97
48)
3DSTN
3.15
4.33
5.98
4.49 1.42
ExpNet49)
4.01
5.46
6.23
5.23 1.13
FAME50)
3.11
3.84
6.60
4.52 1.84
SS-SFN51)
3.09
4.27
5.59
4.31 1.25
Ours(L)
3.18
4.78
5.59
4.52 1.23
Ours(L+B)
3.03
4.76
5.42
4.40 1.23
Ours(L+H)
2.99
4.59
5.31
4.30 1.19
Ours(L+B+H)
2.96
4.38
4.95 4.10 1.02

Table 1

Fig. 3

2ULJLQDO

0RELOH1HW

6KXIIOH1HW'HQVH1HW6('HQVH1HW

2XUV

Table 2

2XUV /

2XUV /%2XUV /+2XUV /%+Comparison results of diﬀerent networks with Ours
model (red–predicted,bule–ground truth)

Herein, N is the number of landmarks, li is the predicted landmarks points, li∗ is their corresponding groundtruth, and di denotes the distance for the i-th frame. It√
is computed as di = wb hb . wb and hb are the width andheight of the bounding box.
4.3 Experimental results and discussion
In Table 1, we compare with other existing mainstream neural network architectures (e.g.,ShuﬄeNet

Fig. 422),23)pose and bounding box loss functions under same
(DenseNet+CBAM) model. Ours(L) only employs landmark coordinates, Ours(L+B) adds the bounding box position based on Ours(L), and Ours(L+H) adds the head

, MobileNet20),21)and DenseNet18)).
Attention
24)
mechanism SE uses global average pooled features
to compute the channel-wise attention.
Model
DenseNet+SE means SE attention added into the
DenseNet architecture.
We learn and compare each of these models with thesame (Ours(L+B+H)) loss function. As the Table 1
shown, our model (DenseNet+CBAM) has better performance than other advanced networks. Compared tothe original DenseNet and DenseNet+SE structures, ourmodel (DenseNet+CBAM) performs with a reduction in
NME(%) of 0.46 and 0.73, respectively.
In Fig. 3, we give comparative visualization results fordiﬀerent networks. Our model predicts the location withthe smallest distance from the ground truth.
In Table 2, we compared Ours(L), Ours(L+B), andOurs(L+H) to verify the eﬀectiveness of both head

131

Comparison results of Ours(L), Ours(L+B),
Ours(L+H), and Ours(L+B+H) with the groundtruth (red–predicted,bule–ground truth)

pose information. Ours(L+B+H) represents an experiment concatenating landmark points, head pose, andbounding box information.
Table 2 results show our method drops 0.21 in Mean
NME values compared with SS-SFN51). Even though
FAME50)performs well in [30◦ , 60◦ ] range, our method
achieves lowest value in both the [0◦ , 30◦ ] and [60◦ , 90◦ ].
In Fig. 4, we also give comparative visualization results for diﬀerent loss functions.
Figure 4 results show that Ours(L+B+H) model predicts keypoint locations (red points) with the smallest distance compared to ground truth (blue points). It demonstrates that incorporating both bounding box and headIIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

pose information can assist the model in improving facecontains the glasses obscuration challenge and the exag-alignment accuracy.

gerated expression pose challenge. Traditional algorithms

In Table 3, we evaluated our model on 300W-Testset-

have large errors in estimating key points of the mouth

3D and Menpo-3D test datasets, which collect under challenging conditions. Our proposed method achieves lower
NME values when compared with other methods.
In Fig. 5, we compared our model with traditional
CNN+3D method52)on 300W private datasets. As shownin the ﬁgure above, the third test image from the left

and face contours. However, our model can accurately
locate the facial key points with small errors.
More landmark detection visualization results of ourmodel are shown in Fig. 6. The main test cases focus onlarge-scale expressions, illumination, and 90-degree side
face challenges. Figure 6 result shows that our model can
locate landmarks on face images clearly and accurately inthose challenging scenarios and can also accurately detectinvisible points for the side-face dataset.
We also compare our method’s predictions (red points)

Comparison of NME on the Menpo-3D and 300W
testset datasets
Method 300WTestset NME(%) Menpo3D NME(%)
FAN16)
2.83
3.70
SRN2)
2.77
3.35
Ours
2.38
2.84

2XUV

&11'

2ULJLQDO

Table 3Fig. 5Comparing results of our method with CNN+3D52)
on images 300W private test set

Fig. 6

with the ground truth (blue points) in Fig. 7. The
results clearly show that our methods can produce landmarks with minor errors compared to the ground truth.
All experiment results on various test datasets(e.g., AFLW2000-3D, 300W-Testset-3D, and Menpo-3D)
demonstrate the robustness of our model even underlarge poses, occlusions, exaggerated expressions, lowresolution, and diﬀerent light conditions.
Next, to evaluate the processing speed, we calculate ouralgorithm’s eﬃciency on an Nvidia GTX 2080 Ti GPU.
Our model’s performance computes on the resolution of
256 × 256. To evaluate the model size and computation
complexity, we calculate the number of network parameters (#Parameters) and Frames Per Second (FPS), a
measure of computational speed.
Table 4 lists comparison reports of network complexity
and runtime with other methods.

Example results of our method on image from 300W-Testset-3D and Menpo-3D databases

132

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）

 

 

 


    





 







Fig. 7Comparing results of our method with ground truth on images from the 300W-Testset-3D(ﬁrst row), AFLW20003D(second-row), and Menpo-3D(third and fourth-row) databases respectively

Comparison in terms of network complexity andruntime
Runtime
Method Year
#ParametersGPU
FPS
CALE53) 2016
∼ 140M
3
54)
DVLN
2017
∼ 132M
DAN55)
2017
∼ 22M
45
SAN56)
2018∼ 57.4M
343ms
50
LAB57)
2018∼ 50.7M
60ms
6
GEAN58) 2020
58.8.0ms
LUVLi59) 2020
17.0msOurs∼ 20.4M
14ms
75

Table 4The overall number of parameters is about 20.4M in
our network, and it takes 33min to train one epoch on
the 300W-LP dataset (61,225 images) and 300VW-train
dataset (95,192 images). The inference speed is around75 fps. As can be seen, our model has smaller parameters
and lower computational complexity than other methods(e.g., LAB57), GEAN58), LUVLi59)).

5.

Conclusion

In this paper, we propose an improved coordinate regression method to estimate the location of facial landmarks. We build upon the superior performance ofDensenet techniques18)by applying multi-scale featuremaps and adding attention modules that can emphasize or suppress information ﬂow within the network bylearning. We further guide the detection by concatenating landmark coordinates, head pose information, andbounding box locations. Experimental results show that
our approach outperforms existing comparative methods
133

in challenging large-scale face alignment tasks. In addition, extensive experiments on challenging datasets (suchas large poses, occlusions, exaggerated expressions, lowresolution, and diﬀerent light conditions) also show that
our training model is stable and robust.

References
1) X. Zhu, Z. Lei, X. Liu, H. Shi, S.-Z. Li: “Face Alignment AcrossLarge Poses: A 3D Solution”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.146–
155 (2016).
2) X. Luo, P. Li, F. Chen, Q. Zhao: “Improving Large Pose Face
Alignment by Regressing 2D and 3D Landmarks Simultaneously and Visibility Reﬁnement”, Proc. of Chinese Conference
on Biometric Recognition. Springer, Cham, pp.349–357 (2018).
3) Y. Liu, A. ourabloo, W. Ren, X. Liu: “Dense Face Alignment”,
Proc. of IEEE International Conference on Computer Vision
Workshops (ICCVW), pp.1619–1628 (2017).
4) Y. Tai, J. Yang, Y. Zhang, L. Luo, J. Qian, Y. Chen: “Face
Recognition with Pose Variations and Misalignment via Orthogonal Procrustes Regression”, IEEE Trans. on Image Processing, Vol.25, No.6, pp.2673–2683 (2016).
5) S. Bhattacharya, G. S. Nainala, S. Rooj, A. Routray: “Local Force Pattern (LFP): Descriptor for Heterogeneous Face
Recognition”, Pattern Recognition Letters, Vol. 125, pp.63–70(2019).
6) Y. Feng, F. Wu, X. Shao, Y. Wang, X. Zhou: “Joint 3D Face
Reconstruction and Dense Alignment with Position Map Regression Network”, Proc. of European Conference on Computer
Vision (ECCV), pp. 534–551 (2018).
7) X. Tu, J. Zhao, Z. Jiang, Y. Luo, M. Xie, Y. Zhao, L.X. He, Z.
Ma, J. Feng: “3D Face Reconstruction From A Single Image
Assisted by 2D Face Images in the Wild”, IEEE Trans. on
Multimedia, Vol. 23, pp. 1160–1172 (2020).
8) T. Devries, K. Biswaranjan, G. W. Taylor: “Multi-Task Learning of Facial Landmarks and Expression”, Proc. of Canadian
Conference on Computer and Robot Vision, pp.98–103 (2014).

IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
9) M. I. N. P. Munasinghe: “Facial Expression Recognition Using Facial Landmarks and Random Forest Classiﬁer”, Proc. ofIEEE/ACIS 17th International Conference on Computer andInformation Science (ICIS), pp. 423–427 (2018).
10) I. Matthews, S. Baker: “Active Appearance Models Revisited”, International Journal of Computer Vision, Vol.60, No.2,
pp.135–164 (2004).
11) T. F. Cootes, C. J. Taylor, D. H. Cooper, J. Graham: “Active
Shape Models-Their Training and Application”, Computer Vision and Image Understanding, Vol.61, No.1, pp.38–59 (1995).
12) D. Cristinacce, T. F. Cootes: “Feature Detection and Tracking with Constrained Local Models”, Proc. of British Machine
Vision Virtual Conference (BMVC), Vol.1, No.2, pp.3 (2006).
13) S. Zhu, C. Li, C. Change Loy, X. Tang: “Face Alignment by
Coarse-to-Fine Shape Searching”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.4998–
5006 (2015).
14) Z. Tang, X. Peng, S. Geng, L. Wu, S. Zhang: “Quantized
Densely Connected U-Nets for Eﬃcient Landmark Localization”, Proc. of European Conference on Computer Vision(ECCV), pp.339–354 (2018).

3483 (2013).
26) M. Zhu, D. Shi, M. Zheng, M. Sadiq: “Robust Facial Landmark Detection via Occlusion-Adaptive Deep Networks”, Proc.
of IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp.3486–3496 (2019).
27) H. Wang, R. Cheng, J. Zhou, L. Tao, H. K. Kwan: “Multistage Model for Robust Face Alignment Using Deep Neural
Networks”, Cognitive Computation, pp.1–17 (2021).
28) X. Guo, S. Li, J. Yu, J. Zhang, J. Ma, L. Ma, W. Liu, H. Ling:“PFLD: A practical facial landmark detector.” Computer Research Repository (CoRR), Vol.abs/1902.10859 (2019).
29) J. Deng, J. Guo, E. Ververas, I. Kotsia, S. Zafeiriou: “Retinaface: Single-Shot Multi-Level Face Localisation in the Wild”,
Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5203–5212 (2020).
30) A. Newell, K. Yang, J. Deng: “Stacked Hourglass Networks for
Human Pose Estimation”, Proc. of European Conference on
Computer Vision (ECCV), pp.483–499 (2016).
31) J. Deng, G. Trigeorgis, Y. Zhou, S. Zafeiriou: “Joint MultiView Face Alignment in the Wild”, IEEE Trans. on Image
Processing, Vol.28, No,7, pp.3636–3648 (2019).

15) J. Yang, Q. Liu, K. Zhang: “Stacked Hourglass Network for
Robust Facial Landmark Localisation”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW), pp.79–87 (2017).

32) X. Wang, L. Bo, L. Fuxin: “Adaptive Wing Loss for Robust
Face Alignment via Heatmap Regression”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 6971–
6981 (2019).

16) A. Bulat, G. Tzimiropoulos: “How Far are We from Solving the
2D & 3D Face Alignment Problem?(and a Dataset of 230,000
3D Facial Landmarks)”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp.1021–1030 (2017).

33) J. Deng, Y. Zhou, S. Cheng, S. Zaferiou: “Cascade MultiView Hourglass Model for Robust 3D Face Alignment”, Proc.
of IEEE International Conference on Automatic Face Gesture
Recognition (FG) pp.399–403 (2018).

17) S. Ren, X. Cao, Y. Wei, J. Sun: “Face Alignment at 3000 Fps
via Regressing Local Binary Features”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.1685–1692 (2014).

34) Y. Zhao, Y. Liu, C. Shen, Y. Gao, S. Xiong: “MobileFAN:
Transferring Deep Hidden Representation for Face Alignment”,
Pattern Recognition, Vol. 100, pp.107–114 (2020).

18) G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger:“Densely Connected Convolutional Networks”, Proc. of IEEE
Conference on Computer Vision and Pattern Recognition(CVPR), pp.4700–4708 (2017).

35) A. S. Jackson, A. Bulat, V. Argyriou, G. Tzimiropoulos:“Large Pose 3D Face Reconstruction from a Single Image via
Direct Volumetric CNN Regression”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp.1031–1039(2017).

19) S. Woo, J. Park, J. Lee, Y., I. So Kweon: “Cbam: Convolutional Block Attention Module”, Proc. of European Conference
on Computer Vision (ECCV), pp.3–19 (2018).

36) H. Wei, S. Liang, Y. Wei: “3D Dense Face Alignment via
Graph Convolution Networks”, Computer Research Repository(CoRR), Vol. abs/1904.05562 (2019).

20) M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L. C. Chen:“Mobilenetv2: Inverted Residuals and Linear Bottlenecks”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 4510–4520 (2018).

37) J. Guo, X. Zhu, Y. Yang, F. Yang, Z. Lei, S. Z. Li: “Towards
Fast, Accurate and Stable 3D Dense Face Alignment”, Proc. ofComputer Vision ECCV European Conference, Glasgow, PartXIX 16. Springer International Publishing, pp. 152–168 (2020).

21) A. Howard, M. Sandler, G. Chu, L. C. Chen, B. Chen, M. X.
Tan, W. J. Wang, Y. K. Zhu, R. M. Pang, V. Vasudevan, H.
Adam: “Searching for Mobilenetv3.” Proc. of IEEE International Conference on Computer Vision (ICCV), pp.1314–1324(2019).

38) B. Chen, J. Wang, Z. Chi: “Improved DenseNet with Convolutional Attention Module for Brain Tumor Segmentation”,
Proc. of Third International Symposium on Image Computing
and Digital Medicine, pp. 22–26 (2019).

22) N. Ma, X. Zhang, H.T. Zheng, J. Sun: “Shuﬄenet V2: Practical Guidelines for Eﬃcient CNN Architecture Design”, Proc.
of European Conference on Computer Vision (ECCV) pp. 116–
131 (2018).
23) X. Zhang, X. Zhou, M. Lin, J. Sun: “Shuﬄenet: An Extremely
Eﬃcient Convolutional Neural Network for Mobile Devices”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 6848–6856 (2018).
24) J. Hu, L. Shen, G. Sun: “Squeeze-and-Excitation Networks”,
Proc. of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 7132–7141 (2018).
25) Y. Sun, X. Wang, X. Tang: “Deep Convolutional Network Cascade for Facial Point Detection”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.3476–

13439) Z. H. Feng, J. Kittler, M. Awais, P. Huber, X. J. Wu: “Wing
Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.2235–2245(2018).
40) H. Rezatoﬁghi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, S.
Savarese: “Generalized Intersection over Union: A Metric anda Loss for Bounding Box Regression”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 658–666 (2019).
41) C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, M. Pantic: “300
Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge”, Proc. of IEEE International Conference
on Computer Vision Workshops (ICCVW), pp. 397–403 (2013).
42) M. Koestinger, P. Wohlhart, P. M. Roth, H. Bischof: “An-IIEEJ Transactions on Image Electronics and Visual Computing Vol.10 No.1 （2022）
notated Facial Landmarks in the Wild: A Large-Scale, RealWorld Database for Facial Landmark Localization”, Proc. ofIEEE International Conference on Computer Vision Workshops (ICCVW), pp. 2144–2151 (2011).
43) S. Zafeiriou, G. Trigeorgis, G. Chrysos, J. Deng, J. Shen: “The
Menpo Facial Landmark Localisation Challenge: A Step towards the Solution”, Proc. of IEEE Conference on Computer
Vision and Pattern Recognition Workshops (CVPRW), pp.
170–179 (2017).
44) X. P. Burgos-Artizzu, P. Perona, P. Dollar: “Robust Face
Landmark Estimation under Occlusion”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp.1513–
1520 (2013).
45) X. Cao, Y. Wei, F. Wen, J. Sun: “Face Alignment by Explicit
Shape Regression”, International Journal of Computer Vision,
Vol.107, No.2, pp.177–190 (2014).
46) X. Xiong, F. De la Torre: “Supervised Descent Method and its
Applications to Face Alignment”, Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pp.532–
539 (2013).
47) R. Yu, S. Saito, H. Li, D. Ceylan, H. Li: “Learning Dense Facial
Correspondences in Unconstrained Images”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 4723–
4732 (2017).
48) C. Bhagavatula, C. Zhu, K. Luu, M. Savvides: “Faster than
Real-Time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Poses”, Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 3980–
3989 (2017).
49) F. J. Chang, A. T. Tran, T. Hassner, I. Masi, R. Nevatia, G.
Medioni: “Expnet: Landmark-Free, Deep, 3D Facial Expressions”, Proc. of 13th IEEE International Conference on Automatic Face Gesture Recognition (FG), pp.122–129 (2018).
50) F. J. Chang, A. T. Tran, T. Hassner, I. Masi, R. Nevatia,
G. Medioni: “Deep, Landmark-Free Fame: Face Alignment,
Modeling, and Expression Estimation”, International Journalof Computer Vision, Vol.127, No.6, pp. 930–956 (2019).
51) B. Chaudhuri, N. Vesdapunt, B. Wang: “Joint Face Detection and Facial Motion Retargeting for Multiple Faces”, Proc.
of IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 9719–9728 (2019).
52) R. Valle, J. M. Buenaposada, A. Valdes, L. Baumela: “FaceAlignment Using a 3D Deeply-Initialized Ensemble of Regression Trees”, Computer Vision and Image Understanding,Vol.189, pp. 102846 (2019).
53) A. Bulat, G. Tzimiropoulos: “Convolutional Aggregation of
Local Evidence for Large Pose Face Alignment”, Proc. of
British Machine Vision Conference (BMVC) (2016).
54) W. Wu, S. Yang: “Leveraging Intra and Inter-Dataset Variations for Robust Face Alignment”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW), pp. 150–159 (2017).
55) M. Kowalski, J. Naruniec, T. Trzcinski: “Deep Alignment Network: A Convolutional Neural Network for Robust Face Alignment”, Proc. of IEEE Conference on Computer Vision and
Pattern Recognition Workshops (CVPRW), pp. 88–97 (2017).
56) X. Dong, Y. Yan, W. Ouyang, Y. Yang: “Style Aggregated
Network for Facial Landmark Detection”, Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 379–388 (2018).
57) W. Wu, C. Qian, S. Yang, Q. Wang, Y. Cai, Q. Zhou: “Lookat Boundary: A Boundary-Aware Face Alignment Algorithm”,
Proc. of IEEE Conference on Computer Vision and Pattern

135

Recognition (CVPR), pp. 2129–2138 (2018).
58) S. M. Iranmanesh, A. Dabouei, S. Soleymani, H. Kazemi,
N. Nasrabadi: “Robust Facial Landmark Detection via Aggregation on Geometrically Manipulated Faces”, Proc. ofIEEE/CVF Winter Conference on Applications of Computer
Vision (WACV), pp. 330–340 (2020).
59) A. Kumar, T. K. Marks, W. Mou, Y. Wang, M. Jones, A.
Cherian, T. Koike-Akino, X.M. Liu, C. Feng: “LUVLi Face
Alignment: Estimating Landmarks’ Location, Uncertainty,
and Visibility Likelihood”, Proc. of IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 8236–
8246 (2020).

(Received September 30, 2021)(Revised February 21, 2022)(Student Member)
She received the M.E. degree from Beihang
University of Software in 2016. From 2019
she stared a Ph.D. in system engineering at
Wakayama University. Her research interests include face detection, 3D face reconstruction, medical image processing, and deeplearning.

Haoqi GAO

Koichi OGAWARAHe received his Ph.D. degree in Information
and Communication Engineering from the
University of Tokyo, Japan, in 2002. From2006 to 2011, he was a guest Associate Professor at Kyusyu University. Currently, he is an
Associate Professor at Wakayama University.
His research interests include robotics, computer vision, and machine learning. He won
the Best Vision Paper Award at the IEEE International Conference on Robotics and Automation 2007.

Call for Papers
Special Issue on Data Interwork, Sharing, and Reusing Technologies
Supporting Digital Transformation Era
IIEEJ Editorial Committee
In order to solve various social issues speedy and economically, Digital Transformation (DX) is expected
to realize Fully Digitalized Society. Therefore, not only the natural language and image/video of the outerworld, individual behavior and even the motivation and Kansei of the sole human are going to be processed
digitally. The concept of smart home and smart city will be the example of such trend. The utilization of thedata automatically acquired from the user will improve the specification of the products, and sharing service
will degrade the hurdle of possession of the expensive products for the customers, in such perspective. To
promote such tide, the interwork, sharing and reusing the digital data without converting into analogue dataor without truncating it is very important.
Especially in image related area, the technologies to convert and process the image data and its attributes
will contribute to produce the virtual outlook of the products without manufacturing and the image analysis
by AI will provide further benefit in business.
This special issue of the paper targets various image-related technologies such as image processing,
generation, expression, transmission and combined applications of these that support and promote datainterwork, sharing, and reusing for the digital transformation era, and calls for any category (Ordinary paper,
Short paper, System development paper, Data paper, Practice Oriented Paper) of papers.
1.

Topics covered include but not limited to
Image Processing, Image Recognition, Image Detection, Image/Video Communication, Image
Input/Output, Computer Graphics, Visualization, Binocular Vision, 3D image processing, Computer
Vision, Pattern Recognition, Big Data, Image Databases, Machine Learning, Deep Learning,
Understandability, Annotation, Attribute Presumption, Web-based Technology, Security. Creativity,
Usability, Interpretability, Human Interface and Interaction, User Experience, Ubiquitous, AR/VR/MR,
Other related fundamental / application / systemized technologies.

2. Treatment of papers
Submission paper style format and double-blind peer review process are the same as the regular paper. If
the number of accepted papers is less than the minimum number for the special issue, the acceptancepaper will be published as the regular contributed paper. We ask for your understanding and cooperation.
3. Publication of Special Issue:
IIEEJ Transactions on Image Electronics and Visual Computing Vol.10, No.2 (December 2022)
4. Submission Deadline:
Friday, July 29, 2022
5. Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org
6. Online Submission URL: http://www.editorialmanager.com/iieej/

136

Call for Papers
Special Issue on
Image-Related Technology for Digital Fabrication
IIEEJ Editorial Committee
In recent years, there has been a lot of research on digital fabrication, which is the process ofcreating something based on digital data, and various fabrication tools such as 3D printers, laser
cutters, 3D scanners, CNC milling machines, etc. have become accessible to the general public,
enabling individuals to create new things. As a result, it has become possible to create new
products on a personal level. In addition to plastic modeling, metal and wood are also available
as materials, and the use of various materials in combination, such as printing electronic circuits,
has been proposed, and the development and application of MEMS is also expected. Thus,
digital fabrication is not only a rapid prototyping of products, but also a technology that can be
adopted in a wide range of fields such as food, clothing, architecture, medicine, education, andentertainment.
In this special issue, we invite category (Ordinary paper, Short paper, System development
paper, Data paper, Practice Oriented Paper) of papers to realize digital fabrication. We would be
grateful if you could submit your papers.
1. Topics covered include but not limited toComputer Graphics, Image Processing, Image Analysis, Image Recognition, Interaction,
Computer Vision, Machine Learning/Deep Learning, User Interface, VR, AR, MR, 3D
Related Technologies, Image Systems, Image Applications, General Image BasicTechnologies
2. Treatment of papers
Submission paper style format and double-blind peer review process are the same as theregular paper. If the number of accepted papers is less than the minimum number for the
special issue, the acceptance paper will be published as the regular contributed paper. We
ask for your understanding and cooperation.
3. Publication of Special Issue:
IIEEJ Transactions on Image Electronics and Visual Computing Vol.11, No.1 (June 2023)
4. Submission Deadline:
Monday, October 31, 2022
5. Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org
6. Online Submission URL: http://www.editorialmanager.com/iieej/

137

Revised: January 6, 2017
Revised: July 6, 2018

Guidance for Paper Submission
1.

Submission of Papers(1) Preparation before submission
・ The authors should download “Guidance for Paper Submission” and “Style Format” from the“Academic Journals”, “English Journals” section of the Society website and prepare the paperfor submission.
・ Two versions of “Style Format” are available, TeX and MS Word. To reduce publishing costs
and effort, use of TeX version is recommended.
・ There are four categories of manuscripts as follows: Ordinary paper: It should be a scholarly thesis on a unique study, development orinvestigation concerning image electronics engineering. This is an ordinary paper topropose new ideas and will be evaluated for novelty, utility, reliability andcomprehensibility. As a general rule, the authors are requested to summarize a paper
within eight pages.
 Short paper: It is not yet a completed full paper, but instead a quick report of the partial
result obtained at the preliminary stage as well as the knowledge obtained from the said
result. As a general rule, the authors are requested to summarize a paper within four
pages.
 System development paper: It is a paper that is a combination of existing technology or it
has its own novelty in addition to the novelty and utility of an ordinary paper, and thedevelopment results are superior to conventional methods or can be applied to othersystems and demonstrates new knowledge. As a general rule, the authors are requested
to summarize a paper within eight pages.
 Data Paper: A summary of data obtained in the process of a survey, product development,
test, application, and so on, which are the beneficial information for readers even though
its novelty is not high. As a general rule, the authors are requested to summarize a paper
within eight pages.
・ To submit the manuscript for ordinsry paper, short paper, system development paper, or data
paper, at least one of the authors must be a member or a student member of the society.
・ We prohibit the duplicate submission of a paper. If a full paper, short paper, systemdevelopment paper, or data paper with the same content has been published or submitted toother open publishing forums by the same author, or at least one of the co-authors, it shallnot be accepted as a rule. Open publishing forum implies internal or external books,
magazines, bulletins and newsletters from government offices, schools, company
organizations, etc. This regulation does not apply to a preliminary draft to be used at anannual meeting, seminar, symposium, conference, and lecture meeting of our society or othersocieties (including overseas societies). A paper that was once approved as a short paper andbeing submitted again as the full paper after completion is not regarded as a duplicate
submission.
(2) Submission stage of a paper・ Delete all author information at the time of submission. However, deletion of reference
information is the author’s discretion.
・ At first, please register your name on the paper submission page of the following URL, andthen log in again and fill in the necessary information. Use the “Style Format” to upload yourmanuscript. An applicant should use PDF format (converted from dvi of TeX or MS Word138

format) for the manuscript. As a rule, charts (figures and tables) shall be inserted into themanuscript to use the “Style Format”. (a different type of data file, such as audio and video,
can be uploaded at the same time for reference.)
http://www.editorialmanager.com/iieej/
・ If you have any questions regarding the submission, please consult the editor at our office.
Contact:
Person in charge of editing
The Institute of Image Electronics Engineers of Japan
3-35-4-101, Arakawa, Arakawa-Ku, Tokyo 116-0002, Japan
E-mail: hensyu@iieej.org
Tel: +81-3-5615-2893, Fax: +81-3-5615-2894

2.

Review of Papers and Procedures(1) Review of a paper・ A manuscript is reviewed by professional reviewers of the relevant field. The reviewer willdeem the paper “acceptance”, “conditionally acceptance” or “returned”. The applicant is
notified of the result of the review by E-mail.
・ Evaluation method
Ordinary papers are usually evaluated on the following criteria:
 Novelty: The contents of the paper are novel.
 Utility: The contents are useful for academic and industrial development.
 Reliability: The contents are considered trustworthy by the reviewer.
 Comprehensibility: The contents of the paper are clearly described and understood by
the reviewer without misunderstanding.
Apart from the novelty and utility of an ordinary paper, a short paper can be evaluated byhaving a quickness on the research content and evaluated to have new knowledge withresults even if that is partial or for specific use.
System development papers are evaluated based on the following criteria, apart from thenovelty and utility of an ordinary paper.
 Novelty of system development: Even when integrated with existing technologies, thenovelty of the combination, novelty of the system, novelty of knowledge obtained fromthe developed system, etc. are recognized as the novelty of the system.
 Utility of system development: It is comprehensively or partially superior compared tosimilar systems. Demonstrates a pioneering new application concept as a system. The
combination has appropriate optimality for practical use. Demonstrates performance
limitations and examples of performance of the system when put to practical use.
Apart from the novelty and utility of an ordinary paper, a data paper is considered novel if
new deliverables of test, application and manufacturing, the introduction of new technology
and proposals in the worksite have any priority, even though they are not necessarily
original. Also, if the new deliverables are superior compared to the existing technology andare useful for academic and industrial development, they should be evaluated.
(2) Procedure after a review
・ In case of acceptance, the author prepares a final manuscript (as mentioned in 3.).
・ In the case of acceptance with comments by the reviewer, the author may revise the paper inconsideration of the reviewer’s opinion and proceed to prepare the final manuscript (as139mentioned in 3.).
・ In case of conditional acceptance, the author shall modify a paper based on the reviewer’s
requirements by a specified date (within 60 days), and submit the modified paper for
approval. The corrected parts must be colored or underlined. A reply letter must be attachedthat carefully explains the corrections, assertions and future issues, etc., for all of theacceptance conditions.
・ In case a paper is returned, the author cannot proceed to the next step. Please look at thereasons the reviewer lists for the return. We expect an applicant to try again after reviewing
the content of the paper.
(3) Review request for a revised manuscript
・ If you want to submit your paper after conditional acceptance, please submit the reply letterto the comments of the reviewers, and the revised manuscript with revision history to thesubmission site. Please note the designated date for submission. Revised manuscripts
delayed more than the designated date be treated as new applications.
・ In principle, a revised manuscript willl be reviewed by the same reviewer. It is judged either
accceptance or returned.
・ After the judgment, please follow the same procedure as (2).

3.

Submission of final manuscript for publication(1) Submission of a final manuscript
・ An author, who has received the notice of “Acceptance”, will receive an email regarding thecreation of the final manuscript. The author shall prepare a complete set of the final
manuscript (electronic data) following the instructions given and send it to the office by thedesignated date.
・ The final manuscript shall contain a source file (TeX edition or MS Word version) and a PDF
file, eps files for all drawings (including bmp, jpg, png), an eps file for author’s photograph(eps or jpg file of more than 300 dpi with length and breadth ratio 3:2, upper part of the body)for authors’ introduction. Please submit these in a compressed format, such as a zip file.
・ In the final manuscript, write the name of the authors, name of an organizations,introduction of authors, and if necessary, an appreciation acknowledgment. (cancel macros in
the Style file)・ An author whose paper is accepted shall pay a page charge before publishing. It is theauthor’s decision to purchase offprints. (ref. page charge and offprint price information)(2) Galley print proof
・ The author is requested to check the galley (hard copy) a couple of weeks before the paper ispublished in the journal. Please check the galley by the designated date (within one week).
After making any corrections, scan the data and prepare a PDF file, and send it to our officeby email. At that time, fill in the Offprint Purchase Slip and Copyright Form and return thescanned data to our office in PDF file form.
・ In principle, the copyrights of all articles published in our journal, including electronic form,
belong to our society.
・ You can download the Offprint Purchase Slip and the Copyright Form from the journal on our
homepage. (ref. Attachment 2: Offprint Purchase Slip, Attachment 3: Copyright Form)(3) Publication
・ After final proofreading, a paper is published in the Academic journal or English transaction(both in electronic format) and will also be posted on our homepage.

140

IIEEJ Trans. on Image Electronics and Visual Computing©2022 IIEEJ

Vol. 10, No. 1 2022

Editor in Chief: Mei Kodama
The Institute of Image Electronics Engineers of Japan
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan