IIEEJ Transactions on
IIEEJ Transactions on
Image Electronics and Visual Computing

Image Electronics andVisual Computing

Special Issue on Journal Track Papers in IEVC2019

Vol. 7, No. 2 2019

I.I.E.E.J

Editorial Committee of IIEEJ
Editor in Chief
Mei KODAMA (Hiroshima University)

Vice Editors in Chief
Osamu UCHIDA (Tokai University)
Naoki KOBAYASHI (Saitama Medical University)
Yuriko TAKESHIMA (Tokyo University of Technology)

Advisory Board
Yasuhiko YASUDA (Waseda University Emeritus)
Hideyoshi TOMINAGA (Waseda University Emeritus)
Kazumi KOMIYA (Kanagawa Institute of Technology)
Masayoshi AOKI (Seikei University Emeritus)
Fumitaka ONO (Tokyo Polytechnic University Emeritus)
Yoshinori HATORI (Tokyo Institute of Technology)
Mitsuji MATSUMOTO (Waseda University Emeritus)
Kiyoshi TANAKA (Shinshu University)
Shigeo KATO (Utsunomiya University Emeritus)

Editors
Yoshinori ARAI (Tokyo Polytechnic University)
Chee Seng CHAN (University of Malaya)
Naiwala P. CHANDRASIRI (Kogakuin University)
Chinthaka PREMACHANDRA (Shibaura Institute of Technology)
Makoto FUJISAWA(University of Tsukuba)
Issei FUJISHIRO (Keio University)
Kazuhiko HAMAMOTO (Tokai University)
Madoka HASEGAWA (Utsunomiya University)
Ryosuke HIGASHIKATA (Fuji Xerox Co., Ltd.)Naoto KAWAMURA (Canon OB)
Shunichi KIMURA (Fuji Xerox Co., Ltd.)
Shoji KURAKAKE (NTT DOCOMO)
Takashi KANAI (The University of Tokyo)
Tetsuro KUGE (NHK Engineering System, Inc.)
Koji MAKITA (Canon Inc.)
Junichi MATSUNOSHITA (Fuji Xerox Co., Ltd.)
Tomoaki MORIYA (Tokyo Denki University)
Paramesran RAVEENDRAN (University of Malaya)
Kaisei SAKURAI (DWANGO Co., Ltd.)
Koki SATO (Shonan Institute of Technology)
Kazuma SHINODA (Utsunomiya University)
Mikio SHINYA (Toho University)
Shinichi SHIRAKAWA (Aoyama Gakuin University)
Kenichi TANAKA (Nagasaki Institute of Applied Science)
Yukihiro TSUBOSHITA (Fuji Xerox Co., Ltd.)
Daisuke TSUDA (Shinshu University)
Masahiro TOYOURA (University of Yamanashi)
Kazutake UEHIRA (Kanagawa Institute of Technology)
Yuichiro YAMADA (Genesis Commerce Co.,Ltd.)
Norimasa YOSHIDA (Nihon University)
Toshihiko WAKAHARA (Fukuoka Institute of Technology OB)
Kok Sheik WONG (Monash University Malaysia)

Reviewer
Hernan AGUIRRE (Shinshu University)
Kenichi ARAKAWA (NTT Advanced Technology Corporation)
Shoichi ARAKI (Panasonic Corporation)
Tomohiko ARIKAWA (NTT Electronics Corporation)
Yue BAO (Tokyo City University)
Nordin BIN RAMLI (MIMOS Berhad)
Yoong Choon CHANG (Multimedia University)
Robin Bing-Yu CHEN (National Taiwan University)
Kiyonari FUKUE (Tokai University)
Mochamad HARIADI (Sepuluh Nopember Institute of Technology)
Masaki HAYASHI (UPPSALA University)
Takahiro HONGU (NEC Engineering Ltd.)
Yuukou HORITA (University of Toyama)
Takayuki ITO (Ochanomizu University)
Masahiro IWAHASHI (Nagaoka University of Technology)

Munetoshi IWAKIRI (National Defense Academy of Japan)
Yuki IGARASHI (Meiji University)
Kazuto KAMIKURA (Tokyo Polytechnic University)
Yoshihiro KANAMORI (University of Tsukuba)
Shun-ichi KANEKO (Hokkaido University)
Yousun KANG (Tokyo Polytechnic University)
Pizzanu KANONGCHAIYOS (Chulalongkorn University)
Hidetoshi KATSUMA (Tama Art University OB)
Masaki KITAGO (Canon Inc.)
Akiyuki KODATE (Tsuda College)
Hideki KOMAGATA (Saitama Medical University)
Yushi KOMACHI (Kokushikan University)
Toshihiro KOMMA (Tokyo Metropolitan University)
Tsuneya KURIHARA (Hitachi, Ltd.)
Toshiharu KUROSAWA (Matsushita Electric Industrial Co., Ltd. OB)
Kazufumi KANEDA (Hiroshima University)
Itaru KANEKO (Tokyo Polytechnic University)
Teck Chaw LING (University of Malaya)
Chu Kiong LOO (University of Malaya)
Xiaoyang MAO (University of Yamanashi)
Koichi MATSUDA (Iwate Prefectural University)
Makoto MATSUKI (NTT Quaris Corporation OB)
Takeshi MITA (Toshiba Corporation)
Hideki MITSUMINE (NHK Science & Technology Research Laboratories)
Shigeo MORISHIMA (Waseda University)
Kouichi MUTSUURA (Shinsyu University)
Yasuhiro NAKAMURA (National Defense Academy of Japan)
Kazuhiro NOTOMI (Kanagawa Institute of Technology)
Takao ONOYE (Osaka University)
Hidefumi OSAWA (Canon Inc.)
Keat Keong PHANG (University of Malaya)
Fumihiko SAITO (Gifu University)
Takafumi SAITO (Tokyo University of Agriculture and Technology)
Tsuyoshi SAITO (Tokyo Institute of Technology)
Machiko SATO (Tokyo Polytechnic University Emeritus)
Takayoshi SEMASA (Mitsubishi Electric Corp. OB)
Kaoru SEZAKI (The University of Tokyo)
Jun SHIMAMURA (NTT)Tomoyoshi SHIMOBABA (Chiba University)
Katsuyuki SHINOHARA (Kogakuin University)
Keiichiro SHIRAI (Shinshu University)
Eiji SUGISAKI (N-Design Inc. (Japan), DawnPurple Inc.(Philippines))
Kunihiko TAKANO (Tokyo Metropolitan College of Industrial Technology)
Yoshiki TANAKA (Chukyo Medical Corporation)
Youichi TAKASHIMA (NTT)Tokiichiro TAKAHASHI (Tokyo Denki University)
Yukinobu TANIGUCHI (NTT)Nobuji TETSUTANI (Tokyo Denki University)
Hiroyuki TSUJI (Kanagawa Institute of Technology)
Hiroko YABUSHITA (NTT)
Masahiro YANAGIHARA (KDDI R&D Laboratories)
Ryuji YAMAZAKI (Panasonic Corporation)IIEEJ OfficeOsamu UKIGAYA
Rieko FUKUSHIMA
Kyoko HONDA

Contact Information
The Institute of Image Electronics Engineers of Japan (IIEEJ）
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
Tel : +81-3-5615-2893 Fax : +81-3-5615-2894
E-mail : hensyu@iieej.org
http://www.iieej.org/ (in Japanese)
http://www.iieej.org/en/ (in English)
http://www.facebook.com/IIEEJ (in Japanese)
http://www.facebook.com/IIEEJ.E (in English)

IIEEJ Transactions on
Image Electronics and Visual Computing
Vol.7 No.2 December 2019
CONTENTS

Special Issue on Journal Track Papers in IEVC2019
57

Upon the Special Issue on Journal Track Papers in IEVC2019

YurikoTAKESHIMA

Contributed Papers
58

Binary Malignancy Classification of Skin Tissue Using Reflectance andTexture Features from Macropathology Multi-Spectral Images

Eleni ALOUPOGIANNI, Hiroyuki SUZUKI, Takaya ICHIMURA,
Atsushi SASAKI, Hiroto YANAGISAWA, Tetsuya TSUCHIDA,
Masahiro ISHIKAWA, Naoki KOBAYASHI, Takashi OBI

67

Robust, Efficient and Deterministic Planes Detection in Unorganized Point
Clouds Based on Sliding Voxels

Jaime SANDOVAL, Kazuma UENISHI, Munetoshi IWAKIRI,
Kiyoshi TANAKA

78

Pairwise Registration of Low Overlapping Unorganized 3D Point Clouds
Using Supervoxel Segmentation

Luis PERALTA, Jaime SANDOVAL, Munetoshi IWAKIRI,
Kiyoshi TANAKA

88

Weakly-Supervised Learning for Continuous Sign Language Word
Recognition Using DTW-Based Forced Alignment and Isolated Word HMM
Adjustment

Natsuki TAKAYAMA, Hiroki TAKAHASHI

97

A Model Ensemble Approach for Few-Shot Learning Using AggregatedClassifiers

Toshiki KIKUCHI, Yuko OZASA

106Visual Simulation of Tearing Papers Taking Anisotropic Fiber Structure into
Account

Saeko SHINOZAKI, Masanori NAKAYAMA, Issei FUJISHIRO

116

aflak: Visual Programming Environment with Macro Support for
Collaborative and Exploratory Astronomical Analysis

System Development Paper
Malik Olivier BOUSSEJRA, Rikuo UCHIKI,
Shunya TAKEKAWA, Kazuya MATSUBAYASHI,
Yuriko TAKESHIMA, Makoto UEMURA, Issei FUJISHIRORegular Section
Contributed Papers
128

An Efficient Entropy Coding of Sparse Coefficients Based on Sparsity
Adaptation and Atom Reordering for Image Compression

Ji WANG, Yoshiyuki YASHIMA

142

Value Estimation of SpO2 Using a Non-Contact Method : Comparison with aContact Method

Yoshimitsu NAGAO, Yanan GAO, Jiang LIU,
Shigeru SHIMAMOTO

Announcements
151

Call for Papers : Special Issue on IEVC2019

152

Call for Papers : Special Issue on CG & Image Processing Technologies
for Automation, Labor Saving and Empowerment

153

Call for Papers: Special Issue on Image-Related Technologies for the
Realization of Future Society

Guide for Authors
154

Guidance for Paper Submission

Published two times a year by the Institute of Image Electronics Engineers of Japan （IIEEJ）
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan
Tel：+81-5615-2893 Fax：+81-5615-2894 E-mail：hensyu@iieej.org http://www.iieej.org/

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Upon the Special Issue on
Journal Track Papers in IEVC2019
Editor: Prof. Yuriko TAKESHIMATokyo University of Technology
The 6th International Conference on Image Electronics and Visual Computing(IEVC2019) was held in Bali, Indonesia on August 21-24, 2019 as the international
academic event of Image Electronics Engineers of Japan (IIEEJ). It was based on the great
success of previous five workshops in 2007 (Cairns, Australia), 2010 (Nice, France), 2012
(Kuching, Malaysia), 2014 (Koh Samui, Thailand), and 2017 (Da Nang, Vietnam). The aimof the conference is to bring together researchers, engineers, developers, and students fromvarious fields in both academia and industry for discussing the latest researches,standards, developments, implementations and application systems in all areas of image
electronics and visual computing.
There were two paper categories in IEVC2019: general paper and late breaking paper(LBP), and in general paper, there were two tracks: Journal track (JT) and Conference
track (CT). In IEVC2019, 33 JT papers, 52 CT papers and 24 LBP were submitted.
Journal track is a newly introduced one and has the advantage to be able to publish thepaper on the journal (IIEEJ Trans. on IEVC) in the “Special Issue on Journal Track in
IEVC2019” planned on December 2019 issue. JT papers were submitted as full paper
version (8 pages) by the conference paper submission deadline (March 2019), to bepeer-reviewed in advance. At the timing of review result notification for conference paper,
initial review result for journal paper will be also sent to authors. JT papers passed theconference paper level judgement, shorter version (2-4 pages) will be asked to submit for
the inclusion into the proceedings. After the conference, authors were given a period of
three weeks to revise the paper, to reflect the initial review result and also comments
received at the conference presentation.
This special issue is limited to JT papers of which revised version were submitted by theannounced deadline, and includes seven papers which passed the review process to be intime for the publication schedule. We also plan the special issue on “Extended Papers
Presented in IEVC2019” for all presenters in IEVC2019 in the next transaction.
Finally, I would like to thank all the reviewers and editors for their time and efforts
towards improving the quality of papers. I would also like to express my deepest gratitude
to the members of the editorial committee of IIEEJ and the staff at IIEEJ office for various
kinds of support.

57

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2019 -Binary Malignancy Classiﬁcation of Skin Tissue Using Reﬂectanceand Texture Features from Macropathology Multi-Spectral Images
Eleni ALOUPOGIANNI† , Hiroyuki SUZUKI†† , Takaya ICHIMURA††† ,Atsushi SASAKI††† , Hiroto YANAGISAWA††† , Tetsuya TSUCHIDA††† ,Masahiro ISHIKAWA†††† (Member), Naoki KOBAYASHI†††† (Fellow), Takashi OBI††† Tokyo Institute Of Technology, Department of Information and Communications Engineering ,†† Tokyo Institute of Technology, Research Institute for Innovation in Science and Technology ,††† Saitama Medical University, Faculty of Medicine ,†††† Saitama Medical University, Faculty of Health and Medical Care

<Summary> This study suggests an analysis procedure on macropathology multi-spectral images(macroMSI), for visual representation of grossly malignant regions of skin samples during excision margin pathological diagnosis. We implemented binary malignancy classiﬁcation on a database of ten highresolution 7-channel macroMSI tissue samples, captured before and after formalin ﬁxing. We reconstructed
spectral reﬂectance by Wiener estimation and described texture using local binary patterns (LBP). Highlighted malignancy regions were derived from an optimal classiﬁer selected by cross-validated performance.
The results show that malignant regions are highlighted fairly accurately and indicate the importance ofanalyzing unﬁxed tissue in conjunction with ﬁxed tissue.
Keywords: macropathology, multispectral imaging, spectral reﬂectance, texture features, skin cancer classiﬁcation

1.

color variation in the resulting image4). Imaging quality
is essential for eﬀective pathology, taking into accountthat in many instances a second evaluation is necessary.
Furthermore, macropathology is dependent on the age,training and experience of the physician5), while lacking

Introduction

Conditions related to skin cancer are a prevalent healthconcern in Japan. In year 2013 alone, 19.706 skin cancer incidents were recorded across the country1)and skincancer crude mortality rates almost doubled from 1999 in
20142). Skin lesion treatment begins with macropathol-standardization and automation as a procedure. Consequently, it is time-consuming and produces high workload for the pathology laboratory. In the advent of digitalpathology, macropathology remains an impractical task.

ogy, which refers to the initial examination of excised
tissue specimens, prior microscopic evaluation3). Usual

Multi-spectral images (MSI), an enhancement of the
RGB format, can alleviate obstacles in image quality andcolor reproduction through the use of narrow band ﬁlters,
expressing a spectral dimension. MSIs exhibit higher sensitivity to image features that are masked in RGB images,
while MSI cameras become more aﬀordable due to technological advancement. Additionally, MSI is preferred
for retrieving spectral surface reﬂectance of objects compared to RGB. Machine learning classiﬁcation at dermatologist level based on conventional or multispectral
images has been already investigated with favorable accuracy8),9). However, macropathology images are generally neglected, with current focus being on multispectral
digital slide images for histological analysis10), improve-

protocol requires compilation of a report describing grossfeatures and photographs of specimens to map dissection sites after formalin ﬁxing and bread-loaﬁng. Relevant gross features are upheld by changes in shape, size,
color or texture6), while melanoma detection employs thecommonly referred as ABCDE macroscopic descriptors(asymmetry, border, color, diameter, evolution)7).The
pathologist’s goal during initial biopsy is to accurately
identify critical tissue areas on the specimen and assess
the condition of their margins, in order to determine
whether further ablation is required. A major concern inmacropathology is protocol and equipment discrepancies
among diﬀerent pathology laboratories. Speciﬁcations of
the camera system, image acquisition algorithm, scene
illumination and display device are factors which cause

ment of color validity11)and cancer tissue classiﬁcation12).

58

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

By applying traditional machine learning techniques ontimer controlled sequential capture under 7 narrow-band
LED illuminations in the visible spectrum, resulting in 7

macropathology MSI (macroMSI) of excised skin tissue,
this study aims to investigate automatic classiﬁcation of

raw RGB images. Based on the illumination and sensitivity overlap at each spectral frequency, either the R, G

tissue malignancy and provide a visual tool for diagnosis
that performs in a consistent manner independently of
capturing conditions.
We propose a novel quasi-automated framework basedon color and texture analysis of macroMSI, capable ofclassifying malignancy of critical regions on excised tissue in order to assist margin identiﬁcation for skin cancer
in current macropathology practice. Our approach attempts to mimic the pathologists’ assessment of color andtexture, by combining hand-crafted features from recon-

or B channel of the raw image was selected to serve asa channel subimage, resulting in a 9-channel macroMSI
as shown in Fig. 2. For modeling and evaluation purposes, we also captured a conventional RGB image andthe reﬂectance spectrum of each POI with a spectrometer (TOPCON SR-3AR, small area measurements). All
the above steps were repeated 3 times for each specimen:
immediately after excision (unﬁxed case), after formalin
ﬁxing (ﬁxed case) and sectioning (cross sectioned case).

structed spectral reﬂectance and local binary patterns,
respectively. Our aim is two-fold. First, we investigate
the eﬀectiveness of features derived from a new macroMSI
dataset, composed of both ﬁxed and unﬁxed skin specimens during macropathology. Secondly, we examine a visual representation of probability malignancy on criticalareas based on binary classiﬁcation score from traditional
machine learning classiﬁers.

2.
2.1

Materials and Methods

Dataset and capture

All skin specimens comprising our dataset were obtained from the Central Pathology Department of
Saitama Medical University Hospital in Saitama, Japan.
The study subjects were patients for which clinical examination indicated need for further treatment. The hospital
council approved data collection and all participants gave
informed consent for the scientiﬁc use of their data. A total of 10 specimens were excised from an equal number ofpatients after clinical examination. A trained pathologist
examined each excised specimen, identiﬁed on average 5
points of interest (POI) on every specimen and labelledeach POI as malignant or not. The same pathologist
later performed formalin ﬁxing and cross sectioned thespecimens. The eﬀective dataset totalled 115 POIs, ofwhich 35 were captured before ﬁxing (21 malignant, 14
benign), 41 after ﬁxing (23 malignant, 18 benign) and 39
after cross sectioning (24 malignant, 16 benign).
The core element of the capturing system was the openplatform camera OLYMPUS AIR A01 (4/3” Live MOS
sensor, high resolution, 1736x2320 pixels, lens M.ZUIKODIGITAL ED 30mm F3.5 Macro) located at 20cm above

(a)

(b)
Fig. 1 Spectral characteristics (in arbitrary units) of thecamera (a) sensitivity and (b) luminous intensity

the capture stage and contained in a dark box. The spectral sensitivities of the RGB camera sensor and the spec-

Fig. 2 Example of the 9 channels of the macroMSI and
its respective conventional RGB image

tral illumination characteristics are shown in Fig. 1. A

59

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）2.2

growing, with the pathologist-selected pixel as the seed.

Suggested analysis framework

In order to visualise malignancy probability fromStarting from the seed, the ROI grows by including neigh-macroMSI, we propose a uniﬁed work ﬂow as describedbouring pixels with intensity value within 8% diﬀerencein Fig. 3. After obtaining the 9-channel macroMSI, weapply white balance correction, where scene illumination
is estimated with the Gray World assumption on a calibration image of a Macbeth color chart. We identify thelabel POIs and segment a respective region of interest(ROI). Afterwards, from each ROI we reconstruct its reﬂectance spectrum and LBP-based texture. The feature
vector is produced by concatenating the two components
and served as classiﬁcation input. For the sake of selecting the optimal classiﬁer, various classiﬁer conﬁgurations are trained and validated using 80% and 10% of thedataset as the respective train and validation sets. Validation is achieved by 5-fold Stratiﬁed Cross Validation(CV). At each CV iteration, features of the train set areused for training dimension reduction, scaling and training the classiﬁer on the reduced transformation of the
train features. Subsequently, performance is validated byclassifying the test set features, after transforming themusing previously trained scaling and dimension reduction.
Through validation stage we retain only trained classiﬁers
with cross-validated Area Under the ROC curve(AUC)17)
larger than 0.75. Optimal classiﬁers are trained again on
previously used 90% of the dataset and tested on the remaining 10%. During testing, for each POI we obtaina binary class label as malignant/benign together witha classiﬁcation prediction score, whose range and value
diﬀer depending on the classiﬁer. We normalize prediction scores on a range from 0 to 100, in order to express
the percentage of malignancy probability. Higher normalized prediction score as belonging to the malignant

from the current ROI pixels. Region growing is applied
on every channel of the macroMSI in order to produce
binary masks for the ROI. The ﬁnal mask for the ROI
consists of pixels enclosed in at least 6 of the channel
binary masks. In order to speed up feature extraction
without loss of generality, we use a ROI where we limit
region growing at a 30 pixel radius from the seed. Additionally, we use the entire grown ROI without radius
limit during the visualization stage.
2.4 Reﬂectance reconstruction
Reconstruction of the reﬂectance spectrum r̂ from MSI
pixel intensity values g is possible, if the camera spectral
sensitivity and illumination are known. We can regard it
as a linear inverse problem solved by Wiener estimation.
In a matrix form in the discrete space, the generalized
solution for estimated reﬂectance becomes:r̂ = M H T (HM −1 H T + Kn )

−1

g

(1)

with the help of a smoothing matrix M , camera system matrix H and noise matrix Kn . We use a smooth-class was interpreted as higher malignancy probability.
Therefore, we can visualize the classiﬁcation results on ansRGB image obtained from the macroMSI, as a mappingof predicted class together with malignancy percentage
using an intensity color map. The trained optimal classiﬁer can then be incorporated in diagnosis. The proposed
processing steps are described in detail in the following
sections.
2.3

ROI segmentation

Each single pixel POI identiﬁed by the pathologist,
needs to be expanded to a wider ROI with same intensity characteristics. This step reduces the inﬂuence ofnoisy pixels in the analysis and enables malignancy visualisation. We achieve ROI segmentation through region

Fig. 3 Flowchart of the proposed analysis framework formacroMSI

60IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

ing matrix modelled by the autocorrelation of the mea-

nesses display correlation, which can be resolved by di-sured reﬂectance spectra from skin specimens grouped bymension reduction. For this purpose we applied Principal

anatomical location. The noise matrix is modelled withdiagonal covariance σ 2 in the order of 0.1, diﬀerent forevery channel. In order to further improve reconstruction, we considered an additional spatial noise model13),

Component Analysis (PCA) and Independent Component Analysis (ICA). PCA projects data on an axis thatmaximizes variance, whereas ICA identiﬁes orthogonal
independent components of a non-gaussian linear mixture
model for the data16). In order to preserve the physicalwhich denoises pixels via Bayesian Interference from thestatistical information of a pixel’s neighbourhood. Reconstructed reﬂectance was computed as the average of reconstructed spectra from every pixel inside the ROI. The
results of reconstruction were evaluated by average Normalized Root Mean Square Error (NRMSE). Additionally, reconstructed reﬂectances of the whole macroMSI
were used to generate a standard RGB (sRGB) image forvisualisation.
2.5

meaning of reﬂectance and textural features, we applieddimension reduction independently on each of them. Furthermore, in order to avoid regressing reduced reﬂectancecomponents back to the channel number, we train dimension reduction on the respective measured spectra. The
number of components kept after reduction was tuned
among values [10, 20, 30].
2.7 Classiﬁcation
Three traditional classiﬁers were compared for the taskof malignancy classiﬁcation; Support Vector Machine(SVM), Random Forest (RF) and k-Nearest Neighbour(KNN). SVM is a supervised machine learning algorithm
that tries to project data onto a hyperplane which maximizes classiﬁcation margin and has been used extensively
for medical classiﬁcation. A sigmoid kernel accomplishedprojection and parameter C was tuned in range [0, 1.5].
On the other hand, RF is a collection of decision trees
searching for the best feature among a subset of features.
Entropy was used as a decision criterion for 50 components. KNN is a simple classiﬁer, which implements nonparametric, lazy classiﬁcation using a similarity distancemetric in the feature space. We tuned neighbour number
among values {1, 3, 5} with correlation as distance metric. AUC and overall accuracy of malignancy predictions
compared to ground truth labels of the pathologist areused to select optimal classiﬁers. By design, SVM and

Local binary patterns

Local Binary Patterns (LBP) are an eﬃcient and widelyused texture desciptor. Conventional LBP functions as aspectral histogram of intensity diﬀerences in a neighbourhood of P pixels across 2P bins, computed in scale R froma central pixel. Due to the frequent appearance of uniform patterns14), a uniform LBP descriptor can decrease
feature size without substantial performance loss. In ourcase pattern rotation is also irrelevant, as homogeneous
skin tissue surrounds lesions, allowing the use of Rotation
Invariant Uniform LBP (RIU-LBP) variation. However,application of LBP on macroMSI shows intricacies because texture does not progress across the spectral dimension, thus we investigate three variations of 3D LBP. For
the ﬁrst two variations, RIU-LBP histogram is calculated
independently for each channel. Afterwards, all band results are concatenated (CatLBP, N histograms of (P + 2)bins) or summed across bins (SumLBP, (P + 2) bins).
The third variation is implemented as the Multispectral
Multiscale LBP (MMLBP)15), which expands RIU-LBP
by adding a multispectral component computed across Pλ

KNN provide values of the discrimination function in order to compute the necessary thresholds for class separation during AUC calculation. On the other hand, RF provides prediction probability values which can substitutediscrimination function values in AUC calculation. The
feature set was chosen among combinations of reﬂectancespectrum (Spect) plus LBP variations. During classiﬁcation, as ﬁxed dataset we considered images of both ﬁxed
and sectioned tissue, because sectioning is performed after formalin ﬁxing.

Pλadjoining spectral channels ( (P + 2)2 bins). We calculated the various LBP descriptors over 8 spatial neighbours, 2 spectral neighbours, at scales 1 and 2, and concatenated the scale features.
2.6

Dimension reduction

An overtly large feature vector not only increases required time for training, but also causes computational
instability and overshadows important components, alongwith introducing problems deriving from the curse of dimensionality. In our case, inherently, both neighbouring spectral frequencies and neighbouring spatial bright-

3.

Results

3.1 Performance of feature extraction
The feature vector contained two types of features: re-61

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）ﬂectance features and texture features.

Employing

MMLBP. For the RGB case, conventional LBP was ap-

Wiener estimation, we reconstructed the reﬂectance spec-

plied on the grayscale-transformed image. Although, thetrum of two individual sources: a conventional RGB im-

larger number of channels of the MSI resulted in larger

age and an MSI captured from the same camera system. Table 1 shows the average NRMSE of reconstruction for diﬀerent tissue datasets. MSI-based reﬂectancereconstruction outperformed RGB-based reconstruction,achieving an average NRMSE of 0.0543 in the complete
dataset, almost half the error of the RGB-based case.
Observing the reconstructed spectra, it is evident that
although reconstruction fails to reproduce non-smoothparts of the reﬂectance curve, the relative height of thecurve is reconstructed accurately, as exempliﬁed in Fig.

feature vectors for MSI-based texture, it also allowed for
greater detail in texture description. For example, eventhough the LBP histograms for RGB-LBP and SumLBP
had the same number of bins, diﬀerent values were assigned to each bin, due to the small variance of LBP values at each image pixel obtained from the former. The remaining two descriptors, namely CatLBP and MMLBP,
can describe texture in greater detail due to the larger
amount of bins assigned to diﬀerent spectral channels(CatLBP) or diﬀerent relations among adjoining chan-

4. Light-green solid line denotes the target measured reﬂectance, while x markers indicate center wavelengths of
the MSI camera channels. Dashed blue and solid magenta lines denote an example of MSI-based and RGBbased reconstructed reﬂectance, respectively. Regarding
image sets for the estimation, MSI-based reconstruction
from imaging of unﬁxed tissue with NRMSE 0.0405 wasmore successful compared to both ﬁxed and cross sectioned tissue. The situation was reversed for RBG-based
reconstruction, with the ﬁxed dataset being superior to
unﬁxed. Although cross-section ROIs are important forunderstanding the interior structure of the tissue, thesmall size of sectioned pieces which contained thin, noisy
ROIs at the edge of the specimen, hindered reconstruction.
We described texture in each ROI of the specimen using
variations of the LBP operator. For the MSI case, threevariations of LBP were extracted; CatLBP, SumLBP andnels (MMLBP). The eﬀect of applying LBP operator in
the spectral dimension is more evident in the classiﬁcation task and will be discussed in section 3.3.
3.2

Dimension reduction performance

After concatenating reﬂectance and LBP values, thefeature vector ended up having up to 351 values (in thecase of CatLBP). Dimension reduction proved especially
useful on texture features. The optimal selection was
PCA on reﬂectance and ICA on texture. PCA had larger
eﬀect on AUC value when applied on the reﬂectance spectrum, whereas ICA improved AUC when applied on all 3
texture descriptors. Judging by the deﬁnitions of the twodimension reduction methods, it is evident that the difference in performance can be attributed to the diﬀerent
underlying models for reﬂectance and texture features.
Indeed, the reﬂectance spectrum is strongly correlated inneighbouring wavelengths, while texture is a mixture ofmaterials and surfaces. Moreover, PCA improved AUC
for RF classiﬁers, but didn’t have any noticeable eﬀect
on KNN or SVM. The AUC results for the case of RFclassiﬁer are shown in Table 2. While the ﬁrst 3 components of PCA in most cases could explain above 96%of the total variance, use of only 3 components did notachieve suﬃcient performance.
3.3

Fig. 4 Comparativeﬂectances

exampleofreconstructed

During validation, the classiﬁcation task was investigated using a variety of conﬁgurations based on SVM,
RF or KNN classiﬁers. Performance comparison of tuned

re-Table 2 Cross-validated AUC per dimension reduction
scheme for RF classiﬁer and ﬁxed tissue input

Table 1 Average NRMSE of reﬂectance reconstruction
from RGB and MSI image source
Source
RGB
MSI

Fixed
0.09550.0643Unﬁxed0.10430.0405

Cross Sectioned
0.12570.0559Validation performance

DimRed
None
PCA
ICA

Overall
0.10850.0543

62

Spect
0.708
0.774
0.733

+CatLBP
0.749
0.728
0.794

+MMLBP
0.749
0.745
0.794

+SumLBP
0.749
0.728
0.794

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）classiﬁers during validation showed that cross-validated

ing dataset using mixed tissue. Ground truth of the test

AUC of all classiﬁers improved with the inclusion of tex-

specimen together with the POI locations that were pin-ture descriptors in the feature vector, rather than using

pointed by the pathologist are presented in Fig. 6. ROI

only spectral reﬂectance, as shown in Fig. 5. RF (50components, entropy criterion) achieved highest AUC of
0.896 using an input set comprising of reﬂectance spectrum and CatLBP texture features with PCA and ICA
dimension reduction respectively. SVM had the second
best AUC overall again with CatLBP used as texture
descriptor. KNN generally had the worst performance
among the three classiﬁers. The choice of a feature set
using Spect+MMLBP with PCA and ICA dimension reduction respectively was the most eﬀective choice in terms

labels are marked with a triangle for malignancy and acircle for benignity. The combination of RF with either
Spect+CatLBP or Spect+MMLBP features was the bestclassiﬁer after testing, with achieved malignancy prediction accuracy at 83.33% on the test specimen as shownin Table 4. On the other hand, SVM achieved less than
70% prediction accuracy. Contrary to expectations raised
from the performance of SVM during validation, SVM
had low performance during testing, which may indicate
over-ﬁtting to the training data.

of AUC. However, comparison with cross-validated accuracy of label prediction against ground truth labels of thepathologist during validation phase in Table 3 showsthat the remaining LBP descriptors have better malignancy prediction properties. In this regard the choice ofSpect+MMLBP or SumLBP. The rationale behind thiscompromise is that, while AUC expresses separability ofthe binary malignancy class, prediction score actually expresses binary malignancy probability, thus feature vector
choice should account for prediction accuracy. Additionally, images of unﬁxed tissue are proven more successful
in terms of AUC as input of the classiﬁer compared tothe ﬁxed tissue (Fig. 5). The mixed dataset had anintermediate performance. This result corroborates theconclusion of reﬂectance reconstruction, which indicates
unﬁxed tissue as a better source for reconstruction. Whenthe feature set was calculated from an MSI rather thanan RGB image source, classiﬁcation improved. As shown

For classiﬁcation with RF, the mispredicted cases referred to a false positive at one of the two POIs on tissue
covered with black hair, as indicated in Fig. 6. Although
RF with Spect+SumLBP achieved perfect prediction on
the test set, the prediction scores were all close to 50%,thus it was discarded. The SVM classiﬁer predicted both
false negatives and false positives. Visualization of classprediction scores during testing as malignancy probabilities of the test ROIs resulted in the heat values shown in
Fig. 6 for unﬁxed tissue. We observe that the segmented
regions are wide and generally coincide with visually identiﬁable regions. Comparing the malignancy probabilities
produced from the two RF classiﬁcation schemes, thecase of Spect+MMLBP shows more distributed malignancy probability. Such probability distribution is moreaccurate and representative of the pathologist groundtruth, considering that the non-malignant area shouldhave a much lower malignancy probability than the ma-in Table 3, classiﬁcation from MSI using reﬂectance spectrum and LBP achieved 80.33% accuracy and 0.82 AUC
during validation, whereas the best RGB-based classiﬁcation scheme achieved 69.42% with 0.50 AUC. The latter
result is unacceptable under the AUC criterion that we
set to select optimal classiﬁers through validation.
3.4

Testing performance(a)

After selecting a handful of optimal classiﬁers throughvalidation stage, we tested them on the remaining testTable 3 CV malignancy prediction accuracy (%) perclassiﬁcation scheme during Validation Stage(top to bottom: SVM, KNN, RF)
RGB
Spect
62.36
69.42
52.50

+LBP
63.99
61.48
54.16

MSI
Spect +CatLBP +MMLBP +SumLBP
60.83
63.33
65.14
68.29
65.14
63.33
65.14
68.29
70.75
73.33
74.97
80.33

(b)
Fig. 5 AUC comparison of optimally tuned classiﬁers after cross validation for all combinations of classiﬁer by feature set (a) and ﬁxing condition of the
input data (b)63IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) Ground truth(b) RF with Spect+CatLBP(c) RF with Spect+MMLBP

Fig. 6 Malignancy classiﬁcation results visualised on an sRGB image of an unﬁxed test specimen
Table 4 Malignancy prediction accuracy (%) per optimal classiﬁer during Testing StageClassiﬁer
RF
RF
SVM

Features+CatLBP+MMLBP+CatLBPParameters
50comp, entropy
50comp, entropy
sigmoid, C=1

lie on the improved reﬂectance reconstruction accuracy
that the unﬁxed dataset provides. Additionally, considering that formalin ﬁxing is a chemical process, which

Accuracy
83.33
83.33
66.67

aﬀects the condition of the tissue, it modiﬁes color andreﬂectance of the unprocessed specimen. Moreover, theunﬁxed dataset contained skin samples from the interiorof the specimen, which may have diﬀerent constitution
compared to exterior tissue and in turn aﬀected classiﬁcation performance. Nonetheless, cross sectioned tissue
should be analysed, due the deep tissue information it
provides, therefore inclusion of unﬁxed tissue in the analysis is recommended.
Regarding potent texture descriptors, MMLBP was revealed to be a compact and practical choice for the analysis of skin macropathology images. Validation results
proved RF classiﬁer as a suitable classiﬁcation model,
which is consistent with its value as a relatively unbiased
predicting model. However, it should be emphasized thatobtaining a large and complete dataset for training is necessary in order to avoid over-ﬁtting. Dimension reduction
seemed more appropriate for textural features rather thanlignant area at the center of the specimen. While theentire core of the test specimen contained malignancy areas, the dark coloured side ROIs are expected to havehigher malignancy probability, compared to the whitecenter, which contains mostly atypical cells, as achievedby the Spect+MMLBP feature vector. In the case ofSpect+CatLBP, malignancy probability for both malignant and non-malignant areas is almost the same around
50% regardless of the correct binary prediction. This fact
highlights the importance of taking into account not only
accuracy, but also validity of the malignancy probability
in the context of macropathology. Even though MMLBP
features during validation were overshadowed by CatLBP,
ﬁnal testing proved MMLBP as a more robust feature
vector.

4.

for reﬂectance features, due to the fact that reﬂectance
was already reconstructed from a limited amount of values equal to the number of multispectral channels. The
choice of PCA for reconstructed reﬂectance spectrum and
ICA for texture features is recommended. The justiﬁcation of the latter may lie in the mathematical deﬁnition of
LBP as a descriptor of frequency of brightness diﬀerence.
Although the size of the feature vector could be reduced
from a few hundreds to just a few tens of values, training dimension reduction on a larger dataset can further
limit the important components in the feature vector andaccelerate the classiﬁcation output.
Although the dataset of the present study was enoughDiscussion

Multispectral imaging of unﬁxed skin tissue proved to
be an eﬀective input set for the binary classiﬁcation task
for malignancy detection, with over 80% accuracy during
testing. Use of MSIs outperformed the use of conventional RGB images at every stage of the analysis. Furthermore, the inclusion of both reﬂectance and texture
features, which mimic the pathologist’s approach during diagnosis, improved classiﬁcation accuracy as well as
AUC value. Unﬁxed and ﬁxed tissue samples were evaluated, revealing the superiority of inclusion of unﬁxed
tissue in the analysis. The logic behind this fact may

64

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

for investigative purposes, extensive experimentation on

Referencesa wider dataset of macroMSIs is required. Analysis of1) Cancer Information Service, National Cancer Center, Japan:“Cancer Incidence and Incidence Rates in Japan in 2009: A
Study of 32 Population-Based Cancer Registries for the Monitoring of Cancer Incidence in Japan (MCIJ) Project”, Japanese
Journal of Clinical Oncology, Vol. 45, pp. 884–891 (2015).

macropathology images poses diﬃculties due to the highvariance of the human skin (skin complex, anatomical
location, age, etc.). Moreover, skin cancer related diseases can have more stages than the dipole malignancy
versus benignity. Such diﬃculties were evident during
our analysis and our limited dataset inhibited classiﬁcation performance. Moreover, captured macroMSIs displayed patches with high saturation and bright reﬂections. Therefore, image quality should be increased bythoroughly wiping liquids from the skin tissue and by incorporating a polarizing ﬁlter in the camera system. A2) M. Nishi:“Epidemiology of Skin Cancer in Japan”, Journal ofTumor, Vol. 4, No. 2, pp. 369–373 (2016).
3) R. Romaguear, M. Nassiri, A. R. Morales:“Tools to Facilitate
and Standardize Grossing”, Histologic, No. 1, pp. 17–21 (2003).
4) Y. Yagi:“Color Standardization and Optimization in Whole
Slide Imaging”, Diagnostic Pathology, BioMed Central, Vol.
6, No. 1, p. S15 (2011).
5) A. F. Jerant, J. T. Johnson, C. D. Sheridan, T. J. Caffrey:“Early Detection and Treatment of Skin Cancer”, American Family Physician, Vol. 62, pp. 357–368 (2000).

notable limitation of the proposed framework is the needfor the pathologist to pinpoint areas respective to thediﬀerent tissue areas on the specimen. In order for theprocedure to be fully automated it is imperative that thecenters of diﬀerent tissue ROIs are automatically identiﬁed with adaptive region segmentation on the MSI, so
that the rest of the steps can be applied. In order toachieve more accurate visual results, malignancy scores
could be adjusted together with clinical information suchas sex, age or medical history, information that is taken
into account during macropathology. Due to the natureof skin lesions, that exist on the surface of the human
body, adaptations of the proposed multispectral analysis
framework can be employed during skin cancer screening
and dermoscopy for early detection purposes.

5.

6) R. L. Siegel, K. D. Miller, A. Jemal::“Cancer Statistics, 2019”,
CA: A Cancer Journal for Clinicians, Vol. 69, No. 1, pp. 7–34
(2019).
7) N. R. Abbasi, H. M. Shaw, D. S. Rigel, R. J. Friedman, W. H.
McCarthy, I. Osman, A. W. Kopf, D. Polsky:“Early Diagnosis of Cutaneous Melanoma: Revisiting the ABCD Criteria”,
Jama, Vol. 292, No. 22, pp. 2771–2776 (2004).
8) J. J. Squiers, W. Li, D. R. King, W. Mo, X. Zhang, Y. Lu, E.
W. Sellke, W. Fan, J. M. Dimaio, J. E. Thatcher:“Multispectral
Imaging Burn Wound Tissue Classiﬁcation System: a Comparison of Test Accuracies Between Several Common Machine Learning Algorithms”, Proc. of Medical Imaging 2016:
Computer-Aided Diagnosis, Vol. 9785, p.97853L (2016).
9) L. Rey-Barroso, F. Burgos-Fernandez, X. Delpueyo, M. Ares,
S. Royo, J. Malvehy, S. Puig, M. Vilaseca:“Visible and Extended Near-Infrared Multispectral Imaging for Skin Cancer
Diagnosis”, Sensors, Vol. 18, No. 5, pp. 1441 (2018).
10) A. Vasaturo, S. Diblasio, D. Verweij, W. Blokx, A. M. Willeke,
J. H. Vankrieken, I. J. Devries, C. G. Fidgor:“Multispectral
Imaging for Highly Accurate Analysis of Tumour-Inﬁltrating
Lymphocytes in Primary Melanoma”, Histopathology, Vol. 70,
No. 4, pp. 643–649 (2017).

Conclusion

11) F. Saleheen, A. Badano, W.-C. Cheng:“Evaluating Color Performance of Whole-Slide Imaging Devices by MultispectralImaging of Biological Tissues”, Proc. of Medical Imaging 2017:
Digital Pathology, Vol. 10140, p.101400R (2017).

We proposed a framework using spectral reﬂectanceand texture features for visualising cancer malignancy
probability on macroMSI of skin specimens to assist margin identiﬁcation during macropathology. Final visual re-12) H. Xu, C. Lu, R. Berendt, N. Jha, M. Mandal:“AutomatedAnalysis and Classiﬁcation of Melanocytic Tumor on Skin
Whole Slide Images”, Computerized Medical Imaging andGraphics, Vol. 66, pp. 124–134 (2018).
13) P. Urban, M. R. Rosen, R. S. Berns:“A Spatially Adaptive
Wiener Filter for Reﬂectance Estimation”, Color and Imaging
Conference, Society for Imaging Science and Technology, Vol.
2008, No. 1, pp. 279–284 (2008).

sults of the proposed procedure depicted fairly accurately
both regions with similar tissue characteristics as well as
their malignancy probabilities. Experiment results suggest RF classiﬁer with a feature vector consisting of reconstructed spectral reﬂectance and MMLBP texture features for the proposed framework. We highlighted the importance of including unﬁxed tissue during analysis andtraining, as well as the superiority of MSI images to conventional RGB images. Although further investigation

14) T. Ojala, M. Pietikainen, T. Maenpaa:“Multiresolution GrayScale and Rotation Invariant Texture Classiﬁcation with Local
Binary Patterns”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 24, No. 7, pp. 971–987 (2002).
15) R. Peyret, A. Bouridane, F. Kheliﬁ, T. Muhammad, S. AlMaadeed:“Automatic Classiﬁcation of Colorectal and Prostatic
Histologic Tumor Images Using Multiscale Multispectral Local
Binary Pattern Texture Features and Stacked Generalization”,
Neurocomputing, Vol. 275, pp. 83–93 (2018).

on a larger dataset is necessary, the suggested analysis
framework of macroMSI was proven to be suitable forenhancing the present pathology practice.

16) A. Hyvarinen, E. Oja:“Independent Component Analysis: Algorithms and Applications.”,Neural networks: the oﬃcial journal of the International Neural Network Society, Vol. 13, pp.

65

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Hiroto YANAGISAWA

411–430 (2000).

He graduated from the Saitama Medical University, Japan. He is currently a medical staﬀ
at the Department of Dermatology, Saitama
Medical University. His clinical and researchinterests include the areas of skin cancer andsurgery. He is a member of Japanese Dermatological Association.

17) M. S. Pepe, T. Cai, G. Longton:“Combining Predictors forClassiﬁcation Using the Area under the Receiver Operating
Characteristic Curve”, Biometrics, Vol. 62, No. 1, pp. 221–229(2005).

(Received September 12, 2019)(Revised November 10, 2019)

Eleni ALOUPOGIANNI

Tetsuya TSUCHIDA

She received a B.Sc&M.Sc. diploma from National Technical University of Athens, Greece,
in 2016. She earned a M.Eng. degree and iscurrently enrolled in a Ph.D. course on Information and Communications Engineering inTokyo Institute of Technology, Japan. She iscurrently working on medical image processing and multi-spectral images.

He received M.D. and Ph.D. degrees from the
University of Tokyo, Japan in 1978 and 1986,
respectively. He is currently a professor at the
Department of Dermatology, Saitama Medical University. His clinical and research interests include the areas of collagen disease,skin cancer and dermoscopy. He is a memberof Japanese Dermatological Association.

Hiroyuki SUZUKI

Masahiro ISHIKAWA (Member)He received a Ph.D degree from Niigata University, Niigata, Japan, in 2006. He is currently an Assistant Professor at the Saitama
Medical University. His current research interests include image processing and computer aided diagnosis.

He received B.E., M.E. and Ph.D. degrees
from Tokyo Institute of Technology in 1998,
2000 and 2006, respectively. From 2003
to 2004, he was a Researcher with Frontier Collaborative Research Center, Tokyo Institute of Technology. From 2004 to 2016,he was a Researcher with Frontier Collaborative Research Center, Tokyo Institute ofTechnology. Since 2016, he has been an Assistant Professor with Institute of Innovative Research, Tokyo Institute of Technology.
His research interests include optical information security, holograms, biometric authentication, and medical and healthcare information systems.

Naoki KOBAYASHI (Fellow)
He received his B.Sc. and M.E. degree from
Tokyo Institute of Technology, Tokyo, Japan,
in 1979 and 1981, respectively, and his Ph.D.
from Niigata University, Niigata, Japan, in
2000. He worked for Laboratories of Nippon
Telegraph and Telephone Corp. from 1981
and 2008. He has been a professor at the
School of Biomedical Engineering, Faculty of
Health and Medical Care of Saitama Medical University since 2008. His research interests include medical image proceeding, image
compression and biosignal processing. He is amember of IEICE, IIEEJ, JSMBE and IEEE.

Takaya ICHIMURAHe earned M.D., Ph.D. degrees and is an assistant professor of the Saitama Medical University. He received the Ph.D. degree from
Kumamoto University, Kumamoto, Japan, in
2005. His current research interests include
nuclear atypia and the molecular nature ofchromatin.

Takashi OBI
He earned B.S. degree in Physics, and M.S.
and Ph.D degree in Information Physics from
Tokyo Inst. of Tech, Japan in 1990, 1992 and1996, respectively. Currently, he is an Associate Professor of Laboratory for Future Interdisciplinary Research of Science and Technology in Institute of Innovative Research, Tokyo
Inst. of Tech. His research focuses on medical
image processing, medical informatics, information system and security, etc. He is a member of IEICE, JAMIT, JSAP, JSNM, JSMPand IEEE.

Atsushi SASAKIHe received M.D. and Ph.D. degrees from
Gunma University School of Medicine in 1980
and 1984, respectively.
He received the
Neuropathology Best Paper Award in 2001,
and the Brain Tumor Pathology Best Paper
Award in 2002. He is currently a professor at
the Department of Pathology, Saitama Medical University. His research interests include
brain tumor pathology and microglia. Dr.
Sasaki is a member of the International Society of Neuropathology (member of the council) and the American Association of Neuropathologists (active member).

66

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2019 -Robust, Eﬃcient and Deterministic Planes Detectionin Unorganized Point Clouds Based on Sliding Voxels
Jaime SANDOVAL† (Student Member), Kazuma UENISHI† (Member), Munetoshi IWAKIRI†† (Member),
Kiyoshi TANAKA† (Fellow)† Shinshu University , †† National Defense Academy of Japan<Summary> Planes detection in unorganized point clouds is a fundamental and essential task in 3D
computer vision. It is a prerequisite in a wide variety of tasks such as object recognition, registration,and reconstruction. Conventional plane detection methods are remarkably slow because they require thecomputation of point-wise normal vectors and are non-deterministic due to their dependency on random
sampling. Therefore, we propose a drastically more eﬃcient and deterministic approach based on slidingvoxels. A sliding voxel is an overlapping grid structure in which we analyze the planarity of the points
distributions to extract hypothetical planes eﬃciently. Each possible plane is validated globally by weighing
and comparing its co-planarity with other sliding voxels’ planes. Experimental results with simulated andrealistic point clouds conﬁrmed that the proposed method is several times faster, more accurate, and morerobust to noise than conventional methods.
Keywords: planes detection, point clouds, pca, ransac, hough transform

1.

Therefore, in this work we contribute with a drastically faster approach: a sliding voxel–based algorithm. It
does not need pre-computed normal vectors; instead, it
directly works with the points distribution of overlapping
3D voxels to acquire surface information. We analyze thescattering of each sliding voxel to locate co-planar regions
and detect hypothetical planes. Then, planes are extracted from the validation of hypothetical planes against
an enhanced subset of the point cloud.
Since ground truth planes data can be obtained from
3D models, we provided experiments with realistic point
cloud simulations generated from these models as wellas ground truth data. To verify the performance of the
proposed method, we measured its eﬃciency and error
against the most popular algorithms: two RANSAC variations 3),14) and the Randomized Hough Transform 17) for
planes detection.
The structure of this paper is as follows. In section 2,
we brieﬂy make deﬁnitions before explaining how the conventional methods work and a critical analysis of these.
Section 3 describes in detail the proposed method, while

Introduction

Point clouds are sets of points in a 3D space that represent the surface of objects and scenes. There are many
point cloud sources such as LiDAR, structured light sensors, stereo vision cameras, and so on. However, highvariations of sensors noise patterns 1),2) are challenging
problems when dealing with point cloud data.
In intelligent robotics, a core task is to recognize environment patterns, especially those from human crafted
objects and scenes which are describable by a set of planar structures. Therefore, the detection of planar surfaces such as obstacles, ﬂoor, stairs, and tables is crucial to several applications; i.e., virtual keypoints detection 3), object detection 4), reconstruction 5), localizationand mapping 6)–9), roofs detection 10) and augmented reality 11). Besides, planes detection is involved in essential
robotics tasks such as placing objects onto planar surfaces 12) or climbing stairs 13).
Conventional planes detection algorithms 3),14) depend
on ineﬃcient point-wise normal vectors computations.
When using 2.5D data (organized point clouds or depth
images), there exist fast algorithms to compute themsection 4 deﬁnes the datasets, their acquisition, as well as
the results of quantitative and qualitative experiments.
Lastly, in section 5 we summarize the results and provide
an insight about future works.

15),16)at the cost of losing accuracy. In addition, their eﬃcient and robust computation in unorganized point clouds
is still an open topic.

67

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）matching but also can be used for lines and planes ﬁt-

2.

Planes Detection in Unorganized Point
Clouds and Related Worksting.
It is an iterative approach consisting mainly in hypoth-

2.1 Planes detection deﬁnition
Let P = {pi ∈ R3 : 1 ≤ i ≤ np } where np is the number

esis and veriﬁcation steps. A subset of n random samples
are drawn from the point cloud (|P| > n) and a hypothetical model M is generated. For each M, an error metricbetween the points and M is computed. The model isveriﬁed by counting the number of inliers τ within an error threshold . If τ does not reach a threshold τ  , theprocess is repeated until k iterations. Once a proper M
is found, then its coeﬃcients should be reﬁned by othermethods such as least mean squares.
k is calculated from the probability of drawing good
samples from the data, i.e., samples that generate accurate models. Let w be the probability of drawing an
inlier from the data, then b = wn will be the probabilityof points in a point cloud. These points are unorganized,
hence, algorithms commonly build structures from them,
namely octrees. They are sampled from the surface of objects, and may not be complete due to occlusion and therange of sensors. Furthermore, sampling density, noise
patterns, and artifacts can vary in large amounts even in
adjacent regions, due to how the points are sampled by avariety of sensing techniques.
Now, let Pdet = {hi ∈ R4 : 1 ≤ i ≤ nh and hi ={nxi , nyi , nzi , di }} where nh is the number of planes detected from a point cloud. The ﬁrst 3 components correspond to the ith plane normal vector n̂i , and the last
di component to the plane normal distance to the origin.
Then, we deﬁne a plane detector as a function that inputs
a point cloud P and outputs a set of planes coeﬃcients
Pdet .

of ﬁnding a good set of samples from P, then to ensure
a probability z of RANSAC to withdraw a valid model itdeﬁnes k aslog(1 − z).
(1)k=log(1 − b)
For instance, suppose we have a noiseless point cloudwith 3 planes. If we want RANSAC to ﬁnd one planemodel with high probability, let’s say z = 0.99, w = 1/3
and n = 3, then we have to iterate up to 122 times perplane. However, if we have noise and other non-planar
points (outliers) then we have to modify w and set a lower
value, then the maximum iterations per plane increase
drastically. When we test for w ∈ {1/4, 1/5, 1/6} themaximum iterations per-plane increases from 122 to 292,
573, and 992 respectively.
This method is non–deterministic and it has two main
disadvantages when applying it to point cloud data.
First, to ﬁnd a model we have to assume that the in-

Figure 1 shows a diagram of a generic planes detector. It inputs a set of 3D points P with noise and sensor
artifacts and outputs a set of detected planes coeﬃcients.
P may consist of several planar and curved surfaces, andits density is not uniform.
A planes detector should consider the presence of theaforementioned nuisances. Therefore, it selects planar
points within a distance threshold , and the planes ofPdet are estimated from these using PCA or least squares.
If the ground truth planes of a point cloud P is Pgt ,ideally, the output of a planes detector should satisfy
Pdet = Pgt . Because Pgt is unknown by the detector,
it has to ﬁgure out how to generate hypothetical planes.
In the remaining parts of this section, we brieﬂy explain
the conventional methods algorithms.

liers/outliers ratio w is relatively low, since even in noiseless point clouds it should approximate the ratio between
the number of points of the smallest plane over the whole
point cloud, thus increasing the maximum iterations perplane. Second, since point clouds are multi-model, we
have to execute RANSAC several times. Clearly, thisadds ﬁnishing conditions diﬃcult to conﬁgure.
Furthermore, from a geometric perspective, RANSAC
is only testing planes models against point-model distance

2.2 RANSACRANSAC 18) is a simple but robust model ﬁtting algorithm, in computer vision is widely used for feature

Artifacts
Input

PointsPlanes detector

inliers, i.e., it is not aware of points curvatures, hence generating spurious models across noisy regions in the case ofrealistic point clouds. There has been application-driven
RANSAC improvements; therefore, in this work, we evaluate two geometry-aware RANSAC variations: Coarse-

OutputNoise

Fig. 1

Diagram of a generic plane detector

68IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

to-Fine RANSAC 3) and Eﬃcient RANSAC 14).
2.3

tional RANSAC methods, it requires point-wise normals
estimation, increasing its computational cost drastically.

Coarse-to-Fine RANSAC

2.5

An iterative Coarse-to-Fine RANSAC, hereinafter referred to as CFRANSAC, was developed to be usedfor virtual keypoints detection 3). This method employs
RANSAC iteratively to detect each model in a coarse-toﬁne approach, with Euclidean clustering spatially separating plane inliers in segments before reﬁning its coeﬃcients.
Geometry-aware RANSAC methods like this ﬁlter distance inliers by using point normals: a local feature ofpoint clouds that is not sensed and has to be estimated.
Apart from the distance threshold , it introduces an angular threshold θ between the hypothetical plane normal
vector and each point normal vector.
First the coarse step detects surfaces which are roughly
planar with big threshold values. However, when these

While the Standard Hough Transform performs numerous point-wise calculations, the Randomized Hough
Transform 19), RHT, exploits random sampling to accelerate the voting process.
Similar to RANSAC, in each iteration, RHT selects asample of 3 points within a given distance, then the spanning plane for those points is calculated using their crossproduct. Finally, the plane is voted once in an accumulator. Then, a plane model is detected by the algorithm
when a cell in the accumulator reaches a given amount ofvotes.
The RHT is not aware of the points geometry, but of theparameter space during the voting process. Thus, it does
not need normal vectors to compute planes coeﬃcientswith reasonable accuracy and speed. A drawback of thisapproach is that it is not aware of the points distributionsof the detected plane models.
Even though, the ball accumulator solved some accumulator problems 17), it is non–trivial to adjust the dis-surfaces are neighbors of similar planar surfaces RANSAC
tends to report inaccurate results. Therefore, the coarse
inliers are reﬁned by using RANSAC again but withstricter thresholds.
Since nearly parallel planes far from each other canbe erroneously detected as one, it can produce an error
when reﬁning the model coeﬃcients. Thus, planes inliers
are spatially clustered, where each cluster is treated asa diﬀerent model if they meet a speciﬁed points number
threshold τ  . Lastly, plane candidates are reﬁtted against
their inliers in a least squares sense.
2.4

Randomized Hough Transform

cretization parameters, among others who aﬀect directly
the precision of the algorithm, including measurements tocontrol the minimum and maximum distance between the
random samples and a restriction on the smallest eigenvalue size of the spanned planes.
The latter restrictions are not possible only with theplane voting process. Therefore, the RHT needs to beEﬃcient RANSAC

aware of the inliers points distribution by recurring to asorted clustering of the most prominent planes.
Since it is non deterministic, the ﬁnishing conditions
have to be chosen carefully. The algorithm will stop when
the remaining points go lower a given threshold, or if thealgorithm fails to build a plane model in a given amountof iterations. If these conditions are not properly conﬁgured, the algorithm may not ﬁnd all the most prominent
planes, or it may take longer to stop. Additionally, theEﬃcient RANSAC 14) (EFRANSAC) executes iteratively RANSAC on disjoint random subsets under theassumption that valid planes models will be detected inmost of the subsets. Therefore, it only accepts models
that are prominent in the number of inliers, and were
found in most of the disjoint subsets. Every acceptedmodel remove its inliers from the point cloud and theprocess iterates again until the ﬁnishing criteria is met.
Unlike CFRANSAC, EFRANSAC ﬁlters out points
with large deviations among their respective normal vectors during random sampling. It also ﬁlters points withnormals deviating more than a deﬁned angle from theplane normal vector. Moreover, it ﬁlters from the inliers
only the largest connected component on the plane modelby discretizing the inlier points translated to the plane coordinates. Finally, the candidate shapes are reﬁtted using
their inliers and removed from P.

performance of the RHT decrease drastically when there
is a large number of planes to be detected, or if the number of non-planar points is increased as shown in the halland arena model experiments of Borrmann et. al 17).

3.

Proposed Method

We focused on two main problems of conventional
planes detection methods: speed and robustness. The
major speed problem of conventional methods is that

Even though its core functionality is faster than conven-

69

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

they need to compute point-wise normals on overlappingneighborhoods

subset of the whole point cloud.

20)

.

3.1

Even though planar surfaces can be described locally,
it is more precise to describe them globally. Nonetheless,algorithms that work on the whole set of points are very
slow, e.g., the Standard Hough Transform.
Quantized noise from RGB-D and structured light sensors makes it even harder to describe locally planar surfaces, forcing algorithms to increase their search radius,
threshold or voxel size; thus preventing to detect smaller
objects.
Speed achieved by downsampling can also vanish small
surfaces and extremely deviate their normal vectors, as
the neighborhood has to be expanded way farther the

Hypothetical planes
coplanar voxels

extraction

from

For each voxel, its centroid c, unit normal vector n̂ anda planarity value P are calculated using the eigenvalues
λ1 ≤ λ2 ≤ λ3 and their corresponding eigenvectors vi ofthe covariance matrix, where:
n̂ =

v1
v1 

(2)

λ1
.
λ2

(3)

and
P =

After the voxel information is processed, the Sliding
Voxel walks through the occupied voxels and calculates
an overall score Sr for the whole neighborhood, i.e. 26

downsampling region to compute them.
Therefore, we propose a method that eﬃciently detects
planes via sliding voxels. Based on the Sliding Window
in images, the proposed method uses a 3D Sliding Window implemented with octree voxels. It travels throughoccupied voxels of a point cloud and calculates geometric
information about the points distribution using neighborvoxels.
Figure 2 outlines the proposed method algorithm.
First, we build an octree with voxel size Vs and calculate

neighbors plus the current voxel:
Sr =

n

i=1

Si : i ∈ Z +
≤27 ,

(4)

where Si is the planarity P of the i-th neighbor voxel.
As this score gets bigger, the less planar is the neighborhood. Therefore, a low score means the current voxel iscoplanar. For a more straightforward parametrization,
this planarity measure is normalized using the maximum
and minimum observed values:

tangent planes. This local plane ﬁtting provides us withcurvature information; therefore, we use sliding voxels toestimate the degree of coplanarity of each voxel.
Since coplanar voxels are more likely to be part of aprominent plane, we sort and mark them as hypotheticalplanes. Finally, planes are extracted from the validationof hypothetical planes against a geometrically enriched

n

Sr∗ =

Sr − min Sri
ni=1

max Sri

,

(5)

i=1

where n is the number of occupied voxels.
Figure 3 illustrates in warmer colors how this score canapproximate regions that have low probability of being
coplanar.
Coplanar voxels are further ﬁltered and sorted in ascending order by their score Sr∗ . The resulting coplanar

start

Build octree

subset Vp is used to ﬁnd planes in a centroids point cloudC, that was enriched with their normal vector to form anapproximate representation of non-overlapping tangential

Estimate per-voxel
tangent planes

Extract hypothetical planes
using sliding voxels

Global verication of
hypothetical planes

end

Fig. 2

Fig. 3

Flowchart of the proposed method

70

Point cloud of a room model, color represents aheatmap of the scores Sr of each voxel

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）planes.
3.2

Table 1 Proposed method parameters
Parameter
Description
Voxel size[m]Size of the octree leaves
Planarity threshold
Maximum value for Sr∗
Inliers threshold[m] Max. euclidean distance to plane
Max cosine distanceMax. cos+d of plane and inliers
Min plane size[#]
Min. support of planes

Global veriﬁcation

At this point, we can map a voxel centroid to their voxel
information such as its score Sr∗ , and the total numberof points inside the voxel. Simultaneously, its normal
vector n̂ and its centroid c are used to approximate thetangential plane pv at c, i.e., a geometrically enrichedversion of the original point cloud: C.
The purpose of this point cloud is to provide an eﬃcient
representation to validate hypothetical planes in a global
sense. For all coplanar voxels Vp , their tangential plane
pv is validated against c ∈ C using a decomposed planeto-plane distance: the Euclidean distance to each c andthe angular deviation between their normal vectors.
Ideally, the magnitude of the inner product betweenn normal vectors of a planar surface tend to be 1, i.e.
n1
i=0,j=0 |n̂i · n̂j | ≈ 1 where i = j. Therefore, inliersn
are selected by thresholding their Euclidean distance andthe normalized angular distance between the hypotheticalplane normal vector n̂hv , and the corresponding tangent
plane normal vector n̂tv . This metric is deﬁned by the

(6)

cos+
d =

2
cos−1 (|coss |) → [0, 1].
π(7)When cos+d takes the value of 1, the planes are orthogonal, and if it is 0, they are parallel. In the proposed
method, it is used as a parameter to decide whether aninlier will be rejected or not.
3.3

ground(d) K model(e) K ground truth (f) K noisy
truth

groundDatasets models

low enough, then the cluster is rejected. Otherwise, itis added to a list of detected clusters which includes its
plane coeﬃcients and inliers.
After all hypothetical planes are processed, they aresorted by their number of inliers. This allows us to reuse
C and remove inliers progressively from the most to theleast prominent plane.
Once the algorithm removes all the centroids inliersof the hypothetical planes, the process ﬁnishes and thecoss as follows
n̂hv · n̂tv
→ [−1, 1],n̂hv  n̂tv (b) R ground truth (c) R noisy
truth

Fig. 4

positive cosine distance cos+d calculated from its similarity

coss =(a) R model

list of clustered hypothetical planes becomes the detected
planes list. A summary of the proposed method parameters is shown in Table 1.

Planes extraction

Inliers of the global point cloud are sometimes too disperse and that can lead to false positives. To avoid thisissue, a fast 1-cluster euclidean clustering is performed
from the reﬁned inliers using a distance threshold deﬁned

4.

Experimental Results

4.1 Datasets and evaluation method
The datasets and ground truth planes used in the experiments can be seen in Fig. 4. Point clouds from Kinect
V1 sensor simulation were built using Blensor 1.0.18 21).
A room model 22) R was used to generate noisy andby the octree voxel size T h = 4Vs , which is twice themaximum possible distance between centroids.
From the resulting inliers, a cluster is constructed andits plane is calculated from these via Principal Compo-

noiseless simultaneous scans. To create a more complicated scenario, a kitchen model 23) K was scanned bytranslating and rotating 10 times the sensor in the simulation software. All scans were performed in world coordinates to correctly register the points via concatenation. The resulting point cloud has a wide variety of noise
patterns which makes diﬃcult even for humans to detect

nent Analysis. Since it is impossible to calculate a plane if
the number of inliers is less than 3, the plane coeﬃcientsare copied from the tangent plane of the voxel with thelowest score in the cluster.
If the cluster does not map enough points on the original point cloud, or if its planarity P (see Eq. (3)) is not

71

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

to Eq. (7)Table 2 Dataset informationName
BBDD[m] Points[#] Planes[#]R
6.09
295,144
9
R noisy
6.09
295,144K
12.19
249,348
14
K noisy
12.19
503,398

2
cos+
cos−1
d =
π

(10)

Also, a match is rejected if their ω > 15[deg] and δ >20[cm]. Thus, its result will be a list of matches K ofground truth planes associated with their best detected
plane match satisfying the above conditions.
det
Let Ki = {Pgti , Pi }, where i = {1, 2, . . . , M } and M
is the total number of matched planes (true positives),
then we can deﬁne the precision γ as

small planar surfaces in some locations. In addition to theregistered noisy scan, we also simulated a clean registration which does not have noise or quantization artifacts.
Due to registration, both K point clouds are incredibly
dense. Hence, voxel grid ﬁlter of leaf size 0.01[m] wasapplied to them.
Numerical information of the datasets can be seen in
Table 2, where BBDD stands for Bounding Box Diagonal
Distance, i.e. the length of a diagonal line that crosses thebounding box of the point cloud.
For both K and R, plane models were extracted directly from the polygons inside Blensor. For each planar
surface, plane coeﬃcients were generated from a polygon
over each plane by using its normal vector and barycenter. This ground truth data allows us to measure numerically and precisely the accuracy of each plane detection
method.
Regarding the processing time Tw , we used wall timeγ=

M,
|Pdet |

(11)and recall ζ of the detection asζ=M,|Pgt |(12)

where |Pdet | is the number of detected planes and |Pgt |is the number of ground truth planes. Therefore, theharmonic mean between precision and recall, i.e. the F1score 24) is deﬁned as
F1 =2
γ −1 + ζ −1

.

(13)

The range of the above metrics is [0, 1] where higher values mean better results.
On the other hand, the (true) eﬃciency Ef f measuressince its clock has more resolution. We note that we
excluded the time it takes to load a point cloud from disk,
also, we included the time for the required preprocessingof each algorithm. For CFRANSAC and EFRANSAC we
included the calculation of normal vectors inside Tw since

how fast the evaluated methods detected correct planes,i.e., the number of true positives M , over the processing
time TwM
Ef f =.
(14)
Tw

they rely heavily on them. For the proposed method, we
also included the neighborhood-aware octree preparationtime.
For each run, a list of ground truth planes Pgt andNote that since the conventional methods are non–deterministic, we used the average and standard devia-

detected planes Pdet coeﬃcients is prepared. Then, wetion of 50 runs.
Qualitative results are evaluated by segmenting thedeﬁned 2 error metrics. First, the angular error ω (indegrees) between the normal vector of an i-th groundgt
truth plane n̂gt
i ⊂ Pi and the normal vector of a j-th
⊂ Pdetdetected plane n̂detjj , is deﬁned as
gt
detω = 180 cos+d (n̂i , n̂j ) → [0, 180].

 pgt · pdet ji  gt   det   .
 pi  pj  output planes list Pdet of each method. Although inlier
points can be obtained from every method natively, theirpatterns can confuse the reader and generate an unfair
comparison. For instance, RHT results may look cleaner
while it mistakenly selects protruding noise patterns as(8)Second, the oﬀset diﬀerence δ between the i-th groundgt
det⊂
truth plane dgt
i ⊂ Pi and the j-th detected plane dj
det
Pj is



(9)
δ = |dgt | − |ddet
j | .

part of a planar surface, producing more signiﬁcant errors in the coeﬃcients of the resulting planes.
A native output of the proposed method is shown in
Fig. 5, for every plane in the detection result, the centroids inliers are shown, then for each plane, a diﬀerent color was chosen according to the Glasbey lookup
table 25), otherwise, points are kept black. Therefore, weigt
For a ground truth plane pgt
i ∈ P , the best detection∈ Pdet that minimizes theirmatch is a plane from pdet
j4
positive cosine distance cos+d as vectors in R , then similar
72

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Figure 6 shows the visual results of the noiseless point
clouds. Figure 6(a) and (f) depict the segmented groundtruth planes of R and K respectively. In Fig. 6(a), we
can visualize the ground truth planes selection for the R
model, avoiding slightly curved surfaces such as the backrest of the sofa and the pillow. Additionally, we avoided
selecting parallel planes that are not far from each other,
Fig. 5Detection example of the proposed method

because it would be sporadic to detect those structuresin noisy point clouds.
The upper row of Fig. 6 shows the results of the R
point cloud. Noticeably, the proposed method has zero
false positives in Fig. 6(b). RHT detected spurious planesin the bookshelf as well as several planes over the sofa
backrest in Fig. 6(c). CFRANSAC erroneously detected
the pillow and the sofa backrest as planar in Fig. 6(d).
Although EFRANSAC detected some planes with good

Table 3

Parameters of the proposed method in the evaluation experiments
ParameterK
K noisyR
R noisy
Voxel size[m]
0.14
0.16
0.12
0.12
Planarity threshold 0.001
0.1
0.02
0.02
Inliers threshold[m]
0.04
0.07
0.07
0.07
Max cosine distance 0.04
0.2
0.05
0.05
Min plane size[#]
500
2500
1300
1000

precision in Fig. 6(e), it tends to detect several spurious
planes while it failed to detect even a prominent planar
structure such as the wall on the left side.
The second row of Fig. 6 shows the results of the K
point cloud. Because of its lack of noise and higher
density, most methods performed accurately except for
RHT, which could not detect the dining table as seen in
Fig. 6(h). EFRANSAC performed fairly good in Fig. 6(j)because the higher density of K allows its random subsets to be more descriptive. However, its precision andrecall are lower than the proposed method (as describednumerically later).
Figure 7 shows the results of executing the evaluatedmethods over the noisy datasets. Figure 7(a) and (f) depict the segmented ground truth planes similar to Fig. 6.
The ﬁrst row of Fig. 7 illustrates the segmentation results on the R noisy point cloud. In Fig. 7(b), the proposed method detected most of the planar structures with

Table 4 Parameters values of the segmentation algorithm
ParameterK
K noisyR
R noisy
k[#]
50
100
50
100
d [m]
0.03
0.05
0.025
0.05
θ [deg.]
45
45
45
45

segmented the results only by looking at the standard
output of the evaluated algorithms: the list of planes coeﬃcients Pdet .
As a prerequisite, point-wise normal vectors are calculated within a support of k-neighbors via local plane ﬁtting. The segmentation algorithm share similarities withthe last step of the proposed method. For each plane∈ Pdet , plane-to-plane inliers are selected withinpdetja distance threshold of d and an angular threshold θ .
Planes are sorted in descending order by their amountof inliers. In that order, we extract and remove the inliers of each plane from the point cloud. This ensures themost prominent planes are segmented correctly and thesegmentation of false positives is minimum.
The parameters of the proposed method are shown in
Table 3 and for the segmentation are shown in Table 4.
4.2

high accuracy and no spurious planes, showing similar
segmentation patterns when comparing its results withthe ground truth. Noticeably, CFRANSAC detected several spurious planes in the noisiest region of the point
cloud in Fig. 7(d), whereas EFRANSAC detected lessspurious planes than CFRANSAC in Fig. 7(e).
The second row of Fig. 7 shows the segmentation re-Experiments results and discussion

The computer used to run the experiments has a CPU
Intel Core i7-6700K with 32GB of RAM, it runs on
Ubuntu 18.04.2 with PCL 1.8.1 26), CGAL 4.11 27), andsults on the K noisy point cloud. Figure 7(g) shows thatthe proposed method detected most of the planes while
having no false positives. In Fig. 7(h), it illustrates that
RHT was able to identify the most prominent planes;
nonetheless, it detected fewer planes than the proposed
method. CFRANSAC detected more false positives inFig. 7(i), while it failed to detect several planar struc-clang++ 3.8.0. The proposed method was implemented
using routines of the PCL library with O3 compiler optimizations.
Figure. 6 and Fig. 7 show the visual results of executing the segmentation algorithm over the ground truth
and the resulting planes of the evaluated methods.

73

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) R – Ground truth(b) R – Proposed(c) R – RHT(d) R – CFRANSAC(e) R – EFRANSAC(f) K – Ground truth(g) K – Proposed(h) K – RHT(i) K – CFRANSAC(j) K – EFRANSAC

Fig. 6

Planes detection inliers using the noiseless dataset(a) R – Ground truth(b) R – Proposed(c) R – RHT(d) R – CFRANSAC(e) R – EFRANSAC(f) K – Ground truth(g) K – Proposed(h) K – RHT(i) K – CFRANSAC(j) K – EFRANSAC

Fig. 7(a) Processing time Tw (lower is better)

Fig. 8Planes detection inliers using the noisy dataset(b) Precision γ (higher is better)(c) Recall ζ (higher is better)

Processing time, precision and recall of the evaluated methods

method visually, now we show objective assessment. Processing time, precision and recall are shown in Fig. 8.
Each bar represents the result of executing an evaluatedmethod over a point cloud of the dataset, where smallerbars denote better results. For the conventional methods we used the average of 50 executions and show theirtures. In Fig. 7(j), EFRANSAC detected slightly moreplanes than the proposed method because it got beneﬁted with the higher density of the K noisy point cloud.
However, its precision is inferior as described numericallylater.
While we conﬁrmed the robustness of the proposed

74

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) F1 score (higher is better)

Fig. 9(b) Eﬃciency Ef f (higher is better)

Accuracy and F1 score of the proposed method compared against the conventional methods

set of the proposed method is smaller and easier to conﬁgure. It has only 5 parameters while EFRANSAC,CFRANSAC, and RHT have 6, 12, and 16 parametersrespectively.

standard deviation as error bars.
Figure 8(a) shows the processing time Tw in logarithmic
scale. There we can conﬁrm that the proposed method isdrastically faster than the conventional methods in every
case.
The variations in terms of the mean angular and oﬀset
error of the evaluated methods are negligible; therefore,
we show their assessment based on more standard metrics
used in binary classiﬁcation tasks: precision γ, recall ζ
and F1 score.

5.

Conclusions and Future Works

Planes detection has numerous applications in 3D computer vision. However, its practical use was limited by theeﬃciency and accuracy of the detection methods. The
conventional methods are relatively accurate but ineﬃcient and non-deterministic, relying on random sampling
which does not always guarantee good results. RHT
achieved faster speeds by using random sampling andHough Voting, but it has several parameters diﬃcult to
conﬁgure, its precision can be profoundly aﬀected by the
resolution of the accumulator, and according to the literature, its performance decreases in the presence of several
planes models and non-planar points. Besides, RANSAC
extensions need the computation of normal vectors to detect planes correctly, which drastically reduces their eﬃciency since normal vectors are not raw data that can beacquired when sensing point clouds.
Therefore, we focused on solving the eﬃciency problem while keeping the accuracy high without needing
precomputed normal vectors or convergence mechanisms
like Hough voting. The proposed method is deterministicand has fewer parameters than the conventional methods;

Figure 8(b) shows the precision γ of the evaluatedmethods ( see Eq. (11) ). This metric measures how relevant were the detection results. Noticeably, the proposed
method precisely detected appropriate planes in all tested
datasets.
In Fig. 8(c), we show the recall ζ as deﬁned in Eq. (12).
This metric tells us how many of the ground truth planes
were detected by the evaluated methods. The proposed
method most of the time detected more ground truth
planes than the evaluated methods. Only in the R noisy
point cloud, its recall is paired with EFRANSAC; noticeably EFRANSAC had a very low precision (<0.6) asshown in Fig. 8(b).
In addition to the above metrics, we evaluated the F1score and eﬃciency Ef f of the evaluated methods as deﬁned in Eq. (13) and Eq. (14) respectively. Figure 9shows the results of applying these metrics on the experiment results. Figure 9(a) shows the overall precisionof the evaluated methods. The proposed method showsa superior precision in every case; it had better scoresthan the best execution of the conventional methods. Figure 9(b) shows the eﬃciency of the evaluated methods inlogarithmic scale. Here, we conﬁrm the proposed method
detects planes more accurately in a drastically more eﬃcient way.
Furthermore, it should be noted that the parameters

hence it is easier to conﬁgure. Moreover, the key to its
robustness and eﬃciency is the sliding voxel, which by utilizing its overlapping grid structure, it can locate coplanar
regions and calculate curvatures drastically faster.
By experiments with ground truth data and realistic simulations, we conﬁrmed that the proposed method
is accurate and drastically faster than the conventional
methods. Furthermore, it can achieve real time speedswith just a single CPU core. For future works, we are

75

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）considering to improve its recall even more by adding ex-

Segmentation Using RGB-D Cameras”, Robot Soccer World
Cup, pp. 306–317 (2011).

tra reﬁnement steps and adaptive thresholds.

16) S. Holzer, R. Rusu, M. Dixon, S. Gedikli, N. Nassir: “Adaptive
Neighborhood Selection for Real-Time Surface Normal Estimation from Organized Point Cloud Data Using Integral Images”,
Proc. of the IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), pp. 2684–2689 (2012).

References
1) G. Atanacio-Jiménez, J.-J. González-Barbosa, J.B. HurtadoRamos, F.J. Ornelas-Rodrı́guez, H. Jiménez-Hernández, T.
Garcı́a-Ramirez, R. González-Barbosa: “Lidar Velodyne HDL64E Calibration Using Pattern Planes”, International Journalof Advanced Robotic Systems, Vol. 8, No. 5, pp.70–82 (2011).

17) D. Borrmann, J. Elseberg, K. Lingemann, A. Nüchter: “The3D Hough Transform for Plane Detection in Point Clouds: A
Review and a New Accumulator Design”, 3D Research, Vol. 2,
No. 2, pp. 3 (2011).

2) C. Nguyen, S. Izadi, D. Lovell: “Modeling Kinect Sensor Noisefor Improved 3D reconstruction and Tracking”, Proc. of the
3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp. 524–530 (2012).

18) M. Fischler, R. Bolles: “Random Sample Consensus: A
Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography”, Communications of theACM, Vol. 24, No. 6, pp. 381–395 (1981).

3) K. Uenishi, J. Sandoval, I. Munetoshi, K. Tanaka: “VKOP:
3D Virtual Keypoint Detector Adapted to Geometric Structures and Its Feature Descriptor”, The Journal of the Instituteof Image Electronics Engineers of Japan: Visual Computing,
Devices & Communications, Vol. 46, No. 2, pp. 283–297 (InJapanese)(2017).

19) L. Xu, E. Oja, P. Kultanen: “A New Curve Detection Method:
Randomized Hough Transform (RHT)”, Pattern Recognition
Letters, Vol. 11, No. 5, pp. 331–338 (1990).
20) H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, W. Stuetzle: “Surface Reconstruction from Unorganized Points”, Proc.
of the 19th Annual Conference on Computer Graphics and Interactive Techniques, pp. 71–78 (1992).

4) D. Lin, S. Fidler, R. Urtasun: “Holistic Scene Understandingfor 3D Object Detection with RGBD Cameras”, Proc. of theIEEE International Conference on Computer Vision, pp. 1417–
1424 (2013).

21) M. Gschwandtner, R. Kwitt, A. Uhl, W. Pree: “BlenSor:
Blender Sensor Simulation Toolbox”, Proc. of the International
Symposium on Visual Computing, pp. 199–208 (2011).

5) Y. Zhang, W. Xu, Y. Tong, K. Zhou: “Online Structure Analysis for Real-Time Indoor Scene Reconstruction”, ACM Trans.
on Graphics (TOG), Vol. 34, No. 5, pp. 159:1–159:13 (2015).

22) A. Handa, T. Whelan, J. McDonald, A. Davison: “A Benchmark for RGB-D Visual Odometry, 3D Reconstruction and
SLAM”, Proc. of the IEEE International Conference on
Robotics and Automation (ICRA), pp. 1524–1531 (2014).

6) J. Weingarten, R. Siegwart: “3D SLAM Using Planar Segments”, Proc. of the IEEE/RSJ International Conference onIntelligent Robots and Systems, pp. 3062–3067 (2006).

23) Marela kitchen red&white, https://3dwarehouse.sketchup.com(2019).

7) K. Pathak, A. Birk, N. Vaskevicius, M. Pﬁngsthorn, S. Schwertfeger, J. Poppinga: “Online Three-Dimensional SLAM byRegistration of Large Planar Surface Segments and ClosedForm Pose-Graph Relaxation”, Journal of Field Robotics,
Vol. 27, No. 1, pp. 52–84 (2010).

24) C. Manning, P. Raghavan, H. Schütze, Introduction to Information Retrieval, Natural Language Engineering, Cambridge
University Press (2010).
25) C. Glasbey, G. Van der Heijden, V. Toh, A. Gray: “Colour Displays for Categorical Images”, Color Research & Application,
Vol. 32, No. 4, pp.304–309 (2007).

8) K. Pathak, A. Birk, N. Vaskevicius, M. Pﬁngsthorn, J. Poppinga: “Fast Registration Based on Noisy Planes with Unknown Correspondences for 3-D Mapping”, IEEE Trans. on
Robotics, Vol. 26, No. 3, pp. 424-441 (2010).

26) R. Rusu, S. Cousins: “3D Is Here: Point Cloud Library(PCL)”, Proc. of IEEE International Conference on Robotics
and Automation (ICRA), pp. 1–4 (2011).

9) K. Lenac, A. Kitanov, R. Cupec, I. Petrović: “Fast Planar
Surface 3D SLAM Using LIDAR”, Robotics and Autonomous
Systems, Vol. 92, pp. 197–220 (2017).

27) CGAL: Computational Geometry Algorithms Library, https://www.cgal.org (2019).

(Received September 13, 2019)10) F. Tarsha-Kurdi, T. Landes, P. Grussenmeyer: “HoughTransform and Extended Ransac Algorithms for Automatic
Detection of 3D Building Roof Planes from Lidar Data”, Proc.
of ISPRS Workshop on Laser Scanning 2007 and SilviLaser
2007, Vol. 36, pp. 407–412 (2007).

Jaime SANDOVAL(Student Member)He received his B.E. degree in Computer Systems Engineering from the Universidad delValle del Fuerte (Mexico) in 2009, and his
M.E. degree in Electrical and Electronic Engineering from Shinshu University in 2017.
Currently, he is a Ph.D. student in the Interdisciplinary Graduate School of Science andTechnology of Shinshu University with a major in Systems Development Engineering. His
research interests are 3D Point Clouds Processing, Computer Vision and Image Processing. He received IEVC2019 Excellent Paper
Award.

11) G. Gordon et al.: “The Use of Dense Stereo Range Data in
Augmented Reality” Proc. of the 1st International Symposiumon Mixed and Augmented Reality, pp. 14 (2002).
12) K. Okada, S. Kagami, M. Inaba, I. Hirochika: “Plane Segment Finder: Algorithm, Implementation and Applications”,
Proc. of the IEEE International Conference on Robotics andAutomation, Vol. 2, pp. 2120–2125 (2001).
13) S. Oßwald, J. Gutmann, A. Hornung, M. Bennewitz: “From 3D
Point Clouds to Climbing Stairs: A Comparison of Plane Segmentation Approaches for Humanoids”, Proc. of 11th IEEERAS International Conference on Humanoid Robots (Humanoids), pp. 93–98 (2011).
14) R. Schnabel, R. Wahl, R. Klein: “Eﬃcient RANSAC for PointCloud Shape Detection”, Proc. of Computer Graphics Forum,Vol. 26, No. 2, pp. 214–226 (2007).
15) D. Holz, S. Holzer, R. Rusu, S. Behnke: “Real-Time Plane76

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Kazuma UENISHI (Member)

Kiyoshi TANAKA (Fellow)

He received his B.E. degree in Computer Science in 2008, and his M.E. degree in Mathematics and Computer Science in 2014 fromthe National Defense Academy of Japan.
Currently, he is enrolled as a Ph.D. student
in the Interdisciplinary Graduate School of
Science and Technology of Shinshu University
with a major in Systems Development Engineering. He is pursuing research in 3D Point
Clouds Processing. He received IEVC2019
Excellent Paper Award, and Excellent Journal Paper Award from IIEEJ in 2018.

He received his B.S and M.S. degrees in Electrical Engineering and Operations Researchfrom National Defense Academy, Yokosuka,
Japan, in 1984 and 1989, respectively. In
1992, he received Dr. Eng. degree from Keio
University, Tokyo, Japan. In 1995, he joined
the Department of Electrical and Electronic
Engineering, Faculty of Engineering, Shinshu
University, Nagano, Japan, and currently heis a full professor in the academic assembly(Institute of Engineering) of Shinshu University. He is the Vice-President of Shinshu University as well as the director of Global Education Center (GEC) of Shinshu University.
His research interests include image and video
processing, 3D point cloud processing, information hiding, human visual perception, evolutionary computation, multi-objective optimization, smart grid, and their applications.
He is a project leader of JSPS Strategic Young
Researcher Overseas Visits Program for Accelerating Brain Circulation entitled Global
Research on the Framework of Evolutionary
Solution Search to Accelerate Innovation during 2013–2016. He is a member of IEEE, IEICE, IPSJ and JSEC. He is the former editor
in chief of Journal of the Institute of Image
Electronics Engineers Japan as well as IIEEJTransactions on Image Electronics and VisualComputing.

Munetoshi IWAKIRI (Member)He received his B. E. degree in Computer Science in 1993, and received his M. E. degreein Mathematics and Computer Science from
National Defense Academy of Japan in 1998.
In 1999, he joined Department of Computer
Science, National Defense Academy of Japan,
as a Research Associate. In 2002, he received
Dr. Eng. degree from Keio University, Tokyo,
Japan. In 2005 he became Lecturer and in2015 he became Associate Professor in thesame institution. He is pursuing research related to Multimedia Processing and Information Security. He is a member of the Information Processing Society of Japan.

77

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2019 --Pairwise Registration of Low Overlapping Unorganized 3D Point Clouds
Using Supervoxel Segmentation
Luis PERALTA† , Jaime SANDOVAL†† , Munetoshi IWAKIRI††† , Kiyoshi TANAKA††††† Graduate School of Science and Technology, Shinshu University ,†† Interdisciplinary Graduate School of Science and Technology, Shinshu University ,††† Department of Computer Science, National Defense Academy of Japan ,†††† Academic Assembly (Institute of Engineering), Shinshu University<Summary> Since its introduction, the Iterative Closest Points (ICP) algorithm has led to developinga wide range of registration methods, most of these variations of ICP itself. Notwithstanding the eﬀorts
on improving the speed and accuracy of ICP, these variations cannot correctly align point clouds which
overlapping ratio is considered low (under 40%) due to an inherited local minima convergence. Furthermore,
more advanced registration techniques that rely on point descriptors also cannot overcome this problem
because the tuning of their parameters tends to be volatile, which leads to making false point correspondences
and consequently failing to perform an accurate registration. In order to solve this problem, we proposea pairwise registration approach that does not entirely rely on point descriptors and leverages the local
minima convergence of ICP to correctly align 3D point clouds with overlapping ratios as low as about 20%.
Our method uses the supervoxel segmentation technique to divide the point clouds into subsets and ﬁnds
those which registration maximizes the overlapping ratio between correct correspondences in the full point
clouds. We veriﬁed the eﬀectiveness of the proposed method through tests in dense models and real-worldscan datasets.
Keywords: 3D point clouds, registration, low overlapping ratio, supervoxel segmentation

1.

is an iterative local optimization, these get stuck in local
minima. Hence, failing to correctly register point clouds
which initial state is not close to the global minima, or
have a low proportion of points in their shared surfaces.
This proportion is known as the overlapping ratio ξ andhas an essential role in the registration of partial overlapping point clouds. As ξ gets lower, it increases the probability of failure of conventional registration methods 2)

Introduction

3D point clouds are three-dimensional data point setsthat represent the shape of objects or scenes, which, inrecent years, their use has gained popularity in the areaof computer vision for the development of intelligent autonomous systems, such as indoor robots and self-drivingcars. Besides, applications such as surface measuring anddigital reconstruction for manufacturing, and geomechanics, to mention a few. The last two examples require theuse of registration, one of the essential tasks in point cloud

such as ICP, normal to plane registration (also known as
Normal-ICP or N-ICP) 3), and Levenberg-Marquardt ICP(LM-ICP) 4).
In order to address this issue, other approaches look
for T between point correspondences obtained from 3D
point descriptors 5). Nonetheless, the success of ﬁnding
correct correspondences depends on its descriptiveness,
which at the same time depends on the normal vectors of
the points. Besides, since the normal vectors are aﬀectedby ξ and no rule speciﬁes what a suitable normal vector
is, it is diﬃcult to rely only on point descriptors.
To the best of our knowledge, the volume of workthat addresses low overlapping ratio point clouds is very
scarce. The only few existing approaches work based onprocessing.
The registration task aims to ﬁnd the Euclidean motion, expressed by a transformation matrix T, that bestaligns two or more point clouds with diﬀerent reference
frames to the same one. Hence, generating a more detailed and complete representation of a model or scene.
The ICP algorithm, introduced by Besl and McKay 1), is the most widely used method to perform registrationof point sets. Its variations focus on improving the accuracy of alignment and expediting the convergence to thebest alignment. However, since the core operation of ICP

78

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Table 1 Conventional registration methods comparison
ICP
N-ICP
LM-ICP
Point-to-point Point-to-plane Point-to-pointφnearestnearestnearestfunctionneighborneighborneighbor
Umeyama’s
Umeyama’s
GradientT
least-squares
least-squares
descent andupdateestimation
estimation
Gauss-Newton

looking for the points or subsets that belong to the sharedsurfaces. Either by using point descriptors

6)

Diﬀerenceor by com-2)

putationally complex selection processes .
In this work, we propose a method capable of registering a pair of point clouds with overlapping ratios below
40%, which is not driven by point matches made fromdescriptors, and computational complexity allows it toﬁnd a solution considerably fast. The approach takes advantage of the local minima convergence of conventional
methods, by iteratively clustering the point clouds withsupervoxel segmentation 7), and looking among the produced subsets the pair which registration leads to the bestalignment.
The remainder of this paper organizes as follows. In
section 2, we introduce the mathematical representation
for pairwise registration of 3D point clouds and review
related work of partially overlapping point clouds registration. We present in detail our approach in section 3.
Details of the experiments, results, and discussion are in
section 4. Lastly, in section 5, we conclude and present
the focus for future work.

2.

Q. Moreover, T is a matrix of six Degrees Of Freedom(DOF), formed by rotation and translation parameters
around and along the three axes.
ICP performs the registration process in iterations, andat each iteration two steps take place:
1. Cpq are estimated by a point-to-point nearest neighbor function.
2. T is updated based on Umeyama’s least-squares estimation method 12).
The iterations are performed until the Cpq do not change,
and there is no possible update in T. Otherwise whenE reaches a speciﬁc Euclidean distance f known as the
Euclidean Fitness Epsilon.
N-ICP performs registration in a very similar manner to ICP, but it computes the normal vectors and uses
the spanning planes to estimate Cpq by a point-to-planenearest neighbor function.
LM-ICP is an improved ICP robust to the initial positions of the point clouds. It estimates Cpq the same

Related Works

The volume of published work relevant to pairwise registration of 3D point clouds is very extensive. However,
only a few approaches take into consideration the overlapping ratio. Thus, we only introduce related work that set
the base for Euclidean registration, address the overlapping ratio, and inspired to deﬁne our approach. Furthermore, we refer the reader to surveys on registration 8)–11)

way as ICP, but updates T based on the LevenbergMarquardt algorithm. Table 1 shows a comparison between these three conventional registration methods.
2.2 TrICP
In conventional methods, the φ functions deﬁne Cpq
taking into considerations the distance between all theto know more about the diﬀerent ICP variants and related approaches.

points. However, when the point clouds do not have thesame number of points or are partially overlapping thesefunctions may estimate a high number of false Cpq . In order to estimate a more accurate error for these cases the
Trimmed ICP (TrICP) 13),14) restricts E by taking into

2.1 Conventional registration methods
ICP is the most straightforward registration algorithm,
and its formulation is considered as the base for pairwise registration. For 3D point clouds, having two point
NQ
P
clouds, P = {pi }N
i=1 and Q = {qj }j=1 , named as source

consideration only the points that belong to the overlapping surface, proposing the Trimmed Mean Squared
Error

and target respectively. ICP looks for the optimal T thatapplied to P best aligns it to Q minimizing the error
E=

NP


2

wi qφi − T (pi )

(1)

T M SE =

i=1

1 
2
wi qφi − T (pi ) .

NP p ∈Pigiven by the Euclidean distance between point correspondences Cpq = {(T (pi ) , qφi )}M
i=1 . These correspondences
are deﬁned by a function φ that sets a weight wi as 1
if a point pi has a correspondence qφi , or 0 otherwise.

(2)ξ

Nevertheless, it deﬁnes the overlapping ratio as ξ =is the number of points in the subset Pξ ,which represents the overlapping part of P to Q. It meansthat Pξ deﬁnes the Cpq and the prior knowledge of theNP
NP , where NP

In ICP φ is the point-to-point nearest neighbor function;
hence, qφi represents the nearest neighbor of T (pi ) in

overlapping area is necessary.

79

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

posed Peralta et al. 2) becomes more suitable than the one

2.3 Practical registration of LiDAR scans
Due to their partial overlapping and complex struc-in TrICP. This method measures ξ between a pair of al-tures, 3D point clouds obtained from LiDAR scanners re-ready registered point clouds as the percentage of points

quire practical considerations to ease and accelerate theirregistration. Cai et al. 15) consider that in practice relative rotations are constrained to the azimuth in terrestrial systems. This consideration removes two DOF andallows to look for a T deﬁned only by translations along
the three axes and a single rotation around the vertical
axis.
Furthermore, to robustly estimate E and reject false
Cpq , they proposed to measure the Euclidean distanceerror only between the points that lay within an inlier
threshold ε that represents the scanning precision of thedevice. Nonetheless, this approach also relies on Cpq deﬁned by descriptors in a pre-processing step.

that are close enough to be considered part of the same
surface as


2M
ξ=
100 .
(3)
NP + NQ
In order to deﬁne Cpq , the method looks for each pointin the source the nearest neighbor in the target within aradius of a ε equal to two times the cloud resolution CR.
That is, for each point in the source which nearest neighbor lays within ε, there is a correspondence. Thus, thetotal of points with a correspondence considering both
point clouds is 2M . We note that CR is the mean Euclidean distance between all the points in a point cloud
and their nearest neighbor.
By combining a non-normalized TMSE and ε, it is possible to set an objective function that estimates the Euclidean distance error only between the correct overlapping points.

2.4

Low overlapping ratio registration methods
Despite the lack of registration methods focused on
low overlapping ratio point clouds, the few existing approaches have shown to obtain good results with or without relying on point descriptors. Wu et al. 6) rely on point

E (T | Cpq , ε) =2wi qφi − T (pi ) ≤ ε

(4)

i=1

descriptors to deﬁne the Cpq but proposes the idea thatin low overlapping ratio point clouds the shared surfaces
are more likely to be found at the boundaries, pruning
the search for Cpq to only these areas. The approach

Then, the T that produces the best alignment is theone that solves the optimization problem:E ∗ = max E (T | Cpq , ε)proposed by Peralta et al. 2) follows the same boundaries
ideas, but without relying on point descriptors. Instead,they proposed a random Hough voting search to look forthe point subsets that best align between each other, andwhich registration correctly aligns the full point clouds.

T(5)

In order to ﬁnd this transformation, we propose to exploit the local minima convergence of conventional methods by clustering the point clouds and look the pair ofsubsets (Pssk , Qssl ) that have a similar shape and size,

Nevertheless, due to its random search and inline construction of subsets for each point in the point clouds,
the processing time required to analyze a pair of point
clouds thoroughly is too long.

3.

M


and therefore its registration produces the T that applied to the full point clouds with a considerable small εaccomplishes Eq. (5). The core process of the proposed
method is composed of the following four steps, detailed
from section 3.2 to section 3.5:
1. Point clouds pre-processing.
2. Point subsets generation.

Proposed Method

3.1 Problem reformulation
In contrast to conventional registration methods, for
low overlapping point clouds, the goal is to estimate theerror of alignment considering only the points that be-

3. Point subsets combinations rejection and registration.
4. Transformations evaluation.

long to the overlapping surfaces. Thus, it is possible tosay that Eq. (2) is a better error estimation than Eq.
(1), making TrICP the ideal method for the problem.
However, due to the way ξ is determined and its priorknowledge, delimit the practicality of the method.
Because of the lack of prior knowledge of ξ in a realistic
situation, the overlapping ratio estimation method pro-

3.2

Point clouds pre-processing

As the ﬁrst step, we compute the normal vectors of P
and Q deﬁned by a normal radius Rn . Then, these normal
vectors are used to compute the FPFH descriptor 5) of
all the points in both point clouds, parameterized by an
FPFH radius Rf pf h .
80

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Despite computing the FPFH point descriptors, unlike

Algorithm 1 Subsets combinations rejection

registration methods that make the Cpq by matching de-Require: Source point subset Pssk .
Require: Target point subset Qssl .
Require: Corresponding descriptors PF P F Hssk of Pssk .
Require: Corresponding descriptors QF P F Hssl of Qssl .
1: for all pf pf h in PF P F Hssk do
2:
Find the 1st and 2nd nearest neighbors fromQF P F Hssl .
3:Compute the ratio between distances of the 1st and2nd nearest neighbors.
4:
if ratio < Cthr then5:
Increase Gs by 1.
6:
else
7:
Continue.
8: if Gs = 0 then9:
Do not register Pssk and Qssl .
10: else
11:
Register Pssk and Qssl .
12:
Save the corresponding T in VT .

scriptors, our approach uses the FPFH descriptor to givea geometrical description to point subsets, as is introduced in section 3.4.
3.3 Point subsets generation
Next, we divide both point clouds into subsets Psv =SS

Q
P{Pssk }SS
k=1 and Qsv = {Qssl }l=1 using the Voxel Cloud
Connectivity Segmentation, also known as supervoxel
segmentation 7).

The supervoxel segmentation clusters point clouds similarly to the 2D superpixels but generating a fast volumetric over-segmentation of 3D point clouds that adheres tothe boundaries. In detail, it divides the 3D space intoa voxelized grid space with a resolution of seed radius
Rseed , where each voxel has a resolution of voxel radiusand no available documentation states the reasons forthese changes, which are beyond the scope of this work.
However, the implementation still works as asserted by
the authors of supervoxel segmentation.

Rvoxel and sets a point seed for each cluster. An expansion distance deﬁned as

D2
D2
2
,
(6)
ΔP = λ c2 + μ 2s + DHiKm
3Rseed

3.4

Point subsets registration and combinations rejection

computes the range of the clusters from the point seeds.
Where λ controls the inﬂuence of color information asa Euclidean color distance Dc in the CIELab space, μcontrols the spatial distance Ds between points, and 
controls the geometric similarity of the points measured
by the FPFH descriptor.
Eq. (6) is the core that deﬁnes the point subsets inthe supervoxel segmentation method. Nevertheless, ourimplementation, as we introduce in section 4, is built with
the Point Cloud Library (PCL) 16) and its documentation
17)
simpliﬁes the expansion distance asD2ΔP = wc Dc2 + ws 2s + wn Dn2 .
(7)
3RseedAfterward, we look for transformations from the registration of subsets, which may be similar in size andshape. Therefore, all the possible subsets combinations(Pssk , Qssl ) are registered using LM-ICP, and the resulting transformations T are saved in a vector VT , which

Unlike Eq. (6), the computation of Eq. (7) is in afeature space consisting of the color Euclidean distanceDc from the normalized RGB space, the space distancemore similar and geometrically related are the subsets.
The rejection of the point subsets process is summarized in Algorithm 1. Starting with a pair of subsets

Ds normalized by Rseed , and the normal distance Dn thatmeasures the angle between normal vectors. Moreover,
wc , ws , and wn replace λ, μ, and  respectively, but still(Pssk , Qssl ) and since the points in P and Q are directlysize is SSP × SSQ if all the combinations are registered.
However, registering all the combinations implies a probability of obtaining transformations which subsets do notcorrespond to geometrically related surfaces. Thus, itis necessary to avoid registering geometrically unrelated
subsets. To achieve this, we set a degree of geometricsimilarity Gs , which measures how similar are a pair ofsubsets by counting the point descriptor correspondences
between them. Hence, the higher the value of Gs , therelated their previously computed FPFH descriptors, it ispossible to make clusters of these descriptors in the sameorder as resulted in the supervoxel segmentation for theare user parameters which represent weights that control

point clouds. Thus, there is a one-to-one related descriptors subset pair (PF P F Hssk , QF P F Hssl ) that describes theoriginal point subsets pair, and is used to perform therejection of subsets combinations. First, descriptor correspondences are estimated in the same manner as Buchet al. 18) and Lowe 19), to measure the geometric similar-

the eﬀect of the distances.
We note that the actual implementation of the supervoxel segmentation available in PCL version 1.8 deﬁnes
the expansion distance as
ΔP = wc Dc +ws Ds
+ wn Dn ,
Rseed

(8)

81

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）ity between the subsets. For each point descriptor pf pf h

Algorithm 2 Iterative registration method

in PF P F Hssk , the ﬁrst and second nearest neighbors in

Require: Source point cloud P .
Require: Target point cloud Q.
1: Compute the normal vectors of P .
2: Compute the normal vectors of Q.
3: Compute the FPFH descriptors of P .
4: Compute the FPFH descriptors of Q.
5: Rseed = Rmax
6: while Rseed > Rvoxel do
P
7:Divide P into subsets, Psv = {Pssk }SS
k=0 .
SSQ
8:
Divide Q into subsets, Qsv = {Qssj }j=0 .
9:for all Pssk in Psv do
10:
for all Qssl in Qsv do
11:
Run Algorithm 1.
12:for all T in VT do
13:
Evaluate T on Eq. (4).
14:
if T accomplishes Eq. (5) then15:
Tb = T
16:
Save Tb in VTb .
17:
else
18:
Continue.
19:
Rseed = Rseed − τ
20: for all Tb in VTb . do
21:
Evaluate Tb on Eq. (4).
22:
if Tb accomplishes Eq. (5) then.
23:
T o = Tb
24:
else
25:
Continue.
26: Apply To to P .

QF P F Hssl are found. If the ratio between their distances
is lower than a correspondence threshold Cthr , the ﬁrst
nearest neighbor is considered a correspondence, and Gs
increases by one. Then, once all point descriptors in
PF P F Hssk are evaluated, the pair (Pssk , Qssl ) is rejected
if Gs has a value of zero, or registered if otherwise.
For the registration of the subsets, we chose LM-ICP
due to its robustness to the initial positions. A property
that makes this algorithm ideal to obtain the registrationtransformation of subsets which may be virtually partof the same surface in the model, but have a diﬀerent
orientation in their respective point clouds.
3.5

Transformations evaluation

Lastly, the transformation Tb that achieves the bestalignment is deﬁned by applying one by one all the T
saved in VT to P and evaluating with Eq. (4). Then, Tb
is the T that accomplish Eq. (5).
3.6Iterative process

All the evaluated transformations come from the registration of considerably similar point subsets. Thus thesize of these subsets plays a signiﬁcant role in the proposed method. Rseed controls the size of these point sub-4.

Experimental Results

The proposed method was implemented in C++ 11 using the Point Cloud Library (PCL) 16) version 1.8. Furthermore, for parallel processing, we implemented thesteps for the subsets rejection, registration, and evaluation of transformations using OpenMP.

sets and should be more signiﬁcant than Rvoxel . The
larger is Rseed , the less but bigger are the generated point
subsets, and since there is no deﬁned optimal value for
Rseed , it is also necessary to ﬁnd the value that generates

4.1

the point subsets that lead to the optimal transformation
To .

Datasets and experimental setup

The utilized datasets consist of dense model point
clouds, and sparse scene point clouds obtained from laser
scanners. The dense point clouds are pairs of the mod-In order to achieve it, we iterate over steps 2 to 4 of thecore process by decreasing the value of Rseed at a step rateτ , from a maximum Rmax to a minimum Rmin . Whichels Bunny, Dragon, Happy Buddha, and Armadillo fromthe Stanford 3D Scanning Repository 20). The sparse

Rmax can be as signiﬁcant enough to at least generatea couple of subsets at the ﬁrst iteration, but due to thespatial relationship between supervoxels and the voxel
grid in the segmentation process Rmin = Rvoxel + τ .
At each iteration, Rseed has a diﬀerent value to generate

point clouds are pairs of the scene Stairs from the ASL
Datasets 21). Each pair of point clouds was chosen basedon having an ξ below 40%, and the corresponding groundtruth alignment available. Table 2 shows the speciﬁc
datasets for each model and scene, the source-target ar-point subsets in both point clouds and register them as
described in section 3.4. Then as explained in section
3.5, the resulted transformations are evaluated to ﬁnd
Tb . All the Tb from each iteration are saved in a vector
VTb in order to perform the same evaluation with Eq.

rangement, their number of points, and corresponding ξ.
We ran the proposed method with the diﬀerent chosen
datasets while recording the corresponding registration
metrics E and ξ of Tb at each iteration, as well as the(4) and deﬁne the optimal transformation To as the bestfrom the best. That is, To is the Tb that accomplish Eq.

processing time. Additionally, in order to have a metricto estimate the error of alignment between the points thatgenuinely overlap the Mean Squared Error with Penalty
was set as(5). Algorithm 2 summarizes the complete iterative
registration method.

82

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Table 2 Datasets and corresponding ξModel/Scene
Datasetsξ (%)
P : bun000
NP : 40,256 points
Bunny
30.90
Q: bun270
NQ : 31,701 points
P : dragonStandRight 0
NP : 41,841 points
Dragon29.64
Q: dragonStandRight 288
NQ : 24,573 points
P : happyStandRight 0
NP : 78,056 points
Happy
19.88
BuddhaQ: happyStandRight 288
NQ : 72,346 points
P : ArmadilloStand 0
NP : 28,220 points
Armadillo
26.66
Q: ArmadilloStand 270
NQ : 24,034 points
P : Hokuyo 0
NP : 181,077 points
Stairs33.04
Q: Hokuyo 3
NQ : 187,959 points

M SEp =

dpq =

NP
1 
wi (dpq ≤ ε)NP i=1qφi − T (pi ) 2 ,1,

if wi = 1
if wi = 0

Table 3 Parameters values utilized in the experiments
Parameter Value [CR] Parameter Value [CR]
Rn
10
ε
2
Rf pf h10
Rvoxel
2.5
wc
0
Rmax
100
ws
2.5
τ
1
wn
0.5
Cthr
0.1

large number of subsets, when it is desired to generate
only a few at the ﬁrst iteration. Hence, these parameters
were scaled-up by 10.
Additionally, we ran the conventional registration
methods with the same datasets to compare the results
against the ground-truth and the proposed method.
4.2

Results and discussion

Being the source and target clouds depicted in red andblue, respectively, Table 4 shows the registrations obtained by the ground-truth, the proposed method, andthe conventional methods for all the datasets.
(9)

Visually, unlike conventional registration methods, theproposed method registers the point clouds keeping theshape of the models and the scene, generating an alignment close to the ground-truth. In the cases of Bunny,
Dragon, Happy Buddha, and Stairs, the resulting alignments are almost perfect. Nevertheless, the lack of descriptiveness in the overlapping surfaces of the Armadillo
datasets causes an alignment farther from the groundtruth. Using the same color pattern as in Table 4, Fig.
1 depicts the subsets that lead to To for all the datasets.

(10)

It measures the Mean Squared Error between the truly
overlapping points, but unlike T M SE, instead of not considering the points of the not overlapping parts, M SEp
gives a penalty of 1 for each pi ∈ P that does not havea correspondence qφi ∈ Q within ε (see Eq. (10)). Since
in low overlapping conditions, the overlap is signiﬁcantly
small, M SEp allows to estimate small values of error ofalignment for point clouds that are correctly registered,and vice-versa.
For the dense model point clouds, the experiments were
performed with the parameters shown in Table 3 interms of CR, except for Cthr , which represents a ratio

By comparing the Armadillo ground-truth in Table 4 andthe position of the subsets in Fig. 1 (d) it is possible to notice the reason for the resulting alignment in this model.
In the ground-truth, most of the overlapping surfaces arein the round part at the back of the Armadillo. This surface is mostly round and similar to a sphere; it does nothave any other geometric description than being round.
Thus, at step 3, the core process allows the registration of
all the combinations coming from this part, without considering if these arise from opposite sides. An analogy tothis eﬀect is the registration of the two halves of a spheresince both do not have any other geometric description∈ [0, 1]. Parameter wc was set as 0 because the datasets
do not have color information, and Cthr as 0.1 to guarantee to ﬁnd correct descriptor correspondences. Rmax andτ , were deﬁned large enough to generate at least two subsets at the ﬁrst iteration, and small enough to gradually
evaluate a wide range of sizes for the subsets. Moreover, εwas set with the same value proposed by Peralta et al. 2)
to ensure the estimation of ξ between the overlapping
points.
The experiments with the sparse scene point clouds
kept the same values for most of the parameters, except
for Rvoxel and Rmax . Since the scale of the stairs scene isthan their roundness, the ﬁnal registration would be abowl-like shape instead of a sphere.
In the ground-truth column of Table 4, the metrics E,
ξ, and M SEp have the values for the perfect alignment.
Therefore, as the values of these metrics measured in theother registration methods become closer to the ones in
the ground-truth, the more similar is the obtained registration to the perfect alignment. In this manner, theseveral times larger than the models, by using the samevalues the supervoxel segmentation started generating a83

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Table 4 Registration comparison between the ground-truth, the proposed method, and conventional methods
Unregisteredpoint clouds

Ground-truth

ProposedmethodICP

N-ICP

LM-ICPBunny

E = 6.89
ξ = 30.99%
M SEp = 0.7239

E = 4.74
ξ = 19.20%
M SEp = 0.8284

E = 1.26
ξ = 5.06%
M SEp = 0.9547

E = 1.55
ξ = 6.21%
M SEp = 0.9444

E = 1.25
ξ = 4.99%
M SEp = 0.9554

DragonE = 5.48
ξ = 29.64%
M SEp = 0.7648

E = 5.62
ξ = 30.18%
M SEp = 0.7605

E = 1.41
ξ = 6.27%
M SEp = 0.9502

E = 1.45
ξ = 6.24%
M SEp = 0.9504

E = 1.43
ξ = 6.36%
M SEp = 0.9495

Happy Buddha

E = 6.08
ξ = 19.88%
M SEp = 0.8084

E = 6.78
ξ = 22.70%
M SEp = 0.7813

E = 3.11
ξ = 9.55%
M SEp = 0.9080

E = 3.41
ξ = 10.67%
M SEp = 0.8971

E = 2.99
ξ = 9.20%
M SEp = 0.9113

Armadillo

E = 4.56
ξ = 20.66%
M SEp = 0.7533

E = 4.03
ξ = 23.63%
M SEp = 0.7813

E = 0.64
ξ = 3.67%
M SEp = 0.9660

E = 0.77
ξ = 4.61%
M SEp = 0.9573

E = 0.64
ξ = 3.63%
M SEp = 0.9663

Stairs

E = 896.42
ξ = 33.04%
M SEp = 0.6682

E = 812.27
ξ = 26.31%
M SEp = 0.7363

E = 132.32
ξ = 4.09%
M SEp = 0.9589

E = 96.11
ξ = 2.93%
M SEp = 0.9706

E = 128.05
ξ = 3.87%
M SEp = 0.9611

84

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) Bunny(b) Dragon(c) Happy Buddha(d) Armadillo(e) Stairs

Fig. 1 Point subsets that lead to To

Table 5 Optimal Rseed and processing times
Model/Scene
Rseed [CR] tT (sec) tM (sec)Bunny70
444.74
2.47
Dragon58
136.88
1.04
Happy Buddha
31
552.603.08
Armadillo
31
94.75
0.73
Stairs927
5,925.2
0.98

cess the model datasets fully and ﬁnd a solution and does
not require user intervention to compare the results withthe ground-truth. On the other hand, due to the scale ofthe sparse scene datasets, the scaled-up values of Rmax
and Rvoxel impact on tT since more values of Rseed haveto be processed.

proposed method also quantitatively surpasses conven-5.

tional methods. It is noticeable how the metrics in the
proposed method are signiﬁcantly closer to the groundtruth in all the datasets. Although there are still somenoticeable diﬀerences, as in the case of Bunny, these do
not indicate a wrong result, but that there is still room for
improvement in the alignment, which can be achieved by
a ﬁne registration method. Thus, the proposed approach

Conclusions

By providing enough shape descriptiveness in the overlapping areas, the proposed approach is capable of ﬁndingtransformations that register a pair of point clouds witha low overlapping ratio close to the perfect alignment,close enough to consider it a coarse registration method.
Therefore, it may be useful to perform multi-view registration (i.e., registration of more than two point clouds),

can be considered a coarse registration method.
Regarding the optimal Rseed and processing time, Ta-

but using a ﬁne registration method between every two
datasets to avoid carrying out the residual error of misalignment.
For future work, we will focus on improving the dependence to the size and shape of the point subsets, as wellas the indirect dependence to point descriptors to ﬁnd re-ble 5 shows the resulting optimal sizes, as well as corresponding total tT and median tM processing times. In
the same way as Peralta et al. 2), this approach is basedon a subsets registration principle. However, as opposedto that method, this one takes only a few minutes to pro-85

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

lated surfaces, by deﬁning better techniques to describe

lation in Human Environments”, KI - Künstliche Intelligenz,
Vol. 24 (2009).

the points that belong to the overlapping areas. More-17) pointclouds.org, Clustering of Point Clouds into Supervoxels,
http://pointclouds.org/documentation/tutorials/supervoxel
clustering.php (2019).

over, we will evaluate the adaptability of the method toreal-time applications.

18) A.G. Buch, D. Kraft, J. Kamarainen, H.G. Petersen, N.
Krüger: “Pose Estimation Using Local Structure-speciﬁc
Shape and Appearance Context”, Proc. of IEEE International Conference on Robotics and Automation (ICRA 2013),
pp.2080–2087 (2013).

References
1) P.J. Besl, N.D. McKay: “A Method for Registration of 3-D
Shapes”, IEEE Trans. of Pattern Analysis and Machine Intelligence, Vol. 14, No.2, pp.239–256 (1992).

19) D.G. Lowe: “Distinctive Image Features from Scale-invariant
Key-points”, International Journal of Computer Vision, Vol.
60, No. 2, pp.91–110 (2004).

2) L. Peralta, J. Sandoval, M. Iwakiri, K. Tanaka: “A Preliminary Study on Low Overlapping Unorganized Point Clouds
Registration Using Hough Voting”, Proc. of IIEEJ Technical
Meeting, Vol. 288, pp.8–13 (2019).

20) M.
Levoy,J.
Gerth,
B.
Curless,
K.
Pull:
“The
Stanford
3D
Scanning
Repository”,
http://graphics.stanford.edu/data/3Dscanrep (2005).

3) Y. Chen, G. Medioni: “Object Modeling by Registration ofMultiple Range Images”, Proc. of IEEE International Conference on Robotics and Automation (ICRA 1991), Vol. 3,
pp.2724–2729 (1991).

21) F. Pomerleau, M. Liu, F. Colas, R. Siegwart: “Challenging
Data Sets for Point Cloud Registration Algorithms”, The International Journal of Robotics Research, Vol. 31, No. 14, pp.
1705–1711 (2012).

4) A. Fitzgibbon: “Robust Registration of 2D and 3D Point Sets”,
Image and Vision Computing, Vol. 21, pp.1145–1153 (2002).

(Received September 13, 2019)5) R. Rusu, N. Blodow, M. Beetz: “Fast Point Feature Histograms (FPFH) for 3D Registration”, Proc. of IEEE International Conference on Robotics and Automation (ICRA 2009),
pp.3212–3217 (2009).

Luis PERALTA (Student Member)He received the B.Eng. degree in Mechatronics in 2014 from Instituto Tecnologicode Celaya, Mexico, and the M.Eng. degree in Electronic Information Systems in2019 from Shinshu University, Japan. He
is currently working towards an Eng.D. degree in Information Communication Systems
at Shinshu University. His research interests
include computer vision, virtual reality, machine learning, and robotics.

6) Y. Wu, W. Wang, K. Lu, Y. Wei, Z. Chen: “A New Method for
Registration of 3D Point Sets with Low Overlapping Ratios”,
Procedia CIRP (13th CIRP conference on Computer Aided
Tolerancing), Vol. 27, pp.202–206 (2015).
7) J. Papon, A. Abramov, M. Schoeler, F. Wörgötter: “Voxel
Cloud Connectivity Segmentation - Supervoxels for Point
Clouds”, IEEE Conference on Computer Vision and Pattern
Recognition (CVPR 2013), pp.2027–2034 (2013).
8) J. Salvi, C. Matabosch, D. Foﬁ, J. Forest: “A Review of Recent Range Image Registration Methods with Accuracy Evaluation”, Image and Vision Computing, Vol. 25, No. 5, pp.578–
596 (2007).

Jaime SANDOVAL

9) J. Santamarı́a, O. Cordon, S. Damas: “A Comparative Studyof State-of-the-art Evolutionary Image Registration Methods
for 3D Modeling”, Computer Vision and Image Understanding,Vol. 115, pp.1340–1354 (2011).

(Student Member)He received the B.Eng. degree in Computer
Systems in 2009 from Universidad del Valledel Fuerte, Mexico, and his M.Eng. degree inElectrical and Electronic Engineering in 2017from Shinshu University, Japan. Currently,
he is a Ph.D. student at the Interdisciplinary
Graduate School of Science and Technologyof Shinshu University with a major in Systems Development Engineering. His researchinterests are 3D point cloud processing, computer vision, and image processing. He received IEVC2019 Excellent Paper Award.

10) F. Pomerleau, F. Colas, R. Siegwart, S. Magnenat: “Comparing ICP Variants on Real-World Data Sets”, Autonomous
Robots, Vol. 34, pp.133–148 (2013).
11) S. Rusinkiewicz, M. Levoy: “Eﬃcient Variants of the ICP Algorithm”, Proc. of Third International Conference on 3-D Digital
Imaging and Modeling, pp.145–152 (2001).
12) S. Umeyama: “Least-squares Estimation of Transformation
Parameters Between Two Point Patterns”, IEEE Trans. on
Pattern Analysis and Machine Intelligence, Vol. 13, No. 4,
pp.376–380 (1991).
13) D. Chetverikov, D. Svirko, D. Stephanov, P. Kresk: “The
Trimmed Iterative Closest Point Algorithm”, Object Recognition Supported by User Interaction for Service Robots, Vol.
3, pp.545–548 (2002).
14) S. Xu, J. Zhu, Z. Jiang, Z. Lin, J. Lu, Z. Li: “Multi-view Registration of Unordered Range Scans by Fast Correspondence
Propagation of Multi-scale Descriptors”, PLOS ONE, Vol. 13,
No. 9, pp.1–18 (2018).
15) Z. Cai, T. Chin, A.P. Bustos, K. Schindler: “Practical Optimal
Registration of Terrestrial LiDAR Scan Pairs”, ISPRS Journalof Photogrammetry and Remote Sensing, Vol. 147, pp.118–131(2019).
16) R. Rusu: “Semantic 3D Object Maps for Everyday Manipu-

86

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Munetoshi IWAKIRI (Member)

Kiyoshi TANAKA (Fellow)
He received a B.Sc. and M.Sc. degree in Electrical Engineering and Operation Research in1984 and 1989, respectively, from the National Defense Academy of Japan. In 1992, hereceived the Dr.Eng. degree from Keio University, Japan. In 1995, he joined the Department of Electrical and Electronic Engineering, Faculty of Engineering of Shinshu University, Nagano, Japan, and currently, he is afull professor at the academic assembly (Institute of Engineering) of Shinshu University.
He is the Vice-president of Shinshu University as well as the director of the Global Education Center (GEC) of the same institution.
His research interests include image and video
processing, 3D point cloud processing, information hiding, human visual perception, evolutionary computation, multi-objective optimization, smart grid, and their applications.
He is a project leader of the JSPS Strategic Young Researcher Overseas Visits Program for Accelerating Brain Circulation entitled Global Research on the Framework of
Evolutionary Solution Search to Accelerate
Innovation during 2013–2016. He is a member of IEEE, IEICE, IPSJ, and JSEC. He is
the former editor in chief of the Journal of
the Institute of Image Electronics Engineers
Japan as well as IIEEJ Transactions on Image
Electronics and Visual Computing.

He received the B.Eng. degree in Computer
Science, and the M.Eng. degree in Mathematics and Computer Science from the National Defense Academy of Japan in 1993 and1998, respectively. In 1999, he joined theDepartment of Computer Science of the National Defense Academy of Japan, as a Research Associate. In 2002, he received theDr.Eng. degree from Keio University, Japan.
The institution where he became a lecturer
and associate professor in 2005 and 2015, respectively. He is a member of the Information
Processing Society of Japan and pursues research related to multimedia processing andinformation security.

87IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2019 --Weakly-Supervised Learning for Continuous Sign Language Word Recognition
Using DTW-Based Forced Alignment and Isolated Word HMM Adjustment
Natsuki TAKAYAMA† (Member) , Hiroki TAKAHASHI†, †† (Member)† Graduate School of Informatics and Engineering, the University of Electro-Communications ,†† Artiﬁcial Intelligence Exploration Research Center<Summary> The reduction of the manual work of annotation is an essential part of sign language
recognition research. This paper describes one weakly-supervised learning approach for continuous sign
language word recognition. The proposed method consists of forced alignment based on dynamic time
warping and isolated word hidden markov model adjustment using ‘embedded training’. While the proposed
forced alignment only requires one manual annotation for each isolated sign language word, it can generate
suﬃcient quality of the annotation to initialize isolated word hidden markov models. ‘Embedded training’
adjusts initial hidden markov models to recognize continuous sign language words using only ordered wordlabels. The performance of the proposed method is evaluated statistically using a dataset that includes
5,432 isolated sign language word videos and 4,621 continuous sign language word videos. The averagedalignment error of the proposed forced alignment was 4.02 frames. The averaged recognition performancesof the initial models were 74.82% and 91.14% in the signer-opened and trial-opened conditions, respectively.
Moreover, the averaged recognition performances of the adjusted models were over 65.00% for all conditions.
The evaluation shows signiﬁcant improvements compared to the previous weakly-supervised learning.
Keywords: DTW, embedded training, HMM, sign language recognition, weakly-supervised learning1.
1.1

videos. Figure 1 (a) shows an example of annotation for
an isolated sign language word “HIRU,” which means “afternoon” in Japanese Sign Language (JSL). Figure 1 (b)shows that of continuous sign language words “HIRU”
and “AISATSU.” “AISATSU” means “greeting,” andthese continuous sign language words make up the sentence “Good afternoon” in Japanese. The red bars indicate the borders of the motion units. Each label inthe units describes the type of motion. “Static” indicatesstatic posture without sign motions. Generally, the manual work of annotation is time-consuming. In particular,

Introduction

Background

Sign language recognition is an important research
topic to improve communication between native signers
who use sign language in their communication, and speakers. Continuous eﬀorts of about thirty years in the ﬁeld
make it possible to recognize continuous sign language
words1)–4). These successes are, however, dependent on
the availability of sign language corpora.
Sign languages are commonly represented by hand motions and shapes, as well as non-manual signals that include posture, facial expressions, gazes, and mouth motions. The representations and grammar of sign languages are, however, diﬀerent in each community, andno uniﬁed rules have been established to describe them.

the annotation of continuous sign language word videos
is a diﬃcult task even to a professional annotator because
there are “Transition” motions between words as shownin Fig.1 (b). For these reasons, it is essential to reducethe manual work of annotation to progress the researchof sign language recognition.
As one of the solutions for the above problem, weaklysupervised learning, which trains models with a simple
and limited amount of manual annotation, has received

Therefore, the sign language corpora for sign language
recognition should be built individually.
The sign language corpora for recognition require largescale videos and annotation. The annotation for sign
language recognition is composed of deﬁned time ranges
based on motion units and their recognition labels.
Figure 1 shows examples of annotations to sign language

much attention in recent years. One of the standard approaches of weakly-supervised learning for sign language
recognition is training with ordered word labels2). The

88

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) Annotation of an isolated sign language word(b) Annotation of continuous sign language words

Fig. 1 Examples of annotation to sign language videos

ordered word labels are a word array without time range
information. Previous weakly-supervised learning does
not require the deﬁnition of time ranges; therefore, theannotation work can be reduced to a reasonable amount.
The previous methods, however, rely on ‘ﬂat-start’ initialization which segments a video uniformly according tothe ordered word labels, and they train initial recognition
models using the uniformly segmented videos. ‘Flat-start’
initialization often negatively aﬀects recognition performance when the errors of the initial segmentation are
signiﬁcant.
1.2

non-professional annotator.
Forced alignment is a process to ﬁnd an alignment ofa time sequence to ordered word labels. Although the
popular forced alignment in sign language recognition isViterbi alignment based on HMM with ordered word labels3), we employ that of DTW. While the HMM-based
forced alignment requires a certain amount of manual
annotation to build the pre-trained models, DTW-based
forced alignment can be available with one manually annotated reference data for each word.
We apply the proposed forced alignment to isolated sign
language word videos and train initial word HMM using
the generated annotation. After the initial training, weapply ‘embedded training’, which is also known as concatenated training, to the initial HMM. ‘Embedded training’ requires only ordered word labels; therefore, diﬃcult
annotation is avoided.
The proposed initialization process provides a signiﬁ-

Motivation and proposed method

Hence, in this paper, we propose one approach to
weakly-supervised learning for continuous sign language
word recognition using forced alignment based on Dynamic Time Warping (DTW)5) and isolated word Hidden Markov Model (HMM) adjustment using ‘embedded
training’ 6),7). The primary motivation of this research iscant improvement of recognition performance over ‘ﬂatstart’ initialization, and the recognition performance ofthe proposed method is close to that of ‘bootstrap’ initialization which utilizes all manual annotations of theisolated sign language word videos. This paper provides
experimental comparisons to show the superiority of the
proposed method over ‘ﬂat-start’ and ‘bootstrap’ initialization.
The remainder of this paper is organized as follows. In

building high-performance continuous sign language word
recognition models with as little manual annotation as
possible. The proposed weakly-supervised learning only
requires one manual word-level annotation for each isolated sign language word, as well as ordered word labelsof continuous sign language words. The word-level annotation deﬁnes the motion units based on the representing
sign language words and gives them word labels, as shownin Fig.1 (a). The word-level annotation of the isolated
sign language words is simple and easier than that of theannotation for continuous sign language words. This styleof annotation can be expected to be feasible even for a

Section 2, we introduce the previous weakly-supervised
learning approaches of sign language recognition. In Section 3, we describe our research settings. In Section 4,
we explain the proposed weakly-supervised learning. In

89

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Section 5, we compare the results obtained by the pro-

Table 1 Summary of the number of words in the sentences

posed method and other initialization methods. Finally,
Number of words
Sentencesin Section 6, we provide our conclusions and suggestions
for future research.
In the following sections, the terms “isolated words”
and “continuous words” indicate isolated sign language
words and continuous sign language words, respectively,
to avoid redundant representations.

2.

3.1

3
17

4
12

5
4

6
8

7
5

8
2

Database

The speciﬁcations of the database are depicted in thefollowing. We suppose a situation where a single signer
signs in front of a camera. We recorded videos with asmart-phone camera and requested the signers to sit andsign in oﬃce environments. Moreover, we requested thatthe signers pose in the static posture at the beginning andthe end of a sign, as shown in Fig.1. All video frames
were recorded at 30 frames per second with 640 × 360
pixels. We note that each signer is recorded in diﬀerent
places. Therefore, each video has a unique background
depending on the signer. The database includes 109 iso-Related Work

This section introduces the previous weakly-supervised
learning approach of sign language recognition. Koller et
al. applied ‘ﬂat-start’ initialization as a pre-process of
training a convolutional neural network to classify handshapes during signs2). They deﬁne mappings between
each word and hand shapes based on Sign-Writing8), and‘ﬂat-start’ initialization divides video frames in words according to the mappings. We note that their ‘ﬂat-start’
initialization is conducted on the sub-unit-level annotation. The sub-unit-level annotation depicts ﬁner motion
types than the word-level annotation1), and the errors of
the initial segmentation may be insigniﬁcant. Unfortunately, adequate mappings between each word and thehand shapes of many sign languages including JSL havenot been established. Therefore, in many cases, ‘ﬂatstart’ initialization has negative eﬀects on recognition
performance.
Koller et al. also proposed iterative forced alignment based on their CNN-BLSTM-HMM system3). Theirmethod mutually updates the annotation and recognition models, and the recognition performance improves
with each repetition. However, in their approach, theinitial annotation should be given by manual or otherpre-trained models. This limitation is common in HMMbased forced alignment.
Automatic model extraction using video subtitles isalso an interesting approach in the ﬁeld. Farhadi etlated JSL words that were signed by ten native signers
and 62 types of continuous words that were signed byeighteen non-native signers and one native signer. The
ten native signers include three males and seven females
in their twenties to 50s. The one native signer of continuous words is a female in her twenties who is included in
the ten native signers of isolated words. They use JSL
in daily communication. The eighteen non-native signers
include fourteen males and four females in their twenties to 50s. The recording of eighteen non-native signers was conducted after several hours of training because
they are beginners of JSL. The continuous words are built
from the 109 words to make short sentences. The number of words in the sentences are summarized in Table 1.
We requested that the signers perform each isolated wordﬁve times. In this paper, we use the term “trials” to indicate these repetitions. As a result, 5,432 videos wereobtained except for the recordings that failed. Moreover,
we recorded four trials for each of the continuous words,
and 4,621 videos were obtained except for recording andsign errors. While all isolated words have manually deﬁned time ranges and those labels, ordered word labels
without time ranges are deﬁned for the continuous words.
All experiments in this study were conducted using
the database. In this paper, we compare the three
types of weakly-supervised learning, namely, ‘ﬂat-start’and ‘bootstrap’ initialization, and the proposed method.
These methods are characterized by available annotations
to train initial models as depicted in Table 2. We notethat all methods use ‘embedded training’ with the ordered word labels to adjust the initial models.

al.9) proposed a data mining method to automaticallyextract common pairs of words and signs from the subtitled videos. Buehler et al.10) proposed a similar approachbased on multiple instance learning. These methods havethe advantage that manual annotation is not required,
but it is diﬃcult to train uncommon words.

3.

2
14

Research Settings

In this section, we explain a database and a base sign
language recognition method to depict the research settings in advance.

90

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Table 2 Available annotation for HMM initialization
Method‘ﬂat-start’‘bootstrap’
Proposed

3.2

Annotation
Ordered word labels
Full manual labels
One manual label for each word

Base sign language recognition
Fig. 2 Word network for sign language recognition

Three types of the weakly-supervised learning aretested on the previously proposed sign language
recognition11). The sign language recognition consists of

HMM to test weakly-supervised learning. λ = (A, B, π)and X = {x1 , x2 , . . . , xT } indicate model parametersof HMM and an observed time sequence, respectively.
A = {aij ; i, j = 1, 2, . . . , N } and B = {bi (x)} represent

body parts tracking using OpenPose12)–15), feature extraction, and parts-based multi-stream HMM. We provide a brief introduction to sign language recognition in
the following.

the state transition and output probabilities, respectively.
These parameters are trained through initial and ‘embedded training’. π = {π1 = 1, πi>1 = 0} represents initialstate transition probability, and we employed ﬁxed π toconstruct left to right HMM. N is the number of hidden
states of HMM, and we employed N = 5 and N = 20 for
the models of the static posture and each isolated word,
respectively. T is the number of observations in an observed time sequence.
The output probability of the HMM is deﬁned as:
γs
S 

N [xst ; μis , diag(Σis )].
1bi (xt ) =(a) Body-parts tracking
In this research, we employ 7, 21, and 21 tracking points
of OpenPose for a body, left hand, and right hand, respectively. The seven points of a body include the joints of
the left and right arms, both shoulders, and the neck.
We removed the points of a part of the torso, lower body,and face because the tracking points of these parts were
almost static during signs in our settings.
(b) Feature extraction
The feature extraction converts the raw tracking data
to abstracted sign features using base feature computation and feature abstraction. Three types of base features, normalized tracking positions, point connections
according to the human skeleton, and ﬁrst-order derivatives of positions and connections, are calculated fromthe raw tracking data. The normalized tracking posi-

s=1

S = 3 is the number of streams associated with bodyparts. γs is a relative weight to a stream s. Althoughγs can be used to set the relative importance of each
body part, we use γs = 1 in this paper to simplify thesetting. N [·; μ, diag(Σ)] is a single multivariate Gaussian
distribution with a diagonal covariance matrix. μis andtions are the transformed two-dimensional coordinates.
The transformation is conducted for each body part. The

Σis are the multivariate mean and covariance, which areassociated with the ith hidden state.
We note that the ﬁrst and last hidden states of HMM
are non-emitted states, and these are used to connect
HMMs in ‘embedded training’ and the classiﬁcation.
The classiﬁcation is based on the Viterbi algorithm
using a linguistic model, which is shown in Fig.2.
This linguistic model supposes that a sentence consists of “Static”→“Word”→“Static.” The loop struc-point connections are distances and directions between
the pairs of neighboring points. The ﬁrst-order derivatives are calculated only for the body because the tracking of hands is sometimes unstable. The feature abstraction is composed of Z-score normalization and Principle
Component Analysis (PCA). Z-score normalization adjusts the scales of each dimension. PCA removes theredundant dimensions.
As a result of these processes, the raw tracking data areture in Fig.2 allows for many word repetitions. The
training of models and classiﬁcation are conducted using
Hidden Markov Toolkit7).

converted to 96-dimensional sign features, which include
25, 37, and 34-dimensional vectors of the body and left
and right hands. The forced alignment utilizes the 96dimensional sign features that were obtained as a resultof the feature extraction.

4.

Weakly-Supervised Learning

4.1 Process overview
In this section, we describe the proposed weaklysupervised learning. An overview of the proposed weaklysupervised learning is illustrated in Fig.3. The boxes and(c) Parts-based multi-stream HMM
We designed left to right parts-based multi-stream

91

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
300
270
240
210

80
60
40

100

StaticTHANKS

Reference frames.

100

20
0

Static300 270 240 210

Static0

THANKS
20

40

Static

60

80

100

10000

80

8000

60

6000

40

4000

20

2000

0

0

20

40

60

80

Target frames.

1000Fig. 4 Example of forced alignment using DTW

Pmin = argmin < P : Δ(X, Y ) >,2

P ∈PTx ,Ty

where < M1 : M2 > is the Frobenius inner product of
two matrices M1 and M2 , which have the same row andFig. 3 Process overview

column dimensions. DTW eﬃciently ﬁnds Pmin based on
dynamic programming.
Figure 4 shows an example of forced alignment using
DTW. The left and top graphs indicate y coordinates ofa right wrist of a reference and target sequence during asign, respectively. We note that Fig.4 shows a comparison of scalar values to visualize the sequences, but the
96-dimensional sign features are used in the experiments.
The central colored matrix shows a cost matrix. The cost
matrix has the squared Euclidean distances between theobservations of the pair of time sequences as its elements.
The vertical axes of the left graph and the cost matrix
are the frame indices of the reference data. The horizontal axes of the top graph and the cost matrix are theframe indices of the target data. The blue line on thematrix shows the alignment path {(i, j); pij = 1} withthe minimum cost, which goes through the minimal costpoints from the bottom left corner to the top right corner. The dotted lines and labels on the left graph show
the annotation of the reference data. The boundaries of
the reference annotation can be represented as horizontal lines in the cost matrix. When ib is an index thatrounded boxes in Fig.3 are the processes and data, respectively. While the annotated sequence means that time
ranges and those labels are deﬁned, the labeled sequence
only has ordered word labels. The unannotated sequence
does not have any time ranges and labels, but it is assumed to have the same ordered word labels as manuallyannotated sequences. First, one sequence is sampled foreach isolated word and annotated manually. This annotated sequence is used to assign annotations to the othersequences using forced alignment based on DTW. Next,
initial word HMMs are trained based on the Viterbi algorithm and Baum-Welch (BW) re-estimation using theannotated sequences. Finally, the initial word HMMs areadjusted based on ‘embedded training’ using the labeledsequences of continuous words.
4.2

Forced alignment based on DTW
5)

DTW is one of the standard techniques to ﬁnd
a temporal alignment between a pair of time sequences. X = {x1 , . . . , xi , . . . , xTx ; xi ∈ Rn } and Y ={y1 , . . . , yj , . . . , yTy ; yj ∈ Rn } indicate a pair of time sequences. PTx ,Ty ⊂ {0, 1}Tx ×Ty is a set of binary alignment matrices. P ∈ PTx ,Ty is a Tx × Ty matrix thatindicates an alignment path according to the DTW constraints, and pij = 1 indicates that xi and yj are aligned.

coincides with a boundary of the reference annotation,
an index of the target boundary can be found by seeking
pib j = 1. This means that the target boundary is determined as a vertical line that goes through a cross point of
the reference boundary and the alignment path, as shownin the cost matrix of Fig.4.

Δ(X, Y ) is a Tx × Ty cost matrix. Given these notations, the alignment matrix with the minimum cost
Pmin ∈ PTx ,Ty is deﬁned as follows:
92

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) Entering ib vertically(b) Entering ib diagonallyFig. 5 Examples of the alignment path conversion

However, the index of the target boundary is notuniquely determined when the multiple target frames arealigned with the reference boundary. This is the casewhen the horizontal part of the alignment path goes along
the dotted horizontal lines in Fig.4.
We avoid this case by using an alignment path conversion. We suppose two cases, as shown in Fig.5. The
squares in Fig.5 show neighborhoods of pib j = 1 on theFig. 6 Process ﬂow of ‘embedded training’

rithm adjusts the parameters based on expectation maximization. We applied the BW algorithm four times toadjust the parameters in this research.
(b) Model adjustment
Figure 6 shows the process ﬂow of ‘embedded training’. First, ‘embedded training’ concatenates isolated
word HMMs according to the ordered word labels of continuous words. Next, ‘embedded training’ applies the BW
algorithm to the concatenated HMM, and the parametersof the isolated word HMMs included in the concatenated
HMM are simultaneously updated. We applied parameter adjustments using ‘embedded training’ on two occasions in this research.

alignment path. The blue dots indicate pib j = 1 on thealignment path. Fig.5 (a) is the case when the alignment
path enters ib vertically, and Fig.5 (b) is the diagonalcase. When the multiple candidates of pib j = 1 exist, thealignment path is converted by employing the maximum
index of j in both cases.
The path conversion is an empirical process. Althoughemploying the minimum index of j and some interpolation techniques are also available, the eﬀects of theseselections on recognition performance were minor in our
research settings.
4.3

5.

Evaluation

This section describes evaluations of the proposed
method. We report the alignment and recognition perfor-

Isolated word HMM adjustment(a) Initial training
The initial training is composed of initial parameter estimation and the adjustment of parameters. First, training observation sequences of an isolated word HMM aresampled according to the annotation, which is generated
by forced alignment. Next, each observation sequence
is divided into equal segments according to the hidden
states of the HMM, and the means and covariances ofeach output probability are initialized using the sample

mances based on the forced alignment and classiﬁcation
by HMM, respectively. Moreover, we show comparisonsto ‘ﬂat-start’ and ‘bootstrap’ initialization. All evaluations in this section are based on the database and sign
language recognition, which are described in Section 3.
5.1

Alignment performance

The performance of the forced alignment was evaluatedbased on the distances of annotation boundaries between
the manual and aligned annotations. This evaluation was
conducted using 5,432 videos of the 109 isolated words
by ten native signers. The average duration of the 5,432

mean and covariances. Next, the observations are assigned to each hidden state using the Viterbi algorithm,
and the parameters are updated. Finally, the BW algo-

93

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Method‘ﬂat-start’
Proposed

Maximum
24.58
16.78

Minimum
9.83
0.46

Mean
16.05
4.02

Word classification accuracy [%]Table 3 Alignment performance [frames]

Signer

100

SD
3.12
3.21

videos is 66.87 frames.
While the distances of ‘ﬂat-start’ initialization are
uniquely determined, those of the proposed method depend on the sampled reference annotation for DTW.
Hence, we ﬁrst divided the dataset into subsets for eachisolated word and calculated the maximum, minimum,
mean, and Standard Deviation (SD) of the distances foreach isolated word. While the distances of the annotation
boundaries of ‘ﬂat-start’ initialization were calculated one
by one with comparison to the manual annotations, thoseof the proposed method were calculated based on leaveone-out cross-validation for each isolated word. We thensummarized the performance values based on macro averaging of all the isolated words.
The alignment performances are summarized inTable 3. The proposed method achieved distances aboutfour times lower than that of ‘ﬂat-start’ initialization.

93.10

76.57

80
60

Trial

34.50

91.14

74.82

50.77

40
20
0

flat-start bootstrap Proposed

Initialize method

Fig. 7 Performance of the initial models

the number of deletions, substitutions, and insertions, respectively7)．

evaluations based on the signer-opened and trial-opened
conditions were conducted. We repeated the sampling
and evaluation for every subject, and the macro averages
for all subjects were calculated.
The performance of the initial models is summarized in
Fig.7. The red and blue bars in Fig.7 represent the wordclassiﬁcation accuracy of the signer-opened and trialopened conditions, respectively. The error bars represent
the SD for the mean accuracies in the cross-validation.
The values on the upper error bar denote the mean accuracies. We note that the summarized values of the
proposed method are the macro averages for all patternsof the forced alignment.
As shown in Fig.7, the performance of the proposed
method was superior to that of ‘ﬂat-start’ initialization
with suﬃcient margins, and comparable to that of ‘bootstrap’ initialization. At the same time, we found thatthe errors of ‘ﬂat-start’ initialization resulted in degradedrecognition performance.

The performance of the initial models using ‘ﬂat-start’and ‘bootstrap’ initialization was evaluated using ten-fold
and ﬁve-fold cross-validations, respectively. The ten-foldcross-validation uses the videos of nine subjects for train-(b) Performance of adjusted models
The performance of the adjusted models was evaluated using 4,621 videos of the 62 continuous words byeighteen non-native signers and one native signer. The

ing and those of one subject for testing. On the other
hand, the ﬁve-fold cross-validation uses the videos of 4/5
trials of all subjects for training and those of 1/5 trials
for testing. We refer to these experimental settings as thesigner-opened and trial-opened conditions, respectively.
For the evaluation of the proposed method, we sampled one sequence for each word from the same subject;forced alignment was applied to other sequences. Next,

performance of the adjusted models depends on the initial models and the composition of the training and test
datasets. Therefore, we combined the cross-validationof initial and ‘embedded training’. There are ten andﬁve initial models respectively of the signer-opened andtrial-opened conditions after the initial training. For theevaluation of the adjusted models, we sampled one ini-

5.2

Recognition performance(a) Performance of initial models
We report the performance of the initial models using
5,432 videos of the 109 isolated words by ten native signers.
The recognition performance was calculated based on
the word classiﬁcation accuracy. The word classiﬁcation
accuracy is deﬁned as:
Accuracy =N −D−S−I
× 100,
N

3

where N, D, S, and I indicate the total number of words,

tial model and applied ‘embedded training’. Similar to94

Word classification accuracy [%]IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Signer-Signer

Signer-Trial

Trial-Signer

Trial-Trial

to ‘ﬂat-start’ initialization, the performance improvementof the proposed method has been worth considering as a100

practical solution.

80
60

In spite of the improvements of the proposed method,

64.49
69.20
65.57
69.28

65.35
65.33

bootstrap

Proposed

69.12

67.70

46.42
51.91
45.58
52.27

40
20
0

flat-start

Initialize method

Fig. 8 Performance of the adjusted models

the evaluation of the isolated words, nineteen-fold andfour-fold cross-validations were conducted based on signers and trials, respectively. We repeated the samplingof initial models and ‘embedded training’ for all initialmodels, and report four types of macro averages.
The summarized performance is illustrated in
Fig.8. We denote the types of macro averages as
Initial training condition - ‘embedded training’ conditionin Fig.8. For example, Signer-Trial in Fig.8 denotesthat the signer-opened and trial-opened conditions wereapplied to initial and ‘embedded training’, respectively.
The pink, red, cyan, and blue bars in Fig.8 represent the word classiﬁcation accuracy of Signer-Signer,
Trial-Signer, Signer-Trial, and Trial-Trial, respectively.
The performance of the proposed method was superior
to that of ‘ﬂat-start’ initialization and comparable to thatof ‘bootstrap’ initialization in every case, as shown in
Fig.8.

6.

Conclusions and Future WorkIn this paper, we proposed weakly-supervised learning for continuous sign language word recognition using
forced alignment based on DTW and isolated word HMM
adjustment using ‘embedded training’. As shown in theevaluation, the proposed forced alignment has achieved
signiﬁcant improvement in the alignment performancefrom that of ‘ﬂat-start’ initialization. Moreover, we havethere are still diﬃculties in contending with the variety of motions as indicated in the maximum and SD of
the alignment distances in Table 3. The proposed forcedalignment supposes that the target data are similar to
the reference data. It is diﬃcult to provide appropriate annotations when the diﬀerence between the target
and the reference data is signiﬁcant. Therefore, such acase should be annotated manually from the viewpointof building corpora. Outlier detection of the time sequences may be available to exclude the expected failurecases and improve the overall eﬃciency of building thecorpora. Moreover, some extensions of DTW16),17) can
improve the stability of time sequence alignment.
While word-level annotation is a feasible solution tobuild corpora, detailed components of sign language, forexample, “Transition” motions, sub-unit-level motions,and asynchronous motions of body parts, should be considered to establish high-level sign language recognition.
The proposed method has avoided the annotation of“Transition” using ‘embedded training’. This approach
has limitations for large-scale continuous words because
the many types of “Transitions” are diﬃcult to learn
as part of the word HMMs. The combination of theautomatic generation of “Transition” models2) and soft
boundary assignments18) are expected to lessen this limitation. The proposed method is independent of the ﬁneness of motion units. Therefore, the proposed method
is available for sub-unit-level annotation if manual annotation by experts is available. Moreover, DTW and‘ﬂat-start’ initialization can be combined to automatically generate sub-unit-level annotation when the adequate mappings between JSL words and motion units are
established. For the asynchronous motions of body parts,
extensions to asynchronous and multi-modal DTW19)and
HMM20)may be available to our approach.
The current research settings have been relatively controlled. Weakly-supervised learning for sign language
recognition in practice must be addressed.

found that the proposed weakly-supervised learning performed comparably to ‘bootstrap’ initialization with only
one manual annotation for each isolated sign language
word and the ordered word labels for the continuous
words. Although the proposed method requires at least
one manual annotation for each isolated word compared

References
1) U. von Agris, J. Zieren, U. Canzler, B. Bauer, K.F. Kraiss:“Recent Developments in Visual Sign Language Recognition”,
Universal Access in the Information Society, Vol.6, No.4,
pp.323–362 (2008).
2) O. Koller, H. Ney, R. Bowden: “Deep Hand: How to Train

95

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled”, Proc. of the IEEE Conference
on Computer Vision and Pattern Recognition, pp.3793–3802(2016).

20) S. Bengio: “An Asynchronous Hidden Markov Model for
Audio-Visual Speech Recognition”, Proc. of the Advancesin Neural Information Processing Systems 15, pp.1237–1244(2003).

3) O. Koller, S. Zargaran, H. Ney: “Re-Sign: Re-Aligned End-toEnd Sequence Modelling with Deep Recurrent CNN-HMMs”,
Proc. of the IEEE Conference on Computer Vision and Pattern
Recognition, pp.3416–3424 (2017).

(Received September 11, 2019)(Revised November 20, 2019)4) N.C. Camgoz, S. Hadﬁeld, O. Koller, H. Ney, R. Bowden:“Neural Sign Language Translation”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp.7784–
7793 (2018).

Natsuki TAKAYAMA (Member)
From 2008 to 2013, he was with SIElectronics, Ltd., Tokyo, Japan. He received the
D.E. degree from the University of ElectroCommunications, Tokyo, Japan, in 2017.
Since 2017, he has been with the Graduate School of Informatics and Engineering,
the University of Electro-Communications,Tokyo, Japan, where he is now a Researcher.

5) H. Sakoe, S. Chiba: “Dynamic Programming Algorithm Optimization for Spoken Word Recognition”, IEEE Trans. on
Acoustics, Speech, and Signal Processing, Vol.26, No.1, pp.43–
49 (1978).
6) K.-F. Lee, H.-W. Hon:
“Large-Vocabulary SpeakerIndependent Continuous Speech Recognition Using HMM”,
Proc. of the International Conference on Acoustics, Speech,
and Signal Processing, pp.123–126 (1988).

Hiroki TAKAHASHI (Member)He received the D.E. degree from the Tokyo
Institute of Technology, Tokyo, Japan, in
2005. Since 2006, he has been with the Graduate School of Informatics and Engineering,
the University of Electro-Communications,Tokyo, Japan, where he is now an Associate
Professor.

7) Cambridge University Engineering Department, Hidden
Markov Toolkit Version 3.4.1, http://htk.eng.cam.ac.uk(2018).
8) SignWriting Web Site, http://www.signwriting.org (2018).
9) A. Farhadi, D. Forsyth: “Aligning ASL for Statistical Translation Using a Discriminative Word Model”, Proc. of the IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition, pp.1471–1476 (2006).
10) P. Buehler, A. Zisserman, M. Everingham: “Learning Sign
Language by Watching TV (Using Weakly Aligned Subtitles)”,Proc. of the IEEE Conference on Computer Vision and Pattern
Recognition, pp.2961–2968 (2009).
11) N. Takayama, H. Takahashi: “Sign Words Annotation Assistance Using Japanese Sign Language Words Recognition”,
Proc. of the International Conference on Cyberworlds, pp.221–
228 (2018).
12) S.-E. Wei, V. Ramakrishna, T. Kanade, Y. Sheikh: “Convolutional Pose Machines”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp.4724–4732 (2016).
13) T. Simon, H. Joo, I. Matthews, Y. Sheikh: “Hand Keypoint
Detection in Single Images Using Multiview Bootstrapping”,
Proc. of the IEEE Conference on Computer Vision and Pattern
Recognition, pp.1145–1153 (2017).
14) Z. Cao, T. Simon, S.-E. Wei, Y. Sheikh: “Realtime MultiPerson 2D Pose Estimation Using Part Aﬃnity Fields”, Proc.
of the IEEE Conference on Computer Vision and Pattern
Recognition, pp.7291–7299 (2017).
15) Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, Y. Sheikh: “OpenPose: Realtime Multi-person 2D :Pose Estimation Using PartAﬃnity Fields”, arXiv preprint arXiv:1812.08008 (2018).
16) M. Cuturi, M. Blondel: “Soft-DTW: a Diﬀerentiable Loss
Function for Time-Series”, Proc of the 34th International Conference on Machine Learning, pp.894–903 (2017).
17) J. Zhao, L. Itti: “ShapeDTW: Shape Dynamic Time Warping”,
Pattern Recognition, Vol.74, pp.171–184 (2018).
18) L. Ding, C. Xu: “Weakly-supervised Action Segmentation
With Iterative Soft Boundary Assignment”, Proc. of the IEEE
Conference on Computer Vision and Pattern Recognition,
pp.6508–6516 (2018).
19) M. Wöllmer, M. Al-Hames, F. Eyben, B. Schuller, G. Rigoll:“A Multidimensional Dynamic Time Warping Algorithm forEﬃcient Multimodal Fusion of Asynchronous Data Streams”,
Neurocomputing, Vol.73, No.1-3, pp.366–380 (2009).

96

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2019 --A Model Ensemble Approach for Few-Shot Learning Using Aggregated Classiﬁers
Toshiki KIKUCHI† (Student Member) , Yuko OZASA†† Keio University<Summary> Despite the recent success in deep neural networks on the visual domain, we need a large
amount of data to train the networks. Previous works addressed this issue as the few-shot learning whichis the task to identify the class of an example in new classes not seen in a training phase with only a
few examples of each new class. Some methods performed well on the few-shot tasks, but need a complex
architecture and/or specialized loss functions, such as metric loss, meta learner, and memory. In this paper,
we evaluate the performance of the ensemble approach aggregating a huge number of simple neural networkmodels (up to 128 models) on standard few-shot datasets. Surprisingly, although the approach is simple, ourexperimental results show that the ensemble approach is competitive with state-of-the-art methods among
similar architecture methods in some settings.
Keywords: few-shot classiﬁcation, model ensemble, deep learning

1.

tion on the evaluation set. Their method and similar
methods7),8) perform well, but they use specialized neural network architectures and loss functions.
One naı̈ve method, without using specialized architecture and meta-learning techniques9)–11) , is merely to
use the feature representation extracted with a standard
classiﬁer. A convolutional neural network (CNN)-basedclassiﬁer trained for the classiﬁcation task can be used
as a feature extractor for many tasks in computer vision12)–14) . However, the feature extracted with a single

Introduction

Deep neural network-based approaches outperform
other conventional approaches on object recognition
tasks1),2) . However, we need a massive amount of labeled data to train deep neural network-based models.
Simply training the model with a small dataset leads toover-ﬁtting. The problem of learning using only a few examples is called one- or few-shot classiﬁcation3),4) . Oneor few-shot classiﬁcation is the task to predict the class ofthe given example by comparing a few examples of eachpossible class. Because the class of the given example isnot seen in a training phase and only a few examples ofeach of these classes are given, the task is challenging fordeep neural networks.
In the common setting for few-shot learning, we canuse prior knowledge from a training set. The training set

classiﬁer is not useful enough for few-shot classiﬁcation
on an evaluation set, because the training set and theevaluation set have disjoint label space. Then, our fundamental issue is how we obtain enough feature for few-shotclassiﬁcation using the training set.
To overcome the issue in a simple way, we here adopt amanner of traditional ensemble learning that is the concept of combining several models to make a ﬁnal decision. Basically, even when we train a classiﬁer on thesame dataset, there are some diﬀerent features enoughfor the task (optimal solutions of the model). Therefore,
we hypothesize that we can obtain a better feature forand the dataset for evaluating few-shot classiﬁcation, ofcourse, have completely disjoint label spaces. However,the characteristic of the examples in both datasets are
similar because they have the same or a similar domain(e.g., handwritten alphabet images5) or RGB object images6) ). In recent work on few-shot learning with deepneural networks, many works use the training set to optimize the network weights. One notable method is Matching Nets proposed by Vinyals et al.6) . They simulate thefew-shot classiﬁcation on the evaluation set by ﬁnding
and aggregating the diﬀerent features for the classiﬁcation task on the disjoint training set. Therefore, we employ many classiﬁers in an ensemble manner, which aretrained for classiﬁcation task on the training set, and aggregate the features extracted with each classiﬁer. Each

one-shot classiﬁcation setting on a training dataset andapply the trained model for one- or few-shot classiﬁca-

97

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

we do not follow this approach.

classiﬁer is expected to provide a diﬀerent feature becauseof the randomness that comes from the random initializa-

Because deep neural network-based approaches outper-tion of each network’s weight and the random sampling

form other conventional approaches in recent work on
the object classiﬁcation task1),2) with large-scale datasets,
many recent works on few-shot learning are based on
deep neural networks. Some works address the overﬁtting problem by simulating the few-shot classiﬁcation
with an additional dataset with disjoint labels and learn
the metrics of the domain instead of training with a few
examples. The ﬁrst work following this approach is proposed by Koch19) . Koch19) used Deep Siamese Networks,
which predict whether the classes of two input images
are the same or not, for a one-shot classiﬁcation task.
Matching Nets6) and Prototypical Nets8) learn the em-of mini-batches, even if all the classiﬁers have the samearchitecture and are trained with the same training set.
Each extracted feature is rich enough for classiﬁcation in
the label space of the training set but not enough for fewshot classiﬁcation on the evaluation set with disjoint label
space. However, by concatenating the features, we canobtain richer features for few-shot classiﬁcation. We callthis ensemble approach “Classiﬁer Aggregation” (ClassAgg). As the classiﬁers are trained to solve standard
classiﬁcation tasks, we do not need specialized architecture and loss function. In addition, the models do nothave memory like the methods by Kaiser et al.15) , Santoro et al.16) , and Ravi and Larochelle10) . However, we
surprisingly found that the simple ensemble approach iscompetitive with state-of-the-art methods.
In this paper, we propose the method to adopt anensemble fashion for few-shot classiﬁcation, and evaluate the eﬀectiveness of the ensemble approach, ClassAgg. Surprisingly, experimental results show that theapproach is competitive with state-of-the-art among similar architecture methods for 5-way 1-shot classiﬁcation
on Mini-ImageNet6) and achieved the same accuracies as

bedding with CNN-based model by simulating the oneshot classiﬁcation setting with a training set when theytrain the models. Sung et al.7) also took a similar approach, and the model learns to compare the input images. Although these approaches demonstrated good performance for few-shot classiﬁcation tasks, they need specialized neural network architectures and extraordinary
loss functions. The ClassAgg approach instead does not
simulate the few-shot classiﬁcation in training phase sothat we do not need them.
Another approach is employing a neural network withmemory. Kaiser et al.15) combined the Siamese Nets withlong short-term memory (LSTM20) ). Santoro et al.16)

the state-of-the-art, with 95 % conﬁdence interval overlap, for 5-way 5-shot and 20-way 5-shot classiﬁcation on
Omniglot Previous Split5),6) . We also show the evaluationof how the number of aggregated classiﬁers contributes tothe performance of few-shot classiﬁcation.

2.

employed Neural Turing Machines21) for one-shot learning. In the approach, they classify a given example using
historical information in the memories. In contrast, theClassAgg approach does not need complex architecture

Related Work

with memories like LSTM.
Finding the good initial condition of the network is analternative approach proposed by Finn et al.11) . They

Many previous works are tackling few-shot classiﬁcation3),4) which is a task to classify an example of theclass not seen in training, with only a few examples foreach class. Merely training the classiﬁer with the few examples causes the model to overﬁt so that we cannot use
this naı̈ve approach.
Earlier approach for one-shot learning used the generative model3),4) . In recent work, Edwards and Storkey17)

proposed the few-shot learning method for learning theinitial condition of neural networks, as meta-information.
The meta-learner provides a good initial condition to ﬁnetune with few-shot classiﬁcation. MAML11) performs
well on few-shot classiﬁcation, but the method needs ﬁnetuning. We do not refer the ﬁne-tuning in the ensemble
approach because of focusing on obtaining good representation for few-shot classiﬁcation from the disjoint dataset.
Meta Nets9) and Meta-Learn LSTM10) also focus on improving the optimization strategies. They use LSTM tolearn the loss gradient of neural nets as meta-information
for updating the parameters of the model for one-shotclassiﬁcation. These methods that use neural networks
with memory have specialized architectures.

used the variational autoencoder18) to learn generativemodels for few-shot learning. The ﬁrst approach for oneshot classiﬁcation on the Omniglot dataset5) uses speciﬁcknowledge for handwritten alphabet characters: pencil
strokes. Their method, called Hierarchical Bayesian Program Learning, can perform well but mainly focuses on
one-shot learning on only handwritten alphabet characters. Since the target domain of these methods is limited,98

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) Feature space with a single classiﬁer%DFNJURXQG
&ODVVLILFDWLRQ)HZVKRW&ODVVLILFDWLRQ݂ଵ ሺ‫ݔ‬ሻ%DFNJURXQG
&ODVVLILFDWLRQ)HZVKRW&ODVVLILFDWLRQ)HZVKRW&ODVVLILFDWLRQ%DFNJURXQG
&ODVVLILFDWLRQ(b) Feature space with some classiﬁers(c) Feature space with a large numberof classiﬁers

Fig. 1 Concept of ClassAgg
×kin support set S = {xsi , yis }N
i=1 .

In comparison to these previous approaches, since theClassAgg approach employs simple CNN-based classiﬁers, our method does not need memory, ﬁne-tuning or ameta-learner.
In terms of using multiple models, there is a traditional

4.

To bring an ensemble manner into few-shot classiﬁcation tasks, we here use an aggregation of a large numberof classiﬁers. Each classiﬁer is trained to solve the classiﬁcation task for the samples in training set B. We callthis task background classiﬁcation. Although few-shotclassiﬁcation and background classiﬁcation have disjointlabel space, features useful enough for background classiﬁcation can be useful for few-shot classiﬁcation to someextent because these tasks use features extracted fromsimilar domains (e.g., alphabet images and RGB object
images). However, a feature extracted with a single classiﬁer is not eﬀective enough for few-shot classiﬁcation because it is trained for the background classiﬁcation task.
Figure 1 shows the feature space extracted from given
image x which may have a label in training set B, support
set S, or testing set T. In Fig. 1, the vertical and horizontal axes indicate the eﬀectiveness of the feature space
for the few-shot classiﬁcation and the background classiﬁcation, respectively. Because the label spaces of thesetasks are disjoint, the axes are orthogonal. Each yellow
ellipse shows the possible feature space that the classiﬁer

technique called ‘ensemble learning’ that is the concept of
combining some models to improve the performance. In
general classiﬁcation tasks, ensemble learning combinesclassiﬁers for reducing the risk of misclassiﬁcation by apoorly performing classiﬁer22) . In that setting, each classiﬁer is trained for the same task as the ﬁnal predictiontask. The ClassAgg is a kind of ensemble learning, but
in few-shot classiﬁcation setting, the models are trained
for the task diﬀerent from the ﬁnal prediction task. The
ClassAgg aims to obtain richer feature representation
with many classiﬁers.

3.

Classiﬁer Aggregation

Problem Setting

We consider the task of one- or few-shot learning. Following a task formulation by Vinyals et al.6) , we havethree datasets: training set B (sometimes called the background set), support set S, and test set T. Support set S
and testing set T have the same label space. In contrast,
training set B has its own label space diﬀerent from thatof support set S and testing set T. Therefore, the classes
that we can see in a training phase are disjoint with thosethat we can see in an evaluation phase. Every example x
in all datasets has its label y.
For performing one- or few-shot evaluation, we determine which example in S has the same class as the given
test example xt ∈ T. We have to do this task only withcan output. The ellipse can be projected to two basis
vectors, the few-shot classiﬁcation (vertical axis) and thebackground classiﬁcation (horizontal axis) to measure theperformance on each task. The range (length of the line
segment) of the result of projecting the feature space toeach axis represents the performance on the task related
to the axis. If a broader range is covered, it means thatthe feature can be more useful for the task. Of course,
the same feature has diﬀerent performances on diﬀerent

prior information of examples in training set B. If the
support set consists of k-labeled examples for each of N
diﬀerent classes, the target few-shot problem is called N way k-shot learning. In this case, we have N ×k examples

tasks. Even if a feature is valuable for a task, the feature
is not always beneﬁcial for another task. Therefore, the99IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

covered range of the classiﬁer for each axis is diﬀerent in

Algorithm 1 Few-shot classiﬁcation with ClassAgg

Fig. 1.

Input: Test example xt ∈ T×kSupport set S = {xsi , yis }N
i=1
Feature extraction functions {fc (x)}c=C
c=1
Output: A predicted class label yispred for given test
example xt// Compute aggregated feature for test example
f t ← [f1 (xt ), f2 (xt ), ..., fC (xt )]// Compute similarities for all support examples
for i = 1, 2, . . . , N × k
f s ← [f1 (xsi ), f2 (xsi ), ..., fC (xsi )]sim[i] ← cosine similarity between f s and f t
end
ipred ← arg max sim

If we train a single classiﬁer for background classiﬁcation as feature extractor f1 , the feature extractor provides a feature space as shown in Fig. 1(a). The feature isenough for background classiﬁcation, but it covers a small
range for the few-shot classiﬁcation. Here, we suppose
to train many classiﬁers that extract the diﬀerent feature spaces and use these feature extractors. Figure 1(b)shows the feature space covered by a large number ofclassiﬁers. Of course, because all classiﬁers are trained
for the background classiﬁcation task, the range covered
for the task does not change from Fig. 1(a). However,
the aggregated feature covers a broader range for fewshot classiﬁcation as long as each classiﬁer learns a diﬀerent feature space. Because of the randomness producedby the random sampling for mini-batches in the training phase and the random initialization of layer weights,
each CNN-based classiﬁer is expected to have a diﬀerent
optimal solution and learn a diﬀerent feature space even
if all the classiﬁers have the same network architecture.
By aggregating them, we can obtain richer feature representation for few-shot classiﬁcation than using a featurefrom a single classiﬁer. Fig. 1(c) shows a covered feature
space with a massive number of classiﬁers. As the number of classiﬁers increases, the coverage improvement for
few-shot classiﬁcation decreases. Because each classiﬁer
is trained for the same task (background classiﬁcation),the probability of having a feature space similar to one of
the feature spaces of previous classiﬁers increases with amore signiﬁcant number of classiﬁers.
We deﬁne aggregated feature f (x) as the concatenationof features extracted from each classiﬁer:
f (x) = [f1 (x), f2 (x), ..., fC (x)],(1)

where fi (x) is the output of the penultimate layer of ithclassiﬁer with given input x, and C is the number of classiﬁers. With the aggregated feature f (x), we measure
similarities between samples. We call this ensemble approach ClassAgg.
We use aggregated feature f (x), which is extracted
from a given image x, for the few-shot classiﬁcation problem. The summary of the algorithm for few-shot classiﬁcation with ClassAgg is outlined in Algorithm 1. In
this algorithm, we predict which example in support set
S has the same class as the test example xt with feature
extractors {fc (x)}c=C
c=1 . For each support example index
i, we compute the cosine similarity sim[i] between aggre-

100ireturn yispredgated features f t and f s which is extracted from xt and a
support example xsi , respectively, to ﬁnd the predi which
maximize sim[predi ]. ipred is the index for the predictedlabel so the ﬁnal predicted label is yispred . In short, using
the cosine similarities of the aggregated features of a test×ksample xt ∈ T and support samples S = {xsi , yis }N
i=1 , weperform N -way k-shot classiﬁcation.
As ClassAgg uses a standard classiﬁcation model, we
do not need to use any complex architecture, such as
memory-augmented neural networks or meta-learning in
reinforcement learning. Besides, we do not need to use
a specialized objective function because each classiﬁer istrained to solve the classiﬁcation problem in a standard
way: optimizing the network weights with cross-entropy
loss function23) . Because each classiﬁer does not dependon other classiﬁers, we can train all classiﬁers in parallel
to reduce computational time.

5.

Experiments

To evaluate the performance of ClassAgg, we conducted experiments on three groups of datasets: Omniglot Previous Split6) , Omniglot Standard Split5) , andMini-ImageNet6) . For each dataset, we performed oneand few-shot classiﬁcations for 1000 episodes. Each
episode has 10 examples in the test set T, so 10000 trials are conducted for each dataset. With the result of
the trial, we report the accuracy with 95 % conﬁdence
intervals.
5.1
5.1.1

Datasets
Omniglot Previous Split

Omniglot5) is the dataset that contains 1623 characters from 50 diﬀerent alphabets. Because every class has
only 20 samples, we augmented the training set with random rotations and shifting. Consequently, we have 200
samples per class. Following previous few-shot learning

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

work6),9) , all images are resized to 28 × 28 to reduce theWe trained 128 classiﬁers for each Omniglot setup andcomputational cost.
6)

Following Vinyals et al. , we split the dataset into 1200
and 423 classes for training set B and evaluation set E,
respectively. For each episode in the evaluation phase, we
randomly sampled some examples from evaluation set E
to generate support set S and testing set T.
For the dataset, we performed 5-way 1-shot, 5-way 5shot, 20-way 1-shot, and 20-way 5-shot classiﬁcation.
5.1.2

Omniglot Standard Split

Following the original condition of one-shot learning in
the Omniglot dataset provided by Lake et al.5) , we also
conducted experiments on the Standard Split in the same
way as the Previous Split, described in Section 5.1.1. The
Standard Split provides 30 training (background) alphabets with 964 classes and 20 evaluation alphabets with
659 classes. The number of training classes in this setup
is fewer than that of in the Previous Split setup. Therefore, this is a more diﬃcult setup for one-shot learning.
Following Munkhdalai and Yu9) , for the dataset, we
performed 5-way 1-shot, 10-way 1-shot, 15-way 1-shot,and 20-way 1-shot classiﬁcation.
5.1.3

Mini-ImageNet

Mini-ImageNet proposed by Vinyals et al.6) is thedataset that contains 100 classes from ImageNet24) . Because each class has 600 examples, the dataset consistsof 60000 color images. We follow the class split proposedby Ravi and Larochelle10) . With the split, we obtain
three groups: training with 64 classes, validation with
16 classes, and testing with 20 classes. Because we train
the networks to solve classiﬁcation in training set B, we
do not use the validation set which has diﬀerent classspace with the training set. Following few-shot learning
work6),9) , all images are resized to 84 × 84 to reduce thecomputational cost.
Following Ravi and Larochelle10) and Munkhdalai andYu9) , for the dataset, we performed 5-way 1-shot and 5way 5-shot classiﬁcation.
5.2

layer, respectively.

Training details

We use a standard architecture for few-shot learning(e.g., References 6) and 10)). The model is a stack offour modules. The module consists of a 3 × 3 convolutional layer with 64 ﬁlters, a ReLU activation, batch normalization25) , and a 2 × 2 maximum pooling layer. For
Mini-ImageNet, the network is trained with dropout26) toavoid over-ﬁtting at a dropout rate p = 0.1 and p = 0.25
after every convolutional layer and before the penultimate

101

Mini-ImageNet. Therefore, we evaluated the performanceof ClassAgg with C = 1 to 128 for each dataset. We train
classiﬁers that have the same standard architecture. Eachclassiﬁer provides the output of the penultimate layer as
the feature vector fi (x) for the given x. The dimensionof feature dimension fi (x) is 64.
Using training set B on each dataset, we train theclassiﬁers to solve the 1200-way (M = 1200), 964-way(M = 964), and 64-way (M = 64) classiﬁcation tasks for
Omniglot Previous Split, Omniglot Standard Split, andMini-ImageNet, respectively.
Before starting to train, every weight of the layersof classiﬁers was randomly initialized with Glorot uniform initialization27) . For optimizing the network, we
use Adam28) . Because of the limitation of the memory, we used randomly sampled mini-batches containing
1024 samples and 512 samples for Omniglot and MiniImageNet, respectively.
To avoid over-ﬁtting, we used 80 % of the training set
B for training and 20 % for validation. Using validationbased early stopping, we stopped training a classiﬁer
when the metric for validation did not improve for morethan P epochs. We used validation loss with P = 2
and validation accuracy with P = 10 for Omniglot-based
datasets and Mini-ImageNet, respectively.
5.3

Results

For each dataset, we show the few-shot classiﬁcation
accuracies with various state-of-the-art baseline methods. For Omniglot Previous Split, the baseline methods include Siamese Nets19) , MANN16) , Matching Nets6) ,Siamese Nets with Memory15) , Neural Statistician17) ,Meta Nets9) , Prototypical Nets8) , MAML11) , and Relation Net7) . For Omniglot Standard Split, the baseline
methods include Pixel kNN, Aﬃne model, Deep Boltzmann Machines reported by29) , Hierarchical Bayesian
Program Learning5) , Siamese Nets19) , and Meta Nets9) .
For Mini-ImageNet, Matching Nets6) , Meta Nets9) , MetaLearn LSTM10) , Prototypical Nets8) , and Relation Net(Naive)7) . We also show the accuracy results for the
methods using deeper architecture as the reference7),30) .
Also, we show the relationship between the number ofclassiﬁers and the accuracy to demonstrate that aggregating a large number of classiﬁers is suitable for few-shotclassiﬁcation. It also shows the limitation of the ClassAgg approach: performance improvement decreases as

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Table 1 Result of few-shot classiﬁcation on Omniglot Previous Split (‘-’: not reported)
5-way Acc. (%)
MethodFine Tune

1-shot5-shot1-shot5-shotConvolutional Siamese Nets
Convolutional Siamese Nets19)
MANN16)
Matching Nets6)
Matching Nets6)
Siamese Nets with Memory15)
Neural Statistician17)
Meta Nets9)
Prototypical Nets8)
MAML11)
Relation Net7)

N
Y
N
N
Y
N
N
N
N
Y
N

96.7
97.3
82.8
98.1
97.9
98.4
98.1
99.0
98.8
98.7 ± 0.4
99.6 ± 0.2

98.4
98.4
94.9
98.9
98.7
99.6
99.5
99.7
99.9 ± 0.1
99.8 ± 0.1

88.0
88.1
93.8
93.5
95.0
93.2
97.0
96.0
95.8 ± 0.3
97.6 ± 0.2

96.5
97.0
98.5
98.7
98.6
98.1
98.9
98.9 ± 0.2
99.1 ± 0.1

ClassAgg-1 (C = 1)
ClassAgg-64 (C = 64)
ClassAgg-128 (C = 128)NN
N

94.8 ± 0.5
98.7 ± 0.3
98.6 ± 0.3

98.8 ± 0.2
99.7 ± 0.1
99.7 ± 0.1

86.2 ± 0.4
95.8 ± 0.2
95.8 ± 0.2

95.9 ± 0.2
98.9 ± 0.1
98.9 ± 0.1

19)

the number of classiﬁers increases.
5.3.1

Omniglot Previous Split

The results of few-shot classiﬁcation on Omniglot Previous Split are shown in Table 1. We also show the result
for ClassAgg-1, a method that uses a feature extractedwith a single classiﬁer, to demonstrate the performanceof the ensemble approach. ClassAgg-128, which is the aggregation of 128 classiﬁers, achieved state-of-the-art levelaccuracy with 95 % conﬁdence intervals for 5-way 5-shotand 20-way 5-shot. With a large number of classes intraining set B, ClassAgg can perform as well as otherstate-of-the-art methods.
5.3.2

20-way Acc. (%)

Omniglot Standard Split

vious methods which follow the standard architecture
proposed by Reference 6). As TCML30) and Relation
Net (Deeper)7) are based on the deeper network than
the standard architecture, we cannot compare the performance to these methods, similar to Grant et al.31) .
5.3.4

The number of classiﬁers

We show the relationship between the number of classiﬁers (horizontal axis) and the few-shot classiﬁcation accuracy (vertical axis) for each dataset in Fig. 2. As thenumber of classiﬁers increase, the performances of the
few-shot classiﬁcation improve with some improvement
decay. Obviously, the accuracies of the harder setting arelower from C = 1 to C = 128. For all datasets and all

We show the results for one-shot classiﬁcation on Omniglot Standard Split in Table 2. Although ClassAgg-128
could not outperform the state-of-the-art method, we cansee the eﬀectiveness of the ClassAgg by comparing theresults for ClassAgg-1 and ClassAgg-128. Whereas in 1shot 5-way in Previous Split setting, ClassAgg achieved
the same accuracy as Meta Nets9) with conﬁdence interval overlap, ClassAgg could not outperform Meta Netsin the Standard Split setting which has fewer classes intraining set B. In the ClassAgg approach, since we use
the classiﬁers, the performance of few-shot classiﬁcation
relies on the background set. Therefore, the ClassAgg ap-

tasks, the average accuracy improves drastically as thenumber of classiﬁers increases from C = 1 to around 20.
The ﬁrst improvement between C = 1 and C = 2 is thehighest one in all settings. As explained in Section 4., theaccuracy improvement by adding one classiﬁer decreases
as the number of classiﬁers increases. Because all classiﬁers are trained for the same background classiﬁcation,

proach can perform better when we can train with morebackground classes.

When using a single classiﬁer for few-shot classiﬁcation(in the case of C = 1), which classiﬁer we use may cause

5.3.3

Mini-ImageNet

The results for few-shot classiﬁcation on MiniImageNet are shown in Table 3. For 5-way 1-shot setting, ClassAgg-64 and ClassAgg-128 outperformed pre-102

the possible feature space that the classiﬁer extract is limited. Due to this, a classiﬁer tends to have a feature space
similar to one of the feature spaces of previous classiﬁers
with a more signiﬁcant number of classiﬁers.
5.3.5

Performance variance of single model

divergent results because of the randomness of initialization of each network’s weight and the random sampling ofmini-batches. However, in the case of ClassAgg-1 (C = 1)
on our experiments, we just used the ﬁrst classiﬁer weIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Table 2 Result of one-shot classiﬁcation on Omniglot Standard Split (‘-’: not reported)
Method5-way (%)

10-way (%)

15-way (%)

20-way (%)

5)

-

-

-

95.5

Pixel kNN
Aﬃne model29)
Deep Boltzmann Machines29)
Hierarchical Bayesian Program Learning5)
Siamese Nets19)
Meta Nets9)

98.45

97.32

96.4

21.7
81.8
62.0
96.7
92.0
95.92

92.5 ± 0.6
97.9 ± 0.3
97.8 ± 0.3

87.0 ± 0.5
96.4 ± 0.3
96.5 ± 0.3

83.5 ± 0.5
95.0 ± 0.3
95.1 ± 0.3

80.8 ± 0.4
94.0 ± 0.3
94.1 ± 0.3

Human performance
29)

ClassAgg-1 (C = 1)
ClassAgg-64 (C = 64)
ClassAgg-128 (C = 128)

Table 3 Result of few-shot classiﬁcation on Mini-ImageNet (‘-’: not reported)
5-way Acc. (%)
MethodBase Architecture
6)

Matching Nets
Meta Nets9)
Meta-Learn LSTM10)
MAML11)
Prototypical Nets8)
Relation Net (Naive)7)

Standard

6)

ClassAgg-1 (C = 1)
ClassAgg-32 (C = 32)
ClassAgg-64 (C = 64)
ClassAgg-128 (C = 128)
TCML30)Relation Net (Deeper)7)

Deepertrained as the feature extractor. Therefore, we evaluatedthe variance of the performance with the single classiﬁer
caused by the randomness.
Table 4 show the results of few-shot classiﬁcation witha diﬀerent single classiﬁer on Omniglot Previous Split,
Omniglot Standard Split, and Mini-ImageNet, respectively. For each setting, these tables show the lowest
accuracy (as Min.), the highest accuracy (as Max.), thediﬀerence between them (as Diﬀ.), and the variance of
the accuracy (as Var.) for each classiﬁer. In this experiment, the used classiﬁer is selected through the classiﬁers
pool which consists of classiﬁers used for ClassAgg-128(C = 128). Consequently, we conducted the few-shotclassiﬁcation evaluation for 128 classiﬁers.
As a result, for all settings, the performance on theharder setting has the more signiﬁcant diﬀerence between
the minimum and maximum performances and the higher
variance. From these results, it appears that it is diﬃcult to obtain richer representation required by the harder103

Fine Tune

1-shot5-shot

N
N
N
Y
N
N

43.56 ± 0.84
49.21 ± 0.96
43.44 ± 0.77
48.70 ± 1.84
49.42 ± 0.78
51.38 ± 0.82

55.31 ± 0.73
60.60 ± 0.71
63.11 ± 0.92
68.20 ± 0.66
67.07 ± 0.69

N
N
N
N

47.98 ± 0.99
54.43 ± 1.00
54.55 ± 1.01
54.82 ± 1.01

58.52 ± 1.01
64.89 ± 0.99
65.38 ± 0.98
65.68 ± 0.97

NN

55.71 ± 0.99
57.02 ± 0.92

68.88 ± 0.92
71.07 ± 0.69

setting with the single classiﬁer. Besides, even the highest accuracies with the single classiﬁer are beaten by our
results with aggregated classiﬁers, shown in Table 1, 2
and 3. This means that the ClassAgg approach works
eﬀectively to obtain rich representation.

6.

Discussion and Future Work

We propose the method to adopt an ensemble mannerfor few-shot classiﬁcation, and evaluate the ensemble approach, ClassAgg. Just aggregating the trained classiﬁers
achieved high accuracies for some tasks of one-shot classiﬁcation. Because the ensemble approach is simple, we do
not need to modify the architecture of a standard classiﬁcation model, and we do not have to use specialized loss
function to train networks.
Each classiﬁer is trained for the classiﬁcation task of the
training set B, which has a label space disjoint from support set S and test set T. Because of the randomness ofthe training (e.g., mini-batch sampling or random weight
initialization), every classiﬁer is expected to provide dif-IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Table 4 Result of few-shot classiﬁcation using a single
classiﬁer(a) Omniglot Previous Split

100.0%

Averaged accuracy

98.0%
96.0%
94.0%
92.0%

SettingMin. (%)

Max. (%)

Diﬀ. (%)

Var. (%)

5-way 1-shot
5-way 5-shot
20-way 1-shot
20-way 5-shot

93.54098.24084.34894.85895.58099.07087.34596.160

2.040
0.830
2.998
1.303

0.001
0.000
0.003
0.001

90.0%
5-way 1-shot
5-way 5-shot
20-way 1-shot
20-way 5-shot

88.0%
86.0%
84.0%

0

20

40
60
80
100 120
The number of classifiers C(b) Omniglot Standard Split

140

SettingMin. (%)

Max. (%)

Diﬀ. (%)

Var. (%)

5-way 1-shot
10-way 1-shot
15-way 1-shot
20-way 1-shot

91.29085.42081.66779.00093.43088.85085.61783.145

2.140
3.430
3.950
4.145

0.002
0.004
0.005
0.006

(a) Omniglot Previous Split

Averaged accuracy

100.0%(c) Mini-ImageNet

98.0%

SettingMin. (%)

Max. (%)

Diﬀ. (%)

Var. (%)

96.0%

5-way 5-shot
5-way 1-shot

56.33046.61063.04053.570

6.710
6.960

0.018
0.021

94.0%
92.0%

number of classiﬁers increases. Besides, the ensemble approach can not perform better when the number of the
background classes is fewer because we employ classiﬁers
for the background set. Therefore, we can conclude that
our approach can work well if the data of the training set
is diverse enough to train a classiﬁer which can extract
useful feature from disjoint evaluation set. Although we
focused only on images, as long as the classiﬁer can getwell-trained for evaluation set, our approach may be extended to few-shot learning on the data from other modalities (e.g., audio and natural language).
Another downside of the ensemble approach is higher
computational cost in a prediction phase. We cannot
merely compare with methods that perform without neural networks, but comparing other neural network-based
methods, our approach has a higher computational costdepending on the number of classiﬁers we employed because we need to run feature extractor C times. While,in terms of computational time, since the computationof our ensemble approach is straightforward to be parallelized, the computational time cannot be a severe issue.

90.0%
88.0%
86.0%

5-way 1-shot
10-way 1-shot
15-way 1-shot
20-way 1-shot

84.0%
82.0%
80.0%

0

20

40
60
80
100 120
The number of classifiers C

140(b) Omniglot Standard Split

Averaged accuracy

70.0%
65.0%
60.0%
55.0%
50.0%
45.0%

5-way 1-shot
5-way 5-shot
0

20

40
60
80
100 120
The number of classifiers C

140(c) Mini-ImageNet
Fig. 2 Eﬀect of the number of classiﬁers

Our future work includes exploring a method thatforces the classiﬁer to learn diﬀerent features so that we

ferent features extracted from the same image although
the label spaces are disjoint.
Our experimental results show that the one- and fewshot classiﬁcation accuracy improves as the number ofclassiﬁers increases, as long as each classiﬁer extracts a
diﬀerent feature. The results also show that it gets morediﬃcult to extract a diﬀerent feature, which is not similar to features extracted from previous classiﬁers, as the104

can achieve high performance with fewer models.

References
1) K. He, X. Zhang, S. Ren, J. Sun: “Deep Residual Learning for
Image Recognition”, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp.770–778 (2016).
2) A. Krizhevsky, I. Sutskever, G. E. Hinton: “ImageNet Classiﬁcation with Deep Convolutional Neural Networks”, Communications of the ACM, Vol.60, No.6, pp.84–90 (2017).

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）3) L. Fei-Fei, R. Fergus, P. Perona: “One-Shot Learning of Object
Categories”, IEEE Trans. on Pattern Analysis and MachineIntelligence, Vol.28, No.4, pp.594–611 (2006).
4) B. M. Lake, R. Salakhutdinov, J. Gross, J. B. Tenenbaum:“One Shot Learning of Simple Visual Concepts”, Proc. ofthe 33rd Annual Conference of the Cognitive Science Society,pp.2568–2573 (2011).

(2014).
22) R. Polikar: “Ensemble Based Systems in Decision Making”,
IEEE Circuits and Systems Magazine, Vol.6, No.3, pp.21–45(2006).
23) I. Goodfellow, Y. Bengio, A. Courville: “Deep Learning”, MIT
Press (2016).

5) B. M. Lake, R. Salakhutdinov, J. B. Tenenbaum: “HumanLevel Concept Learning Through Probabilistic Program Induction”, Science, Vol.350, No.6266, pp.1332–1338 (2015).

24) J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, L. Fei-Fei: “ImageNet: A Large-Scale Hierarchical Image Database”, Proc. of
the IEEE Conference on Computer Vision and Pattern Recognition, pp.248–255 (2009).

6) O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, D. Wierstra: “Matching Networks for One Shot Learning”, Proc. of
Advances in Neural Information Processing Systems, pp.3630–
3638 (2016).

25) S. Ioﬀe, C. Szegedy: “Batch Normalization: Accelerating Deep
Network Training by Reducing Internal Covariate Shift”, Proc.
of the International Conference on Machine Learning, pp.448–456 (2015).

7) F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, T. M.
Hospedales: “Learning to Compare: Relation Network for FewShot Learning”, Proc. the IEEE Conference on Computer Vision and Pattern Recognition, pp.1199–1208 (2018).

26) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever,R. Salakhutdinov: “Dropout: A Simple Way to Prevent Neural Networks from Overﬁtting”, Journal of Machine Learning
Research, Vol.15, No.1, pp.1929–1958 (2014).

8) J. Snell, K. Swersky, R. S. Zemel: “Prototypical Networks forFew-shot Learning”, Proc. of Advances in Neural Information
Processing Systems, pp.4077–4087 (2017).

27) X. Glorot, Y. Bengio: “Understanding the Diﬃculty of Training Deep Feedforward Neural Networks”, Proc. of the International Conference on Artiﬁcial Intelligence and Statistics,
pp.249–256 (2010).

9) T. Munkhdalai, H. Yu: “Meta Networks”, Proc. of the International Conference on Machine Learning, pp.2554–2563 (2017).
10) S. Ravi, H. Larochelle: “Optimization as a Model for Few-Shot
Learning”, Proc. of the International Conference on Learning
Representations (2017).
11) C. Finn, P. Abbeel, S. Levine: “Model-Agnostic Meta-Learning
for Fast Adaptation of Deep Networks”, Proc. of the International Conference on Machine Learning, pp.1126–1135 (2017).
12) X. Wang, L. Lu, H. Shin, L. Kim, M. Bagheri, I. Nogues,
J. Yao, R. M. Summers: “Unsupervised Joint Mining of
Deep Features and Image Labels for Large-Scale Radiology
Image Categorization and Scene Recognition”, Proc. of theIEEE Winter Conference on Applications of Computer Vision,
pp.998–1007 (2017).
13) A. S. Razavian, H. Azizpour, J. Sullivan, S. Carlsson: “CNN
Features Oﬀ-the-Shelf: An Astounding Baseline for Recognition”, Proc. of the IEEE Conference on Computer Vision and
Pattern Recognition Workshops, pp.512–519 (2014).
14) J. Y. H. Ng, F. Yang, L. S. Davis: “Exploiting Local Features from Deep Networks for Image Retrieval”, Proc. of theIEEE Conference on Computer Vision and Pattern Recognition
Workshops, pp.53–61 (2015).
15) L . Kaiser, O. Nachum, A. Roy, S. Bengio: “Learning to Remember Rare Events”, Proc. of the International Conference
on Learning Representations (2017).
16) A. Santoro, S. Bartunov, M. M. Botvinick, D. Wierstra, T. P.
Lillicrap: “Meta-Learning with Memory-Augmented Neural
Networks”, Proc. of the International Conference on Machine
Learning, pp.1126–1135 (2016).
17) H. Edwards, A. Storkey: “Towards a Neural Statistician”,
Proc. of the International Conference on Learning Representations (2017).
18) D. P. Kingma, M. Welling: “Auto-Encoding Variational
Bayes”, Proc. of the International Conference on Learning Representations (2013).
19) G. R. Koch: “Siamese Neural Networks for One-shot Image
Recognition”, Proc. of the ICML Deep Learning Workshop,
Vol. 2 (2015).
20) S. Hochreiter, J. Schmidhuber: “Long Short-Term Memory”,
Neural Computation, Vol.9, No.8, pp.1735–1780 (1997).
21) A. Graves, G. Wayne, I. Danihelka: “Neural Turing Machines”,
The Computing Research Repository (CoRR), abs/1410.5401

10528) D. P. Kingma, J. Ba: “Adam: A Method for Stochastic Optimization”, Proc. of the International Conference on Learning
Representations (2015).
29) B. M. Lake, R. Salakhutdinov, J. B. Tenenbaum: “OneShot Learning by Inverting a Compositional Causal Process”,
Proc. of Advances in Neural Information Processing Systems,
pp.2526–2534 (2013).
30) N. Mishra, M. Rohaninejad, X. Chen, P. Abbeel: “MetaLearning with Temporal Convolutions”, The Computing Research Repository (CoRR), abs/1707.03141 (2017).
31) E. Grant, C. Finn, S. Levine, T. Darrell, T. Griﬃths: “Recasting Gradient-Based Meta-Learning as Hierarchical Bayes”,
Proc. of the International Conference on Learning Representations (2018).

(Received August 30, 2019)(Revised November 19, 2019)

Toshiki KIKUCHI (Student Member)He received his B.Eng. degree in Information
and Computer Science from Keio University,
Japan, in 2018. He is currently a MS student
in Machine Learning and Computer Visionat Keio University. His research interests include machine learning, audio-visual processing, and computer graphics.

Yuko OZASAShe received her PhD degree in engineeringfrom Kobe University in 2015. She was a
postdoctoral researcher at National Instituteof Advanced Industrial Science and Technology (AIST) in 2015. Since 2015 she has been a
research associate at Graduate School of Science and Technology, Keio University. Her
research interests include object recognition
and grounding, multimodal fusion, visual perception, and hyperspectral sensing.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2019 -Visual Simulation of Tearing Papers Taking Anisotropic Fiber Structure
into Account
Saeko SHINOZAKI† , Masanori NAKAYAMA† (Member) , Issei FUJISHIRO† (Honorary Member)† Keio University<Summary> Real paper gets deformed anisotropically due to a uni-directional placement of the innerﬁbers. Few existing CG researchers allow for such ﬁbers to represent virtual papers, and thus anisotropic
paper deformation has not been well represented. In this study, we present a two-dimensional visual simulation model for anisotropic papers, which abstracts mutual mechanical relationships among intersecting
ﬁbers by a network of ﬁller and hinge springs, and incorporates connection points for keeping the shape ofeach bended ﬁber. By releasing the network connections when the ﬁller and hinge spring extend above acertain limit, the paper provides a plausible tear. We succeeded in generating a diﬀerent appearance of thetorn-oﬀ line of papers according as the pulling direction.
Keywords: visual simulation, microstructure, paper, tearing

1.

Introduction

In the paper-forming process, the pulp, which has been
dissolved in water, is ejected from the paper machine at ahigh speed. Even after dewatering and drying, the pieceof paper retains its ﬁber direction. Thereby the deformation eﬀect appears to diﬀer depending on the direction of
the external force applied. This direction, called ‘grain’,is essential for printing, decorating, wrapping, and bookbinding. For example, a newspaper is printed in such a
way as not to fold in a vertical direction when it is open,
and a book is printed in such a fashion so that its readers
might turn its pages easily. The grain also inﬂuences theresult of the tearing of the paper and, indeed, we often
encounter the case where the tear tends to be irregularwhen the piece of paper is pulled in parallel with the ﬁber
direction, or to be straight when pulled in a direction
orthogonally to it. The ﬁber length of softwood is approximately 50 μm and that of hardwood approximately
20 μm, which is too thin for humans to discern. Fibers
with such a minuteness running oﬀ from the tear cause
the blurry outlines to be seen from a macro perspective.
There exist many CG studies on paper tearing. Several recent articles deal with interactive paper tearing,as seen in Schreck et al. 1) and Lejemble et al. 2) , where
they rely only on mesh-based paper models in order to reduce the cost of computation. Busaryev et al. 3) and Pfaﬀ
et al. 4) also apply mesh-based paper modeling schemes106

to simulate paper tearing, and attempt to balance thecomputational cost and the reasonable minuteness using
adaptive remeshing. Directing our attention to the research on the simulation of a cloth, which is also comprised of ﬁbers, Zhao et al. 5) use a model allowing for the
microstructure, and thus improve the appearance of theentire object. However, any cost-eﬀective approach to thesimulated tearing of paper with ﬁber-level details has notbeen focused. For this reason, the simulated tear cannot represent the diﬀerence according to the direction of
the ﬁber nor the elastic force caused by ﬁber connection.
In addition, the minuteness obtained by the mesh-basedmodeling 1)–4) is not so high that it can show the minuteﬁbers running oﬀ the edge of the paper and the blurred
tear speciﬁc to a paper.
In this research, therefore, we attempt to model a pieceof paper with microstructure and tear the piece so as togenerate realistic outlines reﬂecting its own feature dynamics. First, directed ﬁbers are arranged on a twodimensional plane, and neighboring ﬁbers are mutuallyconnected to form a sheet of ﬁbers (simply referred to as‘paper ﬁber sheet’ hereafter). Then, the deformation of
the paper ﬁber sheet is simulated by an abstract modelthat incorporates into a single framework: ﬁller ﬁlling the
sheet; hinge springs repulsing against the ﬁber rotation;and connection points keeping the shape of bended ﬁbers.
Finally, the sheet is torn by considering the dynamic action caused by applying a given pulling force to it.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) An electron microscope photo of real paper:
red area represents ﬁber, while white part ﬁller(b) Proposed model of paper ﬁber sheet

Fig. 1 Structure of real paper and proposed model

The remainder of this paper is organized as follows.
The next section introduces prior work related to paper deformation and modeling. Section 3 explains the
structure of paper ﬁber sheet and our main algorithm ofgenerating and tearing the sheet. Section 4 explains anacceleration method for the ﬁber intersection detection
and an associated data structure for the parallel processing using a GPU. Section 5 shows results of our paper
ﬁber sheet tearing model. Section 6 concludes this paper
and refers to our future work.

2.

realize a repulsion force of a bended paper by enlarging
the cell groups through mutual connection of neighboring groups. However, the spring in the mesh group disappears when the end mass points are separated. Consequently, the deformed shape cannot return to its original state due to the lack of repulsion. In contrast, ourmethod can transmit the elastic force without the loss ofthe spring property because, if one connection is broken,
other components connecting to the same ﬁber can retain
the elastic force.

Related Work3.

A paper consists of pulp extracted from trees and isformed by intertwining ﬁbers. Metaaphanon et al. 6) simulate the fray of yarn using a coarse yarn-based fabric.
Zhao et al. 5) scan the ﬁber structure of cloth using a CT
scanner and then convert to the volume data to reproduce the cloth with ﬁber microstructure. Jiang et al. 7)
simulate the deformation operation to the object with aﬁber structure, taking into account its plasticity, elasticity, and friction using the Material Point Method 8) .
Inspired by the idea proposed in Takagi et al. 9) , we havebuilt upon the way of modeling paper with microstructure
to generate plausible tears. In fact, they model ﬁbers andﬁller as basic paper components in a volumetric manner.
They arrange ﬁbers so as to share a primary direction
and also consider the local ﬁber deformation eﬀects, suchas constrictions and ﬂuﬀs, generated during the paperforming process. In contrast, we model a paper sheetwith ﬁbers so as to express the ﬂuﬀ seen in the papertear, and introduce hinge spring and connection point toexpress the internal force caused by the deformation.
Kitani et al. 10) deﬁne a three-dimensional paper object
as a mass-spring cubic mesh. They express a directionalﬁber by grouping neighbor cells to make the corresponding biased mass-spring parallel-piped. Furthermore, they

107

3.1

Paper Model

Structure of paper ﬁber sheet

In this study, based on observing the structure of real
paper in Fig. 1 (a), we model a paper ﬁber sheet by incorporating ﬁbers and ﬁller, and hinge springs, as shownin Fig. 1 (b). In addition, we connect a dozen ‘ﬁber
fragments’ with connection points to express one bendedﬁber.
The length of each ﬁber fragment is set to 0.06 mm to0.10 mm so that the length of ﬁber will be 1.0 mm to
1.5 mm, which is compatible with the average length ofthe pulp composing a real photocopy paper. Hereafter,
the model for generating and tearing a paper ﬁber sheet issimply referred to as ‘paper model’. Each of components
is detailed below.
Filler is an inorganic pigment added for increasing thesmoothness, whiteness, and printability of paper in thereality. In our method, ﬁller is treated as the connector
between ﬁbers and has the spring property as with ﬁber
fragments whereas it exhibits a diﬀerent behavior. Filler
is assumed to gain the bonding force between ﬁbers andthus to increase or decrease the strength of the paper
relative to its amount.
Hinge springs act to prevent the rotation of ﬁber fragments. By generating a hinge spring that is symmetri-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

ordinate values of the middle point of the ﬁber fragment
are set randomly in the generation area. The angle of the
ﬁber fragment is determined by varying randomly within(a) Without hinge springs
Fig. 2(b) With hinge springs

Eﬀect of hinge springs

cally centered around the intersecting point of ﬁber fragments, the restoring force from the elastic deformation of
the hinge spring is transmitted to ﬁber fragments, andthen the hinge spring restrains the ﬁber fragment rotation. Figure 2 shows the eﬀect of hinge springs whenﬁber fragments are alternately arranged. Figure 2 (a)demonstrates the result of pulling the upper and lower
ends of the ﬁber fragment network connected with thesingle spring placed at the intersecting point in the vertical direction while Fig. 2 (b) the result with hinge springs.
Without the hinge springs, the force is transmitted to theconnecting ﬁber fragment after a ﬁber fragment starts tomove. In contrast, hinge springs repulse the rotation ofﬁber fragment, so that the force is immediately transmitted to the neighboring ﬁber fragment. By repeating this,the force from the sheet edges can be evenly transmitted
across the sheet.
In the paper model, a connection point connects ten toﬁfteen ﬁber fragments into one bended ﬁber. As in thecase with ﬁller and hinge springs, these points generate
the repulsion according to the variation of their lengthand reproduce the action to keep the bended shape ofﬁbers. The connection point itself does not break unlike ﬁller and hinge springs because we do not take into
account the break of ﬁbers themselves.
3.2

a range of ±22.5 degrees from the 0 or 90 degrees depending on the direction of the ﬁber of the paper. The point
vertex 0 at a distance of half of the ﬁber length at thatangle from the midpoint is the starting point of the ﬁber
fragment, and the point vertex 1 located on the opposite
side is the ending point.
Then, ten to ﬁfteen ﬁber fragments are rotated andconnected into one bended ﬁber with reference to the appearance of real ones. Not all ﬁber fragments extend in
the same direction but, if the ﬁber fragments are arranged
vertically, they can extend randomly both in the upperand lower direction. At the same time, the information
on how many ﬁber fragments are included in the ﬁber
is attached to each ﬁber. There is a branch to the suspension of the extension caused by the ﬁber reaching theedge of the generating area when we form cuts, such as
perforations.
After that, the intersections of these generated ﬁbers
are detected. In order to mediate between two intersecting ﬁbers, a hinge spring is placed symmetrically withrespect to the intersection point. At the same time, several intersecting ﬁbers are mutually connected by ﬁller.
In addition, for each ﬁber the connection points are generated so as to connect the ﬁber fragments.
Finally, all vertices are scanned and if a vertex exists
in the pulled area, then the vertex is added to the array.
Note that the array is prepared separately on the pulling
direction. For example, when the sheet is pulled in the
Y -axis direction, the vertices with the Y coordinate value
greater than a certain value are added to the array of thevertices pulled in a positive direction, while the vertices
with less than the value are added to the array of pulled
in a negative direction.
3.3

Generation of paper ﬁber sheet

Repulsion computation

Our method has ﬁve steps for generating paper ﬁber
sheet: determining the initial settings; generating ﬁber

The tearing simulation is executed following Algorithm 1.

fragments; attaching ﬁber fragments to the corresponding cells; intersection detecting of ﬁber fragments andgenerating the inner components; and setting the area
pulled by the external force.
First, the size of paper ﬁber sheet, the number of ﬁbers,
and the aspect ratio of the sheet are set to the user input.
Next, we generate the ﬁber fragments that vary inlength, and form vertical and horizontal joints. The co-

Each computation in the algorithm is executed in parallel by a GPU. Note that the iteration count N = 200in the algorithm was set empirically to reproduce plausible tearing eﬀects in our settings. If the simulation was
not iterated, the ﬁbers would tend to stretch almost inﬁnitely. Hence, we decided to observe tearing results witha series of the iteration number from 100 to 500 by 100
increments. Consequently, the number 200 provided the108

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Table 1

Algorithm 1 Tearing simulation
Move all vertices in the pulled area;
for N=1, . . . , 200
Initialize the force applied to all vertices
Calculate the repulsion of ﬁbers and add it to each vertex
Calculate the repulsion of inner components and add it
to each vertex of the connecting ﬁbers
Set a hiding ﬂag when ﬁller or a hinge spring extends
above a certain limit
Move all vertices according to the calculated force
end
Redraw the window

Variable

Variables and their meanings in Fig. 3

DescriptiondLa

The variation of lengthva

The unit vectorfa

The shrinking force

v0 , v 1

The vector of ﬁber 0 and 1

L0 , L 1

The length of ﬁber 0 and 1

x0 , y0

The position vector for ﬁber 0 with respect to
the X-axis direction and the Y -axis obtainedby rotating the X-axis counterclockwise by 90
degrees on the plane on which the sheet is produced

x1 , y1

The position vector for ﬁber 1 with respect to
the X-axis direction and the Y -axis obtainedby rotating the X-axis counterclockwise by 90
degrees on the plane on which the sheet is produced

r0 , r 1

The vector from the centroid of the ﬁber tothe intersecting point of the components

f0 , f 1

The spinning force caused by the components
pulling ﬁbers

The forces fp0 and fp1 added to the vertex 0 as thestarting point of the ﬁber vector and the vertex 1 as theending point are calculated by the following equations:Fig. 3

Variables used for computing the repulsion caused by
the components

best result in terms of the balance of the elongation andthe shrinking of the ﬁbers.
Sheet pulling is reproduced by moving the corresponding vertices and the repulsion is computed with each component, according to the moving width. The connections
are released when ﬁller or a hinge spring extends over
the repulsion above the certain limit. The vanishment of
torn ﬁler and hinge springs is expressed by the suspensionof rendering due to the restriction of the data structure.
When the sheet is broken after repeating these processes,
the tearing is deemed to have been completed.
The computation is diﬀerent between ﬁber fragments
and other components, as detailed in the following.
As an elastic body deforms, the force is generated torestore the body to its original shape. Since ﬁber fragments in our paper model have the spring property, therestoring force is generated according to the deformationon the both ends of the ﬁbers.
The restoring force f caused by this property is given
by multiplying the amount of the variation of their lengthdL by the vector v of ﬁber fragment:
f = 3.0dL v.

(1)

109

fp0 = +1.5f ,(2)

fp1 = −1.5f .

(3)Note that the coeﬃcients in Formulae (1) to (3) are set toreproduce plausible tear with reference to the appearanceof simulation results. They are the material constants of
the ﬁber itself, and hence they will not change as long as
the same ﬁber material is used.
Figure 3 shows the variables used for computing therepulsion caused by ﬁller, hinge springs, and connection
points. Table 1 explains what each of the variables
means. Note that the repulsion caused by ﬁller, hinge
spring, and connection point can be computed with thesame formula described below by varying only the coeﬃcients.
The shrinking force of the component fa is given by:
fa = αdLa va ,(4)

where α is a constant to which a diﬀerent value is assigned for each component and determines the strengthof shrinking.
The moments caused by f are given by the outer product of fa and r0 , r1 , as follows:

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）z0 = r 0 × f a ,(5)

z1 = r 1 × f a ,(6)

Table 2where that z0 and z1 are the scalar values in the directionof Z-axis in the x0 , y0 and x1 , y1 coordinate systems,
respectively, and indicate the moments of the ﬁber pulling
force by the inner components.
Thus, the forces of spinning ﬁber fragments f0 and f1

The parameter values for ﬁller, hinge spring, andconnection point

Parameter

FillerHinge
springConnection
point

Stretch limit value

0.1

0.05

–

Shrinking strength α

1.5

3.0

1.5

Inﬂuence on vertices β

1.0

1.5

1.5

4.

Accelerated Computation

are given by:
z0
y0 ,
f0 =
L0 /2

(7)

z1
y1 .
L1 /2

(8)

f1 =

A paper ﬁber sheet with a tremendous number of ﬁbers
takes an enormous computation time. Thus we introduced the accelerated computation for the sheet generation and the sheet tearing in our model.
4.1

The repulsion force is calculated by summing up f0 , f1
and fa on the vertices of ﬁber fragments. The forces ofﬁber fragments fp00 , fp01 , fp10 , and fp11 which repulse
the pulling by the connecting components added respectively to vertex 00, 01, 10, and 11 are obtained by taking
into account the direction of the force as follows:
fp00 = +β(fa − f0 ),(9)

fp01 = +β(fa + f0 ),(10)

fp10 = −β(fa − f1 ),(11)

fp11 = −β(fa + f1 ),(12)

where β is a constant to which the diﬀerent value is assigned for each of the components and determines how
much the stretching of the components aﬀects the vertices.
Finally, the force applied to the vertices of each ﬁber
fragment is calculated by summing up the force to pullapart the sheet and the restoring force generated by eachof the ﬁbers, ﬁller, hinge springs, and connecting points.
The vertices of these ﬁber fragments get moved according
to the calculated force, and if the displacement gets overa threshold, the ﬁber connection is released to have thesheet teared.
Table 2 lists the values of the parameters, including
α and β for each component of the paper. Since hinge
spring makes a pair, stretch limit value is set to a halfof the one of ﬁller. Meanwhile, the values of α and β ofhinge spring and connection point are set empirically sothat they connect ﬁber fragments more ﬁrmly than ﬁller.

110Intersection detection

A paper ﬁber sheet with a tremendous number of ﬁbers
takes an enormous computation time for detecting theﬁber intersections, which is necessary for generating ﬁller
and hinge springs. Thus, in this study, we attempt toaccelerate the computation by imposing a grid on theﬁber generation space.
First, the space of ﬁber generation and the number ofgrid cells are predetermined before the sheet generation.
Note that some margin cells are added to the upper andlower and the right and left sides of the original divided
area because without the margins, the ﬁber cannot stickout from the generating area after the extension of ﬁber
fragments.
Next, at most two ﬁber fragments are generated foreach of the cells. In the present paper model, the numberof ﬁber fragments per cell is necessarily set a priori because the ﬁber intersection detection process is executedwith respect to ﬁber fragment, which is linked together
to form one bended ﬁber.
Then, the XY coordinate values of the rectangle, witha ﬁber fragment as a diagonal line are referred to andthe ﬁber fragments are attached to all cells between thevertices, as shown in Fig. 4, where the red ﬁber fragment is attached to the upper two cells and the blue ﬁber
fragment to all the four cells.
Finally, for each of the cells, the intersection of ﬁber
fragments corresponding to the same cell is detected. In
the case of Fig. 5, for green ﬁber, the intersection detection is executed only with two red ﬁbers belonging to
the cell to which the green ﬁber belongs. The intersection
check with blue ﬁber in a distant position can be omitted.
This algorithm is expected to localize the intersection detection. In comparison with a naive case where we check

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Fig. 4

Fig. 5

Fig. 6

Attachment of the ﬁber fragment to cell

Detecting intersection of neighboring ﬁbers in a grida single ﬁber intersected with all the other ﬁbers, thecomputation time for generating a paper sheet was substantially reduced, as shown in Table 3. Note that as
the execution environment, we used a standard PC with
two Intel Xeon E5-2687W 0 3.10GHz CPUs, a 64.00GB
RAM, and an NVIDIA GeForce GTX 1070 Ti GPU.
4.2

Split-edge Data Structure

In our paper model, the internal repulsion generated by
all components is computed, hence the higher the densityof the ﬁbers, the greater the computational complexity.
To reduce the computation time for the simulation, thedata structure described below is applied to our paper
model and enables parallel processing on the GPU.
The repulsion generated by ﬁbers, hinge spring, ﬁller,and connecting points of ﬁbers is calculated by summing
up on the vertices of ﬁber fragments to which each component belongs. Fibers are needed to detect which typeof components connects to themselves and to calculate
the force according to the type to avoid the collision of
the GPU memory writing. Thus, we propose a split-edge
data structure, which is an extension of half-edge data
structure 11) used for mesh processing.
In the split-edge data structure, ﬁller, hinge springs,
and connection points are divided into two, as shown in
Fig. 6, and attached as plug to ﬁber fragments. Red lines

111

The reference relationship in split-edge data structurein Fig. 6 represent ﬁbers, green lines plugs, and yellow
boxes the components such as ﬁller, hinge springs, andconnection points. Each plug has four variables: Crosreferring to which type of objects it comes from; Pairreferring to another plug belonging to the same Cros;
Wire referring to the ﬁber fragment which it connects
to; and Next referring to the next plug connecting to thesame ﬁber fragment. Dividing the components into twoplugs enables ﬁber fragments to trace the components
connecting to itself only with recognizing the ﬁrst plug.
Table 4 compares the computation time required for
one frame of simulation rendering of tearing a 1 cm ×8 cm size sheet of paper, which is pulled in parallel withthe direction of ﬁber without and with the split-edge data
structure.

5.

Results and Evaluation

Figure 7 shows the results of pulling apart a 1 cm ×8 cm sheet of paper in the vertical direction. The numbers of the components used for the experiments are tabulated in Table 5. All of the simulated papers consistof approximately one quarter of ﬁber density of real photocopy paper. Note that this density was judged to be
maximum in our current execution environment due to
its GPU memory constraints. Further results could beobtained by utilizing a higher-grade GPU. We succeededin generating diﬀerent appearances of torn-oﬀ line of asheet of paper according as the direction of pulling.
In the case that the direction of pulling was in parallelwith the ﬁber direction (top), the tear became composedof the bumped parts and the dented ones because theﬁbers are arranged in such a way as to stick out fromthe line of the tear. While in the case of being pulled
orthogonally (middle), the shape of the tear is roundedIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Table 3Comparison of computation time for intersection detection (unit: msecs)
Number of
ﬁbers

Fiber
generation

Attachment
to cellIntersection
detection

Round-robin

3,900

61

-

104,111203

104,375

Our method

3,887

59

17

84

187

346

Total

Pulling perforated sheet

Pulling orthogonallyto the ﬁber direction

Pulling in parallelwith the ﬁber direction

Drawing(a) Without texture
Fig. 7
Table 4(b) With ﬁller and texture

Comparison of computation times for applying
split-edge data structure (unit: msecs)

Without applying

With applying

607,328

3

Table 5Top
MiddleBottom(c) Real paper

Comparison of the tear of virtual paper with that of real paper

Ratio
4.9 × 10−6

The numbers of component occurrences in Fig. 7
FibersFillerHinge
springs

Connection
points98,377
98,376
98,054

636,026
661,848627,542

6,353,922
6,611,460
6,289,268

1,131,043
1,131,664
1,124,802

and smooth because the force just causes to expand theconnection of ﬁbers. When the perforated sheet is torn112(bottom), the non-cut parts were torn in the area outside
the perforation and look like tearing of the real paper.
Figure 8 gives the enlarged images captured when
the sheet with texture pulled in parallel with the direction of ﬁber (Fig. 8 (a)) and when pulled orthogonally(Fig. 8 (b)). The sheet is applied the character texture
which is the dark area in Fig. 8. When comparing thetwo, the ﬁber stretches straight in Fig. 8 (a), whereas theﬁber network is expanded and the connection is released
in order from the sparse part of the network in Fig. 8 (b).
The computation time is diﬃcult to measure because
the moment when all of hinge springs between upper andlower part of paper are released should be detected. To

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a)(a) In parallel with the ﬁber direction

Fig. 9(b) Orthogonally to the ﬁber direction
Fig. 8Diﬀerence in the appearance of tears due to theﬁbers’ direction

achieve that detection, at least two kinds of detection areneeded: the tearing detection with all of the components
belonging to the ﬁber network and whether that tear occurs the global paper tearing detection for one entire paper. In order to execute these checks, we need to trace
the ﬁber network. It is needed to devise a graph search algorithm that solves the ﬁber network tracing, however it
is beyond our instant development of the algorithm withtracing ﬁbers in a reasonable simulation time. Hence thetimes for tearing simulation are not included in this paper.
In order to evaluate the physical validity of the simulation, we focused on the diﬀerence of the amount of paper
movement among the three cases of Fig. 7. Figure 9shows the state of the paper after 20 minutes from thestart of the simulation in Fig. 7. In the tearing simulation, the same amount of force is continuously applied toa model with the same density on the same time scale,and thereby the amount of movement is the same after
the paper is completely torn. Enomae 12) shows that thetensile strength measured when a sheet of photocopy paper is pulled in parallel with the ﬁber direction is signiﬁcantly greater than the one measured when being pulled

113

(b)

(c)

Papers in Fig. 7 twenty minutes after starting simulation: (a) In parallel with the ﬁber direction, (b)Orthogonally to the ﬁber direction, and (c) Perforated paper

orthogonally. We thought this could serve as a major criterion for delineating natural paper sheet deformation. In
fact, on the paper sheet pulled orthogonally to the ﬁber
direction in Fig. 9 (b), the position of printed charactersis shifted more than the one on the paper sheet pulled inparallel with the ﬁber direction in Fig. 9 (a). The perforated paper in Fig. 9 (c) is torn in the same direction
as in Fig. 9 (a), but the perforation weakened the paper
ﬁber sheet and thus the position of printed character isshifted more than the one in Fig. 9 (a), and less thanthe one in Fig. 9 (b). This provides a good proof thatthe tensile strength of the paper sheet pulled in parallelwith the ﬁber direction in Fig. 9 (a) is greater than thepaper sheet pulled orthogonally to the ﬁber direction in
Fig. 9 (b).
Figure 10 shows the tearing results varying thestrength of paper ﬁber sheet by the change in the amountof ﬁller. In the case of tearing the paper without ﬁller(top), the sheet tends to be narrow in the horizonal direction of the ﬁgure and the tendency that the sheet
stretches in the vertical direction is apparent from thedistorted texture. In contrast, in the case of adding ﬁller
twice as the quantity in Fig. 7 (bottom), the density of the
sheet is higher because the texture seems slightly darker.
Compared with the tearing result in Fig. 7, the bulge inthe vertical direction of the sheet is reduced. In addition,
the inﬂuence to the inside of the tear caused by the breakhardly occurs.
Figure 11 shows the result in tearing a postage stamp
size sheet of perforated paper with 102,234 ﬁbers, 702,184
ﬁller occurences, 7,022,298 hinge springs, and 1,176,054
connection points. This sheet is torn orthogonally to theprimary direction of the ﬁbers. This example exempliﬁes

With ﬁller twice as
the reference quantity

Without ﬁller

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) In parallel with the ﬁber direction
Fig. 10(b) Orthogonally to the ﬁber direction

Comparison of the diﬀerence between the amount of ﬁller(a) Original image(b) Resulting image
Fig. 11

A pair of torn postage stamps

the uneven tear compared to the machine cut part, andrepresents the feel of hand tearing.

6.

(c) Perforated paper

Conclusion and Future WorkIn this paper, we proposed a paper sheet tearing simulation model that takes into account the anisotropic microstructure and internal force generated by the paper
components, so that we have the following two characteristic properties:• Direction-dependent deformability and• Irregularity of torn-oﬀ lines.
As a result, we succeeded in representing the uneven or
rounded tear according to the direction of ﬁbers. Furthermore, we provided a good example to show the possibility that the object with the microstructure makes theappearance from a macro perspective more realistic.

114

Contrary to our paper model, the real ﬁbers are thickand branched, while the actual ﬁller is a plane in such a
way to ﬁll the space in neighboring ﬁbers. Thereby thesimulated papers look a little too much transparent andstretched. We plan to reproduce the ﬁller with mesh so
as to ﬁll the area surrounded by the ﬁbers, in order toincrease the density of paper ﬁber sheet. To this end,
the radical improvement of the data structure and incorporation of a novel computation algorithm of the repulsion force which produces an appropriate result using thatstructure are needed. In addition, the physical validity
for each of the parameters should be reconsidered. Since
these changes require substantial design and implementation as well as evaluation, we decided to have them left
as further research.
At this moment, the force is applied in such a way

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）that a sheet of paper is only pulled apart, because theforce computation is limited in two-dimensions. However, in the reality, it is common to tear a sheet of paperby twisting it. To incorporate the twisting force into ourmodel, the repulsion of the internal components against
the three-dimensional force must be considered. Thiskind of extension would be needed to represent more dailyinteractions with paper products.
We also plan to extend the present model so as to handle other deformations. For example, folding is considered as one of the highest aﬃnitive operations with ourpaper model because folded papers show diﬀerent results
depending on the ﬁber direction as well as tearing.
A real paper is made from multiple sheet layers andthus has a certain thickness. By considering the thickness
eﬀect in a thin volumetric model, we can express the inner
layers without ﬁller appearing at the top of paper sheet
when the coated paper is torn.

ACKNOWLEDGMENT
This work has been supported in part by JSPS KAKENHI under the Grant-in-Aid for Scientiﬁc Research (A)No. 17H00737.

References
1) C. Schreck, D. Rohmer, S. Hahmann: “Interactive Paper Tearing”, Computer Graphics Forum, Vol. 36, No. 2, pp. 95–106(2017).
2) T. Lejemble, A. Fondevilla, N. Durin, T. Blanc-Beyne,
C. Schreck, P.-L. Manteaux, P. G. Kry, M.-P. Cani: “Interactive Procedural Simulation of Paper Tearing with Sound”,
Proc. of the 8th ACM SIGGRAPH Conference on Motion inGames, pp. 143–149 (2015).
3) O. Busaryev, T. K. Dey, H. Wang: “Adaptive Fracture Simulation of Multi-Layered Thin Plates”, ACM Trans. on Graphics,
Vol. 32, No. 52, pp. 1–6 (2013).
4) T. Pfaﬀ, R. Narain, J. M. de Joya, J. F. O’Brien: “Adaptive
Tearing and Cracking of Thin Sheets”, ACM Trans. on Graphics, Vol. 33, No. 110, pp. 1–9 (2014).
5) S. Zhao, W. Jakob, S. Marschner, K. Bala: “Structure-aware
Synthesis for Predictive Woven Fabric Appearance”, ACM
Trans. on Graphics, Vol. 31, No. 75, pp. 1–10 (2012).
6) N. Metaaphanon, Y. Bando, B.-Y. Chen, T. Nishita: “Simulation of Tearing Cloth with Frayed Edges”, Computer Graphics
Forum, Vol. 28, No. 7, pp. 1837–1844 (2009).
7) C. Jiang, T. Gast, J. Teran: “Anisotropic Elastoplasticity for
Cloth, Knit and Hair Frictional Contact”, ACM Trans. on
Graphics, Vol. 36, No. 152, pp. 1–14 (2017).
8) D. Sulsky, Z. Chen, H. Schreyer: “A Particle Method forHistory-Dependent Materials”, Computer Methods in Applied
Mechanics and Engineering, Vol. 118, No. 1, pp. 179 – 196(1994).
9) S. Takagi, M. Nakajima, I. Fujishiro: “Volumetric Modeling ofColored Pencil Drawing”, Proc. of the 7th Paciﬁc Conference
on Computer Graphics and Applications, pp. 250–258 (1999).

11510) K. Kitani, T. Tanaka, Y. Sagawa: “CG Expression of Paper
Tearing”, The Trans. of the Institute of Electrical Engineers ofJapan, Vol. 128, No. 12, pp. 1735–1740 (2008).
11) M. Mäntylä, Introduction to Solid Modeling, New York, NY,
USA: W. H. Freeman & Co. (1988).
12) T. Enomae, Basics and Printability of Paper –Structure, Physical Property, Processing, and Printing Quality Evaluation–,
http://www.enomae.com/Paper%20Science%20seminar2/(2019).

(Received April 29, 2019)(Revised November 26, 2019)

Saeko SHINOZAKI
She received her B. E. and M. E. in computersciences in 2017 and 2019 both from Keio University. She is currently a master student ofGraduate School of Media and Governance,
Keio University. Her research interests include paper tearing simulation.

Masanori NAKAYAMA (Member)
He is currently a Research Fellow in the Department of Information and Computer Science at Keio University. His principal occupation is chief bonze at Ankokuin, Chiba. He
received his B.E. and M.E. at Keio University
in 2002 and 2004. His current research interests include photoreal-rendering, free-form
surface modeling, data processing based on
spherical geometry, panoramic display, stereogram, apparel CAD education, and 3D human body measurement. He is a member of
IPSJ and SAS.

Issei FUJISHIRO(Honorary Member)
He is currently a Professor at Department ofInformation and Computer Science, Facultyof Science and Technology, Keio University.
He received his B. E. and M. E. in information sciences and electronics in 1983 and 1985
both from University of Tsukuba and his Doctor of Science in information sciences from the
University of Tokyo in 1988. Before joining
Keio University in 2009, he worked as a faculty member for the University of Tokyo, University of Tsukuba, Ochanomizu University,
and Tohoku University. His research interests include modeling paradigms and shape
representations, applied visualization designand lifecycle management, and smart ambient media with multi-modal displays. He is amember of Science Council of Japan, a fellowof the Japan Federation of Engineering Societies, a honorary member of IIEEJ, and asenior member of Information Processing Society of Japan.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

7]WXIQ(IZIPSTQIRX4ETIV -- Special Issue on Journal Track Papers in IEVC2019 -JSV 'SPPEFSVEXMZI ERH )\TPSVEXSV] %WXVSRSQMGEP %REP]WMW
1EPMO 3PMZMIV &3977).6%]  6MOYS 9',-/-] 1IQFIV  7LYR]E 8%/)/%;%]]  /E^Y]E 1%879&%=%7,-]]] =YVMOS 8%/)7,-1%]]]] 1IQFIV  1EOSXS 9)196%]]]]]  -WWIM *9.-7,-63] ,SRSVEV] 1IQFIV] /IMS 9RMZIVWMX] ]] 2SFI]EQE 6EHMS 3FWIVZEXSV] ]]] /]SXS 9RMZIVWMX] ]]]] 8SO]S 9RMZIVWMX] SJ 8IGLRSPSK]  ]]]]] ,MVSWLMQE 9RMZIVWMX]

7YQQEV]"/I][SVHW EWXVSRSQ] TVSZIRERGI ZMWYEP TVSKVEQQMRK ZMWYEPM^EXMSRGIMZIH F] E TM\IP W WIRWSV SV E ZEPYI HIVMZIH JVSQ MX-RXVSHYGXMSRIK VEHMEP WTIIH XS[EVHW XLI )EVXL HIVMZIH XLEROWGSPPEFSVEXMZI WYTTSVX EQSRK VIWIEVGLIVW MR SVHIV XS QEOILEVHP] I\MWXW ER]QSVI MR VIEP PMJI 8LMW MW RSX SRP] WLS[RF] XLI IZIV MRGVIEWMRK EZIVEKI RYQFIV SJ GSEYXLSVW MRTETIVW FYX EW [IPP F] XLI PEXIWX LMKLP] QIHMEXM^IH XIGLRSPSKMGEP HIZIPSTQIRX XLEX EPPS[IH XLI 1 FPEGO LSPIXS FI HMVIGXP] MQEKIH -QEKMRK WYGL E JEV E[E] SFNIGX VIUYMVIW E XIPIWGSTI VSYKLP] XLI WM^I SJ XLI )EVXL XS QMXMKEXI[MXL ER ERKYPEV WM^I EW WQEPP EW 1 7YGL JIEX GERSRP] FI HSRI F] GVIEXMRK E ZMVXYEP XIPIWGSTI XLVSYKL XLIGSSTIVEXMSR SJ QER] XIPIWGSTIW WGEXXIVIH SZIV XLI [LSPI)EVXL`XLI TETIV F] %OM]EQE IX EP XLEX VIZIEPIH XLI%WXVSTL]WMGEP HEXE X]TMGEPP] GSRWMWXW SJ QYPXMWTIGXVEP
TL]WMGMWX QE] HMZI MR SRI QE] IRGSYRXIV HEXEWIXW [MXLWTEXMEP E\IW YWYEPP] VMKLX EWGIRWMSR ERH HIGPMREXMSR FYX;EZIPIRKXL

8LI WTEGMEP TIVMSH SJ E KMZIR VE] VI
116

IX EP  
SV SFNIGXW IK FPE^EVW IQMX E TSPEVM^IH PMKLX  
8MQI(]REQMG TLIRSQIRSR QE] MRGPYHI XMQI:EVMSYW WSJX[EVI XSSPOMXW EVI VIKYPEVP] YWIH F] EWXVSTL]WMGMWXW 7SQI XSSPW EVI QEMRP] ZMI[IVW YWIH XS ZMWY
EPM^I VE[ HEXE SV XLI VIWYPX SJ ER EREP]WMW [LMPI SXLIVJIVIRX VIWIEVGLIVW -R E [SVH ER MRXIKVEXIH IRZMVSRQIRX@7[MWW %VQ] ORMJI JSV EWXVSRSQMGEP EREP]WMW ERH ZMWYEP
M^EXMSR LEW ]IX XS ETTIEV &IWMHIW [LIVIEW HEXEWIXW QE]
EPVIEH] FI WLEVIH ZME STIR VITSWMXSVMIW RSX XLI WEQI GER
FI WEMH EFSYX XLI GSQTPIXI VE[ EREP]WMW TVSGIWW [LMGL
MRZSPZIW IZIV]XLMRK JVSQ XLI SVMKMREP HEXE XS XLI SYXTYX
8LMW SJ GSYVWI MRGPYHIW XLI EREP]XMGEP TVSKVEQ HIZMWIHF] XLI EWXVSRSQIV FYX EW [IPP EPP XLI WXITW F] [LMGL XLMWIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）FIXXIV WLEVMRK TVEGXMGIW SJ EREP]XMGEP TVSGIWWIW ERH TVSZIRERGI HEXE XS MQTVSZI VITVSHYGMFMPMX] ERH TSXIRXMEP JSV
GSPPEFSVEXMSRLIEZ] GYWXSQM^EFMPMX] SR XLI QETTMRK ERH VIRHIVMRK SJEGUYMVIH HEXE IK JVSQ QIEWYVIQIRXW SV RYQIVMGEP WMQYPEXMSRW  8LI] YWI XLI GSRGITX SJ RSHI XS VITVIWIRX E[SVO JSV 0IEVRMRK %WXVSTL]WMGEP /RS[PIHKI EMQW EX FIGSQMRK XLI @7[MWW %VQ] ORMJI SJ QYPXMWTIGXVEP HEXEERH ZMWYEPM^EXMSR TVMQMXMZIW JSV XLI EWXVSRSQMGEP HSQEMRERH RSRI TVSZMHIW VIYWEFPI I\TSVXEFPI ERH MRXIVEGXMZIP]TVSKVEQQMRK IRZMVSRQIRX XLEX EPPS[W XS PSEH E HEXEWIXXS ETTP] XVERWJSVQEXMSRW SR MX ERH XS ZMWYEPM^I XLI SYXTYXW SJ XLIWI XVERWJSVQEXMSRW MR VIEP XMQI XLYW TVSZMHMRKE JEWX ERH WQSSXL JIIHFEGO PSST XS EWXVSRSQIVW WMKRMJMGERXP] JEWXIV XLER XLI QER] XSSPW YWIH MR EWXVSRSQMGEPZIPSTIH XS WSPZI EWXVSRSQ] MWWYIW:MI[IVW ERH EREP]^IVW JSV EWXVSRSQMGEPYWIHEVH JSV EWXVSTL]WMGEP MQEKIW   ERH GSQQSR HEXE XVERW _ 8LI EWSJ XLIQ [MXL XLIMV S[R MHMSW]RGVEWMIW ERH WTIGMEPM^EXMSRWXVSRSQIV F] FIMRK EFPI XS W]WXIQEXMGEPP] GLIGO XLI ZM
WYEPM^IH SYXTYX SJ XLI TVSKVEQ EW XLI ZMWYEP TVSKVEQ MW

STIR WSYVGI WSJX[EVI XLI ETTEVIRXP] QSWX JEQSYW ERHQSWX YWIH SJ [LMGL MW 7%3-QEKI (7   [LMGL GER STIRSV VETMHP] TVSXSX]TI ER EREP]XMGW TMTIPMRIYRHIVXEOMRK XS VIYWI I\MWXMRK JVII QSHIPMRK WSJX[EVI WYGL ;I GEREW [IPP MHIRXMJ] RI[ HIZIPSTQIRXW XS ZMWYEPM^I ZIV] PEVKI
HEXEWIXW XLEX GERRSX FI PSEHIH MR E WMRKPI QSHIVR GSQTYXIV W VYRRMRK QIQSV] %W WIRWSV XIGLRSPSK] MQTVSZIWMX GER SRP] FI I\TIGXIH XLEX WYGL XSSPW [MPP FIGSQI IZIVQSVI MQTSVXERX MR XLI JYXYVI   
2SRIXLIPIWW XLI XSSPW EFSZI EVI ZMI[IVW ERH XLI]HS RSX TVSZMHI QER] HEXE EREP]XMGW JIEXYVIW MJ ER]IWTIGMEPP] XLI HIWMKR HIGMWMSRW ERH XLI MQTPIQIRXEXMSR SJQEGVS WYTTSVX MR WYGL E [E] XLEX MRXIVEGXMZI EREP]WMW

6IPEXIH ;SVO

1SHYPEV ZMWYEPM^EXMSR IRZMVSRQIRXW7IZIVEP XSSPW QEOI YWI SJ E ZMWYEP TVSKVEQQMRK ET
TVSEGL MR SVHIV XS EGLMIZI FIXXIV EGGIWWMFMPMX] JSV HSQEMR
I\TIVXW %:7 SV -&1 (EXE )\TPSVIV EVI XLI TMSRIIVWSJ ZMWYEP TVSKVEQQMRK W]WXIQW JSV ZMWYEPM^EXMSR ETTPMGEXMSRW [LMGL LEZI FIIR VIPIEWIH EW GSQQIVGMEP WSJX[EVI JVSQ XLI W   4EVOIV I\TPEMRW XLEX 7'-6YR
WYEP TEVEHMKQ XS MRGPYHI RYQIVMGEP WMQYPEXMSRW   8LMW
GPEWW SJ WSJX[EVI MW GEPPIH @QSHYPEV ZMWYEPM^EXMSR IRZMVSRQIRXW 7YGL IRZMVSRQIRXW EVI LMKLP] HIXEMPIH MR E
TETIV F] &YRKEVX^   8LI FEWI SJ WYGL QSHYPEV ZMWY
EPM^EXMSR IRZMVSRQIRX W]WXIQW MW XLEX EW XLI REQI MRHMGEXIW XLI] WLSYPH FI @QSHYPEV MR XLEX XLI] MRGPYHIE YWIVI\XIRHEFPI QSHYPI PMFVEV] @:MWYEPM^EXMSRW EVIYWIH XS VITVIWIRX XLI SYXTYX SJ XLI TVSKVEQ 1SVISZIVXLI] TVSZMHI XLI YWIV [MXL ER MRXIKVEXIH ZMWYEP TVSKVEQQMRK @IRZMVSRQIRX
8LIWI QSHYPEV ZMWYEPM^EXMSR IRZMVSRQIRXW TVSGIWW XLI

3XLIV GSRGYVVIRX XSSPW EVI MR GLEVKI SJ HEXE EREP]XMGW XLI SPHIWX SJ [LMGL FIMRK -6%*   8LIR 4]6%*[EW HIZIPSTIH [MXL XLI WYGGIWWJYP SFNIGXMZI SJ TVSZMHMRKE QSVI YWIVJVMIRHP] TVSKVEQQMRK W]RXE\ XS XLI -6%*
IRZMVSRQIRX   4]6%* [MXL E 4]XLSRFEWIH WLIPP W]R
XE\ MRGPYHIW E FYMPXMR MQEKI EPKIFVE ERH QER] XVERWJSVQEXMSRW VSYXMRIP] YWIH MR EWXVSTL]WMGW 4]6%* GER I\IGYXI WGVMTXW XS IREFPI GSHI VIYWI 8LIR GEQI %WXVST] E
4]XLSR PMFVEV] XLEX WSPZIW QSWX SJ XLI GSQTYXMRK RIIHW SJEWXVSTL]WMGMWXW   %WXVSRSQMGEP EREP]WMW GER RS[ FI GSRIXG  EX XLI GSWX JSV XLI YWIV SJ [VMXMRK XLI 4]XLSR GSHI
SR SRI W S[R
2S[ [I GER WII XLI YRJSVXYREXI HMZMHI FIX[IIR ZM
WYEPM^EXMSR ERH EREP]WMW XSSPW MR XLI EWXVSRSQ] WSJX[EVITSWMRK XVERWJSVQEXMSRW SR XLIQ SRP] XLIR HS XLI] I\TSVX
ERH VIRHIVMRK 8LI] LEZI MR GSQQSR XLEX XLI] PIZIVEKIZMWYEP TVSKVEQQMRK MR SVHIV XS GSRGIRXVEXI ERH EPPS[
117

E ZMI[IV )ZIR JSV XLI [MHIP] EGGPEMQIH %WXVST] I\XIVIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

REP PMFVEVMIW IK @QEXTPSXPMF EVI RIGIWWEV] XS ZMWYEPM^IEFPI XS MRXIVEGXMZIP] VITVSHYGI XLI I\EGX WEQI RYQIVMXLI VIWYPXW -R XLMW JEWLMSR XLI WSJX[EVI IGSW]WXIQ JEMPWXS TVSZMHI E WMRKPI MRXIKVEXI IRZMVSRQIRX EPPS[MRK VETMHTVSXSX]TMRK SJ EREP]XMGW TMTIPMRIW 8LIVI MW XSS QYGL PEKGEP EREP]WIW XLER XLI GSRZIRXMSREP EWXVSRSQ] XSSPW 8LI
VIEHIV QE] VIJIV XS *MK 

FIX[IIR XLI GSQTSWMXMSR SJ E TVSKVEQ ERH XLI ZMWYEPM^EXMSR SJ MXW SYXTYX -HIEPP] WYGL W]WXIQ [SYPH WXMQYPEXIWSR IX EP [MXL XLI @LYQERMRXLIPSST XIVQMRSPSK]  8LI LYQER MI XLI EWXVSRSQIV QYWX FI OITX MR XLI(IWMKR KSEPWMW HIWMKRIH XS QIIX XLI JSPPS[MRK VIUYMVIQIRXW

6IYWEFMPMX] ERH I\XIRHEFMPMX]PSST HYVMRK XLI HIWMKRMRK SJ ER EREP]XMGW TMTIPMRI [LMGL
EFPI GSQTSWMXI RSHIW VIJIVVIH XS EW @QEGVS LIVIEJXIV7]WXIQ 3ZIVZMI['SRZIRMIRX QEGVSW WLEPP RSX MQTIHI XLI VIWTSRWMZIRIWW
VIUYMVIQIRXW WXEXIH FIPS[ 1EGVSW GER FI WLEVIH EQSRK'VMXIVME JSV E RI[ EWXVSRSQMGEP EREP]WMWXSSPQSVI RSHIW QE] FI MQTPIQIRXIH EW WMHIPSEHIH WLEVIH;I LEZI WYVZI]IH KIRIVEP ZMWYEP TVSKVEQQMRK W]WXIQWXMSR  'SQFMRMRK XLI QSHYPEV JIEXYVIW SJ KIRIVEP ZM
SJ EWXVSRSQ] MW XLI WYGGIWW FIRGLQEVO JSV XLI W]WXIQ XLMWTETIV TVIWIRXW *VSQ XLIR JSV XLI W]WXIQ XS FI SJ YWIJSV EWXVSRSQIVW MX VIUYMVIW XLI JSYV JSPPS[MRK GVMXIVME )REFPI GSPPEFSVEXMSR XLVSYKL WLEVIEFMPMX]JIEXYVIW 1EOI LYQERMRXLIPSST E VIEPMX] ERH -RGPYHI IRHXSIRH TVSZIRERGI QEREKIQIRXJSPPS[W E ZMWYEP TVSKVEQQMRK ETTVSEGL WMQMPEV XS XLIXSSPW QIRXMSRIH MR WIGXMSR  SJ GSQFMRMRK RSHIW [MXLMR
E RSHI KVETL XS EPPS[ XLI YWIV XS GSQTSWI EPKIFVEMG XVERWERH E RYQFIV SJ SYXTYX WPSXW JVSQ [LMGL XLI XVERWJSVQIH
RHMQIRWMSREP MQEKI
EPKIFVE MRXIVJEGI  WMQMPEV XS XLEX SJ 2YQ4] [LMGL GER
FI YWIH XS FYMPH EREP]XMGW TMTIPMRIW ERH WQSSXLP] ZMWYEPM^I XLI VIWYPXMRK GSQTYXEXMSRW
8LIR XS KIX EVSYRH XLI WLSVXGSQMRKW SJ GSRZIRXMSREPXMZI  _ MW XS TVSZMHI E YRMZIVWEP GSPPEFSVEXMZI ERH MRXIKVEXIH IRZMVSRQIRX XS EREP]^I ERH ZMI[ EWXVSRSQMGEPHEXE [MXL ZIV] JEWX MXIVEXMSRW ;LMPI @QEXTPSXPMF TVSZMHIW TYFPMWLMRK UYEPMX] KVETLW MX MW JEV JVSQ WYMXEFPIJSV JEWX MXIVEXMSRW SR VIPEXMZIP] FMK HEXEWIXW 1SVISZIVXLIVI MW RS FYMPXMR WSPYXMSR JSV GSHI WLEVMRK EQSRK VI
FYX XLIR RS TVSZIRERGI MW WYTTSVXIH VIHYGMRK EGGIWWM

118

KYEKI XLEX WYTTSVXW '' GEPPMRK GSRZIRXMSRWEMQW EX QEOMRK GSPPEFSVEXMZI HIZIPSTQIRX ERH GSHIWLEVMRK EW IEW] EW TSWWMFPI ZME MXW MQTSVXI\TSVX JIEXYVIWERH GSQTPMERGI [MXL WXERHEVHW MR EWXVSRSQ]


7TIGMEPM^IH XS XEGOPI EWXVSRSQMGEP MWWYIW

HEVHW MR EWXVSRSQ] *-87 IXG  FYX MX HSIW EW [IPP EHETXQSHYPEV ZMWYEPM^EXMSR IRZMVSRQIRX GSRGITXW JSV QSVI WTIZMWYEPM^EXMSR IRZMVSRQIRXW ERH [I WE[ XLEX XLIMV ZM
WYEP TVSKVEQQMRK MRXIVJEGIW GSRGIRXVEXI SR XLI HS[R
WXVIEQ QETTMRK ERH VIRHIVMRK TLEWIW SJ XLI ZMWYEPM^EXMSR

TPIXI MRXIVEGXMZI JVIIHSQ SZIV MRTYXW ERH EREP]WIW HS[R
WXVIEQ ZMWYEPM^EXMSRW YWI PMKLXP] TEVEQIXIVM^EFPI FYMPXMRJIEXYVIW XEMPSVIH JSV EWXVSRSQ]
)EWI SJ YWI ERH VIWTSRWMZIRIWW%GLMIZMRK EWXVSRSQIVMRXLIPSST

MW GPIEVP] MRHMGEXIH F] GSRRIGXMSRW FIX[IIR FS\WLETIH
RSHIW 8LI MRXIVJEGI MW VIWTSRWMZI XLI SYXTYX SJ E ZMWYEPTVSKVEQ MW VIJVIWLIH MR VIEP XMQI EW XLI TVSKVEQ MW FIMRKYTHEXIH [MXL ZIV] QMRSV HIPE] WS XLEX XLI YWIV RSX FIGSRJYWIH EFSYX XLI TVSZIRERGI SJ XLI HEXE FIMRK WLS[R1SVISZIV IVVSVW EVI WLS[R YWMRK E WXEGO XVEGI XS WLS[
EX [LMGL I\EGX RSHI ER IVVSV SGGYVVIHIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
2SHI
%ZIVEKISRPIJXGSRXMRYYQ

3YXTYXRSHI
2SHI
%ZIVEKISRIQMWWMSRPMRI

2SHI)\XVEGXHEXEGYFIJVSQ*-87 PI

3YXTYXRSHI3YXTYX
RSHI

2SHI
%ZIVEKISRVMKLXGSRXMRYYQ

3YXTYX[MRHS[1EOIIUYMZEPIRX[MHXL3YXTYX[MRHS[3YXTYX[MRHS[8LVIWLSPHWEVIMRXIVEGXMZIP]WIX
ERHW]RGIHMRXLIRSHIIHMXSV'328-2991)1-77-32'328-2991)1-77-32

0-2))1-77-32
6-+,8

0)*8

*MK 



4VSZIRERGI QEREKIQIRX7SQI EWXVSRSQIVW LMKLPMKLX XLI MQTSVXERGI SJ TVSZIRERGI QEREKIQIRX % TVSZIRERGI HEXE QSHIP WXERHEVH

8VERWJSVQEXMSR RSHIW'SQTSWEFPI XVERWJSVQEXMSR

MW GYVVIRXP] MR HIZIPSTQIRX F] XLI -RXIVREXMSREP :MVXYEP
3FWIVZEXSV] %PPMERGI [LSWI REQI MW 4VSZIRERGI(1  ZIV] WMQMPEV XS XLEX SJ 4VSZIRERGI(1 8LMW HEXE QSHIPMW WXMPP E HVEJX ERH HSIW RSX LEZI ER] VIJIVIRGI MQTPI

4VSZIRERGI(1 HEXE QSHIP(IWGVMTXMSR[LIVI RSHIW GER FI JVIIP] GSQTSWIH F] PMROMRK E RSHI W
SYXTYX WPSX XS ERSXLIV RSHI W MRTYX WPSX [MXL XLI QSYWIGYVWSV    8LI RSHI IHMXSV W PIJX TERI GSRXEMRW XLI RSHIPMWX ERH XLI HSGYQIRXEXMSR SJ XLI GYVVIRXP] WIPIGXIH RSHI
&] WIPIGXMRK E RSHI MR XLI RSHI PMWX XLI ZMI[ [MPP HM

:EPYI RSHIW 'SRXEMR E TEVEQIXIV SJ E GIVXEMR HEXEX]TI XLEX GER FI MRTYX F] XLI YWIV SV XLEX GER FI
I\XIVREPP] WIX F] ERSXLIV XSSPHEXE X]TI GER FI VIHMVIGXIH XS ER SYXTYX RSHI ;LIR
E RI[ SYXTYX RSHI MW GVIEXIH E RI[ SYXTYX [MRHS[GSRXEMRMRK E ZMWYEPM^EXMSR SJ XLI HEXE EVVMZMRK XS XLMWRSHI MW STIRIH 8LI SYXTYX [MRHS[ ERH XLI SYXTYXRSHI WLEVIW XLI WEQI RYQFIV 7YGL SYXTYX [MRHS[WGER FI WIIR MR XLI QMHHPI SJ *MK % RSHI QE] EW [IPP FI XLI VIWYPX SJ XLI GSQTSWMXMSR SJWIZIVEP RSHIW 8LMW JIEXYVI MW VIJIVVIH XS EW QEGVS 0EXIVWIGXMSR  [MPP I\TERH JYVXLIV MR HIXEMPW LS[ QEGVSW EVI(IWGVMTXMSR SJ %PKSVMXLQW-R SVHIV XS EGLMIZI LMKL WTIIH ERH VIWTSRWMZIRIWW GEVIRSHI KVETL 8LI RSHI KVETL FEWMGEPP] VIEHW JVSQ PIJXMRTYX XS VMKLX SYXTYX  MR EGGSVHERGI [MXL XLI PIJXXSVMKLX GSRZIRXMSR SJ QSWX [VMXMRK W]WXIQW119

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

JSVQERGI [LMPI TVSZMHMRK E LMKLIVPIZIP W]RXE\ ERH E

8EFPI WEJIV TEVEHMKQ XS [VMXI LMKLP] GSRGYVVIRX TVSKVEQWFSSWXMRK XLI TVSHYGXMZMX] SJ XLI MQTPIQIRXIV %PP SJ GSQ(78
XVERWJSVQW
IHKIW
SYXTYXW-RTYX
XVERWJSVQ MH
MRTYX MRHI\
3YXTYX
XVERWJSVQ MHSYXTYX MRHI\GEOI EOE GEOI WXERHMRK
JSV 'SQTYXEXMSREP Q%/)  *MVWX [I [MPP TVIWIRX XLI FE
WMG WXVYGXYVI SJ GEOI I\GPYHMRK QEGVS WYTTSVX 8LIR
[I [MPP I\TPEMR LS[ QEGVSW EVI MQTPIQIRXIH [LMPI LMKLPMKLXMRK XLI GLSMGI SJ HEXE WXVYGXYVIWGEOI 'SQTYXEXMSR Q%/)(]REQMG 7]RXE\ 8VII HEXE WXVYGXYVI
1 ETL8 VERWJ SVQ-H 1 IXE8 VERWJ SVQM
1 ETL3YXTYX 0MWXL-RTYXMM
1 ETL3YXTYX-H 3TXMSRL3YXTYXMM
8 VERWJ SVQ-H
9RWMKRIH -RXIKIV
8 VERWJ SVQ-H
9RWMKRIH -RXIKIV8EFPI GEOI W QEMR HEXE WXVYGXYVI MW GEPPIH @(78  JSV (]
8EFPI ;LEX QYWX FI OITX MR QMRH MW XLEX E (78 VITVIWIRXWE HMVIGXIH EG]GPMG KVETL SJ RSHIW 8S VITVIWIRX WYGLKVETL [I QYWX OIIT E GSPPIGXMSR SJ XVERWJSVQEXMSR RSHIWJSVQEXMSR RSHI ER MRWXERGI SJ 8 VERWJ SVQ-H MW YWIH EW[LSWI MRRIV HEXE MW GSRXEMRIH MR E 1 IXE8 VERWJ SVQ HEXEWXVYGXYVI MW XLIR MRWIVXIH MRXS XLI WSVXIH QET GSRXEMRMRKEPP XVERWJSVQW&YX OIITMRK XLI GSPPIGXMSR SJ XVERWJSVQEXMSR RSHIW MW
RSX IRSYKL IHKIW MR XLI HMVIGXIH EG]GPMG KVETL QYWX EW[IPP FI OITX MR QIQSV] 8LMW MW HSRI MR XLI @IHKIW1IXE8VERWJSVQ HEXE WXVYGXYVI

1 IXE8 VERWJ SVQ
X
8 VERWJ SVQMRTYX HIJEYPXW 0MWXL3TXMSRL8 MM[LIVI 8 MW XLI X]TI SJ XLI ZEPYIWTEWWIH EVSYRH [LIR E KVETL MW GSQTYXIH
YTHEXIH SR-RWXERXXYVI 3TXMSR 3YXTYX ;LIR E (78 MRWXERGI MW GSRWXVYGXIH ERH IZIV] XMQI E
RI[ RSHI MW EHHIH XLI KVETL MW GLIGOIH JSV GSRWMWXIRG]
*SV I\EQTPI RS GMVGYPEV TEXL GER IZIV FI GVIEXIH GEOI[MPP RSX EPPS[ XLEX ERH KVEGIJYPP] EFSVX XLI EHHMXMSR SJ E
WGVIIR XS PIX XLI YWIV ORS[ [LEX IVVSV [EW WYGGIWWJYPP]EZSMHIH

1 IXE8 VERWJ SVQ HEXE WXVYGXYVI%W VIJIVIRGIH MR 8EFPI  XLI GSRXIRX SJ E RSHI MW WXSVIH3YXTYX VITVIWIRXW XLI RXL SYXTYX WPSX SJ XLI RSHI MR
HI\IH F] 8 VERWJ SVQ-H EW XLI XYTPI 8 VERWJ SVQ-H R 7MQMPEVP] XLI HEXE WXVYGXYVI -RTYX VITVIWIRXW XLI RXL
MRTYX WPSX SJ XLI RSHI MRHI\IH F] 8 VERWJ SVQ-H EW XLIXYTPI 8 VERWJ SVQ-H R  8LI GSPPIGXMSR SJ IHKIW SJ E
PMWX SJ MRTYXW EW WLS[R MR 8EFPI  -RHIIH E WMRKPI SYXWPSXW SRIXSQER] VIPEXMSRWLMT  8LI SXLIV [E] EVSYRHWMRKPI MRTYX WPSX EXXEGLIH XS QER] SYXTYX WPSX HSIW RSXXS ER SXLIV RSHI W W MRTYX WPSX W 

3YXTYX-H  )EGL 3YXTYX-H MW EWWSGMEXIH [MXL ER SYXTYXWPSX`VITVIWIRXIH [MXL XLI HEXE WXVYGXYVI 3YXTYXEFSZI` SV [MXL RSXLMRK YREXXEGLIH  8LI GSRGITX @ERSYXTYX WPSX SV RSXLMRK MW VITVIWIRXIH EW XLI HEXE WXVYG
120

EW E 1 IXE8 VERWJ SVQ% 1 IXE8 VERWJ SVQ MW E 8 VERWJ SVQ XS [LMGL QIXEHEXE
MW EHHIH 8LI 8 VERWJ SVQ GSRXEMRW XLI EGXYEP HEXE HIWGVMFMRK LS[ XLI GSQTYXEXMSR JSV XLMW RSHI WLSYPH FI TVSGIWWIH 1 IXE8 VERWJ SVQ W QIXEHEXE GSRXEMRW XLI @MRTYX
VITVIWIRXW XLI YWIVIHMXEFPI HIJEYPX MRTYX ZEPYIW JSV XLIRSHI ;LIRIZIV XLI RSHI MW YTHEXIH IK F] YTHEXMRKE HIJEYPX MRTYX ZEPYI  XLI @YTHEXIHER MRWXERX MW RSX VIEPP] E XMQIWXEQT FYX E QIEWYVIQIRXSJ E QSRSXSRMGEPP] RSRHIGVIEWMRK GPSGO SR [LMGL XLIRSHI [EW YTHEXIH %W [I [MPP WII PEXIV MR WIGXMSR SR GEGLI WXSVMRK XLI MRWXERX EX [LMGL XLI RSHI MW YTHEXIH
MW GVYGMEP WS XLEX GEGLI FI JYRGXMSREP
8 VERWJ SVQ W HEXE WXVYGXYVI XEFYPEXIH MR 8EFPI  MW
NYWX E XYTPI SJ ER %PKSVMXLQ ERH XLI MRWXERX SR [LMGL
XLI %PKSVMXLQ HEXE WXVYGXYVI [EW YTHEXIH %PKSVMXLQEW WLS[R MR 8EFPI  MW ER IRYQIVEXMSR XLEX GER FI SRISJ XLI XLVII JSPPS[MRK ZEVMERXWIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）*YRGXMSR % TSMRXIV XS E TYVI JYRGXMSR XLEX XEOIW ER
EVVE] SJ HEXE SJ X]TI 8 ERH VIXYVRW ER EVVE] SJ HEXE SJX]TI IMXLIV 8 SV ) [LIVI 8 MW XLI HEXE X]TI SR [LMGL
GSQTYXEXMSRW EVI HSRI ER IRYQIVEXMSR SZIV QYPXM

-M2 MW RSX EXXEGLIH XS ER] SYXTYX WPSX ERH LEW RS
HIJEYPX ZEPYI -R XLEX GEWI SRN MW YRGSQTYXEFPI ERH
ER IVVSV MRHMGEXMRK XLEX E HITIRHIRG] MW QMWWMRK MWVEMWIH ERH TVSTEKEXIH
-M2 LEW ER EWWSGMEXIH HIJEYPX ZEPYI \ ERH MW RSX EX

XI\X ERH ) MW ER IVVSV X]TI XLEX GER FI VITVIWIRXIH
EW E LYQERVIEHEFPI WXVMRK JSV IEWI SJ HIFYKKMRKXEGLIH XS ER] SYXTYX WPSX 8LIR M2
M\
HIWMKREXMRK'SRWXERX % GSRWXERX SJ X]TI 8  8LMW EGXYEPP] VITVI-M2 MW EXXEGLIH XS ER SYXTYX WPSX WE] 3N(N XL SYXTYX SJ RSHI ( [LIVI ( MW E HITIRHIRG] SJS(
2 EXXEGLIH XS -M2  8LIR M2
MN 

WIRXW E GSRWXERX RSHI 7YGL E RSHI LEW RS MRTYX WPSXERH LEW E WMRKPI SYXTYX WPSX XLEX EP[E]W VIXYVRW 8 W
ZEPYI
1EGVS 'SRXEMRW E LERHPI XS E QEGVS 1 EGVS,ERHPIWII WIGXMSR  %PP MR EPP [I RS[ LEZI I\TPEMRIH EPP XLI HEXE WXVYGXYVIWYWIH XS WXSVI E RSHI KVETL ;I [MPP XLIR WII LS[ XLI
GSQTYXMRK MW EGXYEPP] VYRRMRK



'SQTYXMRK SYXTYX [MXL GEGLI'SQTYXEXMSR QSHIP+MZIR E (78 VITVIWIRXMRK E RSHI KVETL [I MXIVEXI;MXL XLI EFSZI TVIQMWIW [I GER XLIR EWWYVIHP] GSQ2
2
TYXI J M2   MR  ERH XLYW [I LEZI SN  8LI ZEPYIW SJHITIRHIRGMIW WYGL XLEX S(
N  GER FI VIGYVWMZIP] GSQTYXIH

8LI VIGYVWMSR [MPP XIVQMREXI EW XLI KVETL LEW RS G]GPI
2WXEVX WYGL EWRS SJ 2WXEVX W MRTYX WPSXW EVI EXXEGLIH XS ERSXLIV RSHI W
SYXTYX WPSX %HHMRK GEGLI XS XLI QSHIP6IGYVWMZIP] VIGSQTYXMRK XLI ZEPYI EX ER SYXTYX WPSXJVSQ WGVEXGL IZIV] XMQI XLI ZEPYI QYWX FI HMWTPE]IH

3YXTYX-H *SV
ZEPYI EX XLMW SYXTYX WPSX MW GSQTYXIH ERH XLIR VIXYVRIH
% ZEPYI EX ER SYXTYX WPSX MW GSQTYXIH EW JSPPS[W 0IX
2 E RSHI [MXL R MRTYX WPSXW ERH Q SYXTYX WPSXW *SV
M ERH N X[S RSR^IVS MRXIKIVW -M2
2 W MXLMRTYX WPSX ERH 3N2
2 W NXL SYXTYX WPSX 8LI
ZEPYI EX XLI MXL SYXTYX WPSX SJ RSHI 2 MW RSXIH EW S2M [LMPI ZEPYI EX XLI NXL SYXTYX WPSX SJ RSHI 2 MW RSXIH EW
S2
N 
2
2
2
0IX J XLI JYRGXMSR WS XLEX J M2   MR ! S   SQ 
MI J MW IUYMZEPIRX XS XLI JYRGXMSR XLEX MW IZEPYEXIH [LIR
2HEXE M2   MR IRXIVW RSHI 2 
0IX W WE] [I [ERX XS GSQTYXI SRI SJ XLI S2JSVN N Q *SV IEGL M WYGL EW 
M
R [I LEZI IM
XLIV SRI SJ XLI XLVII JSPPS[MRK GEWIW8EFPI 8VERWJSVQ HEXE WXVYGXYVI

0IX 3N2 XLI NXL SYXTYX WPSX SJ RSHI 2
XLI WQEPPIWX MRWXERX MR XMQI X32 [LIR XLI SYXTYX WPSX W MWXLISVIXMGEPP] GSQTYXEFPI XS XLI ZEPYI S2N MR XLI GYVVIRXRSHI KVETL W WXEXISR JYRGXMSR XLEX GSQTYXIW X32
2YTHEXIH SR&& 2XYTHEXIH SR&& YTHEXIH SR (

&
&

&
YTHEXIH SR 2 ! QE\ &&& YTHEXIH SR (M
&
&

&

$
YTHEXIH SR (R


'
'
'
'
'
'
'
' 
'
'
'
'
'
%

[LIVI 2 MW E RSHI VITVIWIRXIH EW ER MRWXERGI SJ XLIHEXEX]TI 1 IXE8 VERWJ SVQ WII 8EFPI   2X MW 2 W
8 VERWJ SVQ ERH (M JSV  M R XLI R HITIRHIRGMIWSJ 2 R QE] FI  MJ 2 LEW RS HITIRHIRG] MR XLEX GEWIXLI VIGYVWMSR IRHW 
2  YTHEXIH SR 2 MW
XLI WQEPPIWX MRWXERX MR XMQI [LIR XLI SYXTYXW SJ 2
EVI GSQTYXEFPI MR XLI GYVVIRX WXEXI SJ XLI JYPP RSHIKVETL;I XLIR GSQTYXI EPP XLI S2
ERH ETTIRHN
2YTHEXIH SR 2 XS XLI ZEPYI SN  WXSVMRK XLIQ FSXL MRXS

8 VERWJ SVQEPKSVMXLQ%PKSVMXLQYTHEXIH SR -RWXERX8EFPI 8LI ETTVSEGL YWIH MW XS OIIT XVEGO SJ XLI MRWXERXW [LIR

(EXE WXVYGXYVI SJ %PKSVMXLQ IRYQIVEXMSR%PKSVMXLQ W ZEVMERX PMWX*YRGXMSR 4YVI JYRGXMSR TSMRXIV [MXL WSQI QIXEHEXE
REQI HIWGVMTXMSR ZIVWMSR RYQFIV HI
JEYPX ZEPYIW MRTYX ERH SYXTYX X]TIW IXG'SRWXERX 8
1EGVS
1 EGVS,ERHPI WII WIGXMSR 

121

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）QIQSV]

'SRWMHIVMRK XLEX [I EVI RS[ EX XMQI X 
PIX X32 ! YTHEXIH SR 2  2I\X XMQI S2
N RIIHW XS FI
GSQTYXIH EKEMR WE] EX XMQI X  MJ X32 ! X32 XLIRXLIVI [EW RS GLERKI MR XLI KVETL WXEXI XLEX [SYPH GEYWIS2N XS GLERKI WS [I NYWX VIXVMIZI XLI GEGLIH ZEPYI -J SR
XLI GSRXVEV] X 32 " X 32  XLIR S2
N [MPP FI VIGSQTYXIH
ERH XLI RI[ ZEPYI EHHIH XS XLI GEGLI EPSRKWMHI XLI RI[
X 32  8LI GEWI X 32 X 32 GERRSX SGGYV
-R EHHMXMSR XLI GEGLI HEXE WXVYGXYVI RIIHW XS WYTSR WIZIVEP XLVIEHW EW E FEGOKVSYRH TVSGIWW

8LMW MW

FEWIH SR FYGOIXPIZIP QYPXMVIEHIV PSGOW MQTPIQIRXEXMSRSJ XLI GSRGYVVIRX LEWL QET YWIH JSV GEGLI PMIW MR XLI@GLEWLQET GVEXI  



1EGVS HEXE WXVYGXYVI

9 YMH
7XVMRK(78
0MWXL1 EGVS-RTYXM-RWXERX
7XVMRK-RTYX
8 ]TI-H [LIVI 8 ]TI-H MW E HEXEI\TIGXIH X]TI SJ XLI ZEVMEFPI XLEXHIJEYPX

3TXMSRL8 M

SJ X]TI 0MWX 1 EGVS-RTYX  -R XLI GYVVIRX MQTPIQIRXEXMSR XLI PMWX SJ MRTYXW MW MRJIVVIH ERH VIGSQTYXIH IZIV]XMQI XLI QEGVS W MRRIV (78 MW YTHEXIH XLI MRRIV (78 W
YREXXEGLIH MRTYX WPSXW EVI GSRWMHIVIH XS FI XLI [LSPI

1EGVS 7YTTSVX JSV GEOI(IWMKR HIGMWMSRW

8LI VITVIWIRXEXMSR SJ E QEGVS MW E GLYRO SJ QIQSV]VYRXMQI MW GSQTSWIH SJ E 9- XLVIEH XLEX VIRHIVW XLI YWIVMRXIVJEGI ERH LERHPIW XLI YWIV W MRTYXW ERH WIZIVEP GSQTYXEXMSR XLVIEHW XLEX GSQTYXI XLI RSHI IHMXSV W SYXTYXW%W E VIWYPX XLI 9- XLVIEH VIUYMVIW VIEH EGGIWW XS HMWTPE]
XLI QEGVS SR XLI RSHI WGVIIR FYX VIUYMVIW EW [IPP [VMXIEGGIWW XS YTHEXI XLI QEGVS SR YWIV MRTYX 8LI GSQTYXEXMSR XLVIEH VIUYMVIW VIEH EGGIWW XS VIXVMIZI XLI QEGVS W(78 ERH VYR GSQTYXEXMSR %W E VIWYPX MX MW GPIEV XLEXE QEGVS QYWX FI WLEVIH FIX[IIR XLVIEHW FILMRH E VIEH
[VMXI PSGO ;I GEPP 1 EGVS,ERHPI [LMGL WLS[IH YT MR
8EFPI   ER EXSQMGEPP] VIJIVIRGIGSYRXIH WLEVIH TSMRXIVXS E VIEH[VMXI PSGO XS XLI EGXYEP GLYRO SJ QIQSV] SJ E
1 EGVSWIGXMSR

8EFPI 1 EGVSMH
REQI
HWX
MRTYXWYTHEXIH SR
1 EGVS-RTYX
REQI
WPSX
X]TI MH(EXE WXVYGXYVIW JSV QEGVS WYTTSVXHEXE WXVYGXYVIW ER @YTHEXIH SR -RWXERX MW ETTIRHIH
 6IZEQTMRK XLI GSQTYXEXMSR PSKMG
8LI EPKSVMXLQ YWIH JSV XLI [LSPI KVETL MW XLI WEQIEW XLI SRI I\TPEMRIH MR WIGXMSR  SRP] XLI ETTVSEGL
2 XLI @YTHEXIH@YTHEXIH SR ZEPYI MW ETTIRHIH MR FSPH MR XLI JSPPS[MRKIUYEXMSR 2YTHEXIH SR&'2XYTHEXIH SR&'&'& QEGVS W YTHEXIH SR '&'&'YTHEXIH SR (&'&'YTHEXIH SR 2 ! QE\ &

'
&
'

&
'
&
'
YTHEXIH SR (M
&
'
&
'

&
'

$
%
YTHEXIH SR (R

8EFPI  -X MRGPYHIW E 99-( ZIVWMSR  VERHSQP] KIR [LMGL MW YRMUYIHIITGSTMIH ERH WIRX XS XLI GSQTYXEXMSR XLVIEH XS IRWYVIXLEX XLI QEGVS W WXEXI HSIW RSX GLERKI HYVMRK GSQTYXE

REQI 8LI 99-( MW KIRIVEXIH SR GVIEXMSR SJ E RI[ QEGVSRIWW IZIR EJXIV XLI QEGVS MW WLEVIH EQSRK YWIVW 2I\X E
QEGVS GSRXEMRW E LYQERVIEHEFPI REQI YWIH JSV HMWTPE]ERH ER IQFIHHIH (78 XLEX HIWGVMFIW XLI FILEZMSV SJ XLIQEGVS -R EHHMXMSR [LMPI XLI SYXTYXW SJ E QEGVS EVIHIXIVQMRIH F] XLI SYXTYX RSHIW SJ XLI IQFIHHIH (78MX MW RIGIWWEV] XS OIIT XLI PMWX SJ MRTYXW JSV XLI QEGVS122

MRTYX WPSXW MW GSTMIH ERH VI[MVIH XS XLI GSVVIWTSRHMRK MR
TYX WPSXW MR XLI QEGVS W (78 8LIR XLI QEGVS W (78 MW
IZEPYEXIH ERH XLI QEGVS W SYXTYX ZEPYIW EVI GEPGYPEXIH
 1EGVS YWIV MRXIVJEGI8LI QEGVS IHMXMRK YWIV MRXIVJEGI MW MRXIKVEXIH MRXS*MKYVI QEGVS IHMXSV 8LI QEGVS REQIH @7II 1ERKE MR XLMWIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

3YXTYXRSHI1EGVS2SHI

1EGVS)HMXSV3YXTYXRSHI-RHITIRHIRXMQTSVXI\TSVXSJQEGVSW

1EGVS-RTYX
*-87 PI

1EGVS3YXTYX
;EZIJSVQ
1EGVS3YXTYX1EGVS-RTYX*VEQI-RHI\*MK 

(JVEQI

7MQTPI I\EQTPI SJ E QEGVS XLEX STIRW ER MQEKI JVSQ XLI 1E2+% HEXEWIX 

VIWIRXMRK E JVEQI RYQFIV 8LI QEGVS XLIR SYXTYXW XLI[EZIJSVQ ERH XLI MQEKI HEXE EX XLI TVSZMHIH JVEQI 8LMWMW XLI VIWYPX XLEX GER FI WIIR MR SYXTYX [MRHS[W  ERH% RI[ QEGVS QE] FI GVIEXIH F] E VMKLX GPMGO SR XLIKVETL IHMXSV ERH XLIR WIPIGXMRK XLI @'VIEXI RI[ QEGVSSTXMSR 3R GPMGOMRK ER IQTX] QEGVS [MPP TST YT 3RGI
E QEGVS LEW FIIR GVIEXIH MX GER FI VIYWIH ERH EHHIH
EW QER] XMQI EW SRI [MWLIW MRXS XLI RSHI IHMXSV &]HSYFPIGPMGOMRK SR E QEGVS XLMW QEGVS W IHMXSV MW STIRIH
MJ MX [EW RSX ERH JSGYWIH 8LI QEGVS IHMXSV MW QSWXP]I\TSVX SJ QEGVSW YWMRK E XEMPSVQEHI HEXE JSVQEX -R
WSQI FEWMG WERMX] GLIGOW XS TVIZIRX IRHPIWW PSSTW JVSQSGGYVVMRK *SV I\EQTPI MJ E QEGVS MW EHHIH [MXLMR MXWIPJYWIV JVSQ TVSGIIHMRK ERH WLSSX XLIQWIPZIW MR XLI JSSXHIJEYPXZEPYI)ZEPYEXMSRRMRK (IFMER  [MXL ER -RXIP 6 'SVI

81

M9 '49 $MW WYTTSVXIH SR EPP QEMRWXVIEQ STIVEXMRK W]WXIQW 0MRY\QEG37 ERH ;MRHS[W   (YVMRK IZEPYEXMSR QMHWM^I
EWXVSRSQMGEP HEXEWIXW E JI[ LYRHVIHW SJ QIKEF]XIW WYGLWYVZI] [IVI PSEHIH ;I [MPP GSRWMHIV XLI I\XVEGXMSR SJIUYMZEPIRX [MHXL EW ER I\EQTPI
123



'SQTEVMWSR [MXL I\MWXMRK KIRIVEP W]WXIQW-X WLSYPH FI RSXIH XLEX XLI EFMPMX] XS FVMRK EFSYX WLEVIEFMPMX] ZME QEGVSW MW RSX RI[ ERH QSWX GSRZIRXMSREPQSHYPEV ZMWYEPM^EXMSR IRZMVSRQIRXW WYGL EW XLSWI GMXIH
MR WIGXMSR  EPVIEH] WYTTSVX QEGVSPMOI JIEXYVIW ,S[MRTYXW RSX SR XLI ZMWYEPM^EXMSR QIXLSH EW EPVIEH] I\OMRH SJ QSHYPEV ZMWYEPM^EXMSR IRZMVSRQIRX XLEX EPPS[W XS

YWIH MR EWXVSRSQ]
'SQTEVMWSR [MXL I\MWXMRK EWXVSRSQMGEPXSSPMRK

8LI PMWXMRK SR *MK  WLS[W E GSQQIRXIH &EWL-6%*
WGVMTX XLEX GSQTYXIW XLI IUYMZEPIRX [MHXL SJ E HEXEGYFIYWMRK XLI WEQI QIXLSH EW XLI RSHI IHMXSV SYXPMRIH MR*MK  8LI VIEHIV QE] VIJIV XS 'EVVSPP W FSSO JSV TVI
 -R *MK  XLI@STIR[MHXL MR XLI VMKLXQSWX SYXTYX RSHI 3YXTYX   8LI
VIWYPX SJ IEGL SYXTYX RSHI MW ZMWYEPM^IH MR E GSVVIWTSRHMRK SYXTYX [MRHS[ 'SRXMRYYQ IQMWWMSR MW GSQTYXIH F]
RSHI  SR XLI PIJX WMHI ERH F] RSHI  SR XLI VMKLX
WMHI 8LI EZIVEKI SJ XLI IQMWWMSR PMRI MW GSQTYXIH F]
RSHI  -RXIVQIHMEV] VIWYPXW EVI ZMWYEPM^IH ZME SYXTYXRSHIW  ERH %W XLI VIEHIV QE] MRJIV JVSQ XLI GSHI MR XLI *MK  W

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
 7YFHEXEGYFI JSV IEGL FERH

MR *MK 

HYGMRK GSKRMXMZI PSEH [LMPI XLI GSQTYXMRK WTIIH MW RSXRSHIW   ERH 

E[O&)+-2_JSV M !  M M _TVMRXJ QERKE0-2'9&)JMXW?A? HA@R M a
a " JMPISJJPMWX
E[O &)+-2_JSV M !  M M _TVMRXJ QERKE0-2'9&)JMXW?A? HA@R M a
a " JMPISRPMWX
E[O &)+-2_JSV M !  M
 M _TVMRXJ QERKE0-2'9&)JMXW?A? HA@R M a
a " JMPISJJPMWX

MQTEGXIH JSV E  1& MRTYX HEXEWIX 3RP] XLI MRTYXHEXEGYFI ERH XLI GSRWXERX TEVEQIXIVW XLEX EVI VIPIZERXMR GSQTYXMRK XLI IUYMZEPIRX [MHXL EVI I\TSWIH F] XLIQEGVS 8LI GSQTYXIH SYXTYX MW WXVMGXP] MHIRXMGEP [LIXLIV 8LIR GSQTYXI EZIVEKI SR IEGL FERHMQGSQF $JMPISJJPMWX SJJEZIVEKIJMXW GSQFMRI!EZIVEKI
MQGSQF $JMPISRPMWX SREZIVEKIJMXW GSQFMRI!EZIVEKI
MQGSQF $JMPISJJPMWX SJJEZIVEKIJMXW GSQFMRI!EZIVEKI

E QEGVS MW YWIH SV RSX 8LVSYKL XLI WLEVMRK ERH XLI I\
TSVX SJ QEGVSW WYGL EW XLMW SRI KMZIR EW I\EQTPI GSQQSRTEVEQIXIVM^IH EREP]XMGW TSWWMFMPMXMIW EVI SRP] E JI[ GPMGOW

 'SQFMRI SJJFERH MQEKIW MR *MK  RSHIW  ERH 
IGLS  " WGEPISJJHEX
IGLS  "" WGEPISJJHEXMQGSQF SJJEZIVEKIJMXWSJJEZIVEKIJMXW SJJEZIVEKIJMXW @GSQFMRI!EZIVEKI [IMKLX!$WGEPISJJHEXXVEXMRK YWIV NYHMGMSYWP] HIWMKRIH E LMIVEVGL] SJ QEGVSE[E]

GEPPW 1EGVSW EPPS[ XS LMHI HIXEMPW XS GSRGIRXVEXI SR XLIPSSOERHJIIP ERH XLI JIEXYVIW SJ MQTSVXERGI JSV XLI EW 1EOI IUYMZEPIRX[MHXL QET MR *MK  RSHI 
MQEVMXL SJJEZIVEKIJMXW  SREZIVEKIJMXW SJJSREZIVEKIJMXW
MQEVMXL SJJSREZIVEKIJMXW   JPY\JMXW
MQEVMXL JPY\JMXW  SJJEZIVEKIJMXW IUYMZEPIRX[MHXLJMXW*MK XVSRSQIVW -R JEGX EWXVSRSQIVW`SYV YWIVW`PIX YW ORS[XLEX XLI] I\TIGX E W]WXIQ XLEX GER KMZI MRWXERX JIIHFEGO)\XVEGXMRK IUYMZEPIRX [MHXL [MXL &EWL-6%*

8LMW MW IWTIGMEPP] MQTSVXERX JSV RSZMGI YWIVW ;MXL XLIWLEVMRK SJ QEGVSW XLIVI MW RS XLISVIXMGEP PMQMX XS XLIMRK HIHMGEXIH ZMI[IVW WYGL EW (7   ;MXL XLI I\EQTPISJ IUYMZEPIRX [MHXLW QER] GSRWXERXW QYWX FI TVIGMWIP]
EHNYWXIH )ZIV] XMQI E GSRWXERX MW GLERKIH EPP XLI RI\X
GEWGEHMRK STIVEXMSRW QYWX FI VIVYR 2SX SRP] XLMW TVSGIWW MW XMQIGSRWYQMRK ERH IVVSVTVSRI`IRPEVKMRK ER EP
VIEH] XSS PSRK JIIHFEGO PSST` FYX TVSZIRERGI MW EW [IPP

LIVI WLMRIW F] MXW JEWX JIIHFEGO PSST PIWW XLER E WIGSRH
MW RIIHIH XS VIJVIWL XLI ZMWYEPM^EXMSRW MR ER SYXTYX [MRHS[ EJXIV E GLERKI MR XLI RSHI IHMXSV ERH EYXSQEXMGQEREKIQIRX SJ TVSZIRERGI 'SQTYXMRK WTIIH MW IRSYKLXS VIWTSRWMZIP] ERH MRXIVEGXMZIP] KMZI JIIHFEGO XS XLI EW-X WLSYPH JYVXLIVQSVI FI RSXIH XLEX EPP ZEPYIW XLEX GER
FI WIPIGXIH ERH QERMTYPEXIH MRWMHI SYXTYX [MRHS[W EW
WLS[R MR *MK  EVI VIHMVIGXIH ERH FSYRH XS E ZEPYI RSHI[LMGL XLIR GER FI VIYWIH [MXLMR XLI RSHI IHMXSV 8LI
FMRHMRK MW FMHMVIGXMSREP IHMXMRK XLI ZEPYI SJ XLI RSHIJVSQ XLI RSHI MRXIVJEGI [MPP YTHEXI XLI VITVIWIRXEXMSR SJXLI ZEPYI MR XLI SYXTYX [MRHS[ EW [IPP 8LMW EPPS[W XLIIXIVW [LMPI HIWMKRMRK ER EREP]XMGW TMTIPMRI 'PYXXIVJVII
EWXVSRSQIVMRXLIPSST MW XLIR EGLMIZIH%R MQTPIQIRXEXMSR JSV 4VSZIRERGI(1%W [I GSRWMHIV XLI JYPP PMWX SJ EPP YWI GEWIW XLEXGIWWJYPP] EHHVIWWIW IEGL SJ XLIQ LIRGI [I GER GSRGPYHITSMRX VSYRHMRK IVVSVW VIWYPXW SFXEMRIH [MXL -6%* ERH:MVXYEP 3FWIVZEXSV] %PPMERGI [LSWI VIUYMVIQIRXW EVIPMWXIH FIPS[)UYMZEPIRX [MHXL [MXL E QEGVS

8VEGIEFMPMX] SJ TVSHYGXW @8VEGO XLI PMRIEKI SJ E TVSH
XSSPW 8LMW WIGXMSR KMZIW E GSRGVIXI I\EQTPI SJ QEGVSYWEKI JSV IUYMZEPIRX [MHXLW *MKYVI  WLS[W ER I\EQ
TPI SJ E QEGVS MQTPIQIRXMRK XLI GSQTYXEXMSR SJ IUYMZEPIRX [MHXLW XLI WEQI TVSGIWWMRK EW XLI SRI HSRI MR*MK  8LI QEGVS IRGETWYPEXIW EPP XLI PSKMG MQTPIQIRXIH
EW WLS[R MR XLI RSHI IHMXSV MR *MK  SRP] I\TSWMRK XLIVIPIZERX GSRWXERXW XLEX XLI EWXVSRSQIVW EVI I\TIGXIH XSKVEHYEPP] EHNYWX YRXMP XLI] KIX E WEXMWJEGXSV] SYXGSQI
8LI EHZERXEKI SJ YWMRK QEGVSW EVI GPIEV ZMWYEP GPYXXIVSR XLI SVMKMREP RSHI MRXIVJEGI MW [MHIP] VIHYGIH XLYW VI
124

HMVIGXIH EG]GPMG KVETL QIIXW XLMW YWI GEWI%GORS[PIHKQIRX ERH GSRXEGX MRJSVQEXMSR @*MRH XLITISTPI MRZSPZIH MR XLI TVSHYGXMSR SJ E HEXEWIX XLI
TISTPISVKERM^EXMSRWMRWXMXYXIW XLEX SRI QE] [ERX XSEGORS[PIHKI SV GER FI EWOIH JSV QSVI MRJSVQEXMSRIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

1EGVS2SHI
3YXTYXRSHI)\TPEREXMSRERHQIXEHEXEEFSYXWIPIGXIHRSHILIVIXLIIUYMZEPIRX[MHXLQEGVSRSHI*MK 

5YEPMX] ERH VIPMEFMPMX] EWWIWWQIRX @%WWIWW XLI UYEPMX] ERH VIPMEFMPMX] SJ ER SFWIVZEXMSR TVSHYGXMSR WXIT
SV HEXEWIX 8LI ZIVWMSR ERH XLI YRMUYI 99-(W SJ

SV QEGVSW MW E TPERRIH JIEXYVI 9WMRK 99-(W XS MHIRXMJ]

HEXEWIX EVI PSKKIH [LMGL EPPS[W XS GLIGO UYEPMX] ERH'SRGPYWMSR

8LMW TETIV MW E WYFWIUYIRX 

VIPMEFMPMX] EX ER] XMQI MR XLI JYXYVIZMWYEP TVSKVEQQMRK IRZMVSRQIRX XLEX TVSZMHIW JEWXPSKKIH ERH E HIXEMPIH WXEGO XVEGI MW WLS[R XS XLYWERH VIWTSRWMZI QEGVS WYTTSVX MR XLI EWXVSTL]WMGEPHSQEMR 'SPPEFSVEXMSR IEWI SJ YWI VIWTSRWMZIRIWWMRGVIQIRXEP MQTVSZIQIRXW ERH TVSZIRERGI EVI XEOIR ZIV]

7IEVGL MR WXVYGXYVIH TVSZIRERGI QIXEHEXE @9WI
TVSZIRERGI GVMXIVME XS PSGEXI HEXEWIXW JSV[EVH;IF%WWIQFP] EWXVSRSQMGEP EREP]WMW ERH EGGIWW XS HEXEGSYPH FI QEHI EGGIWWMFPI XS E FVSEHIV EYHMIRGI -RHIIH

TVSGIWWMRK WXIT SV HIVMZIH JVSQ HEXE [LMGL [IVI XEOIRTVSJIWWMSREPWTSWWMFPI IVVSV WSYVGIW MR XLI KIRIVEXMSR SJ E TVSHYGXF] E KMZIR JEGMPMX] -X MW XLISVIXMGEPP] TSWWMFPI XS GPEWXLI I\EGX RSHI IHMXSV XLEX KIRIVEXIH XLIQ EW E WIVMEPM^IH RSHI IHMXSV MW MRGPYHIH MR EPP KIRIVEXIH *-87


%GORS[PIHKQIRX
8LMW [SVO MW WYTTSVXIH F] .747 /%/)2,- +VERX
2YQFIVW / ERH ,



6IJIVIRGIW(MWGYWWMSR ERH *YXYVI ;SVO

/ %OM]EQE % %PFIVHM ; %PIJ / %WEHE 6 %^YPE] %/@*MVWX 1 )ZIRX ,SVM^SR 8IPIWGSTI 6IWYPXW - 8LI 7LEHS[SJ XLI 7YTIVQEWWMZI &PEGO ,SPI 8LI %WXVSTL]WMGEP .SYVREP
0IXXIVW :SP  2S  T 0  

ERH WLEVMRK XLIQ [MXL JIPPS[ EWXVSRSQIVW 1EGVS WYTTSVX [EW E LMKLP] E[EMXIH JIEXYVI XLEX SYV HS^IR SJ YWIVW[IVI IEKIV XS YXMPM^I -RHIIH EWXVSRSQIVW I\TIGX XS YWIXYVROI]PMOI WSJX[EVI [MXL XLI JIEXYVIW XLI] RIIH EPVIEH]
MRGPYHIH

WIZIVEP KMKEF]XIW  7QEVXIV TVSFEFMPMWXMG GEGLI KEVFEKI
GSPPIGXMRK VIWSYVGIW XLEX EVI ZIV] YRPMOIP] XS FI UYIVMIH[SYPH FI E LYKI MQTVSZIQIRX &IWMHIW SR XLI 9- WMHIFIMRK EFPI XS KVSYT WIZIVEP RSHIW MRXS E WMRKPI QEGVSJVSQ XLI RSHI IHMXSV WGVIIR [SYPH FI I\XVIQIP] GSRZIRMIRX ERH MW E HIWMVEFPI JIEXYVI *MREPP] LEZMRK E TYFPMG
125



8 3OE 7 8WYNMQSXS = -[EXE 1 2SQYVE 7 8EOIOE[E@1MPPMQIXVI;EZI )QMWWMSR JVSQ ER -RXIVQIHMEXI1EWW &PEGO,SPI 'ERHMHEXI MR XLI 1MPO] ;E] 2EXYVI %WXVSRSQ] :SP 
2S  TT _  - *YNMWLMVS 2 7E[EHE 1 2EOE]EQE ,= ;Y / ;EXER
EFI 7 8EOELEWLM 1 9IQYVE @8MQI8YFIW :MWYEP )\TPSVEXMSR SJ 3FWIVZIH &PE^EV (EXEWIXW .SYVREP SJ 4L]WMGW 'SRJIVIRGI 7IVMIW :SP  2S  T   ( ' ;IPPW ) ; +VIMWIR @*-87 % *PI\MFPI -QEKI 8VERWTSVX 7]WXIQ -QEKI 4VSGIWWMRK MR %WXVSRSQ] T   



1

3

&SYWWINVE

/

1EXWYFE]EWLM=

8EOIWLMQE

4PYKKEFPI :MWYEP 4VSKVEQQMRK )RZMVSRQIRX [MXL 5YMGO*IIHFEGO 0SST 8YRIH JSV 1YPXM7TIGXVEP %WXVSTL]WMGEP'SRJIVIRGI 7GM:MW  TT _  

1 3 &SYWWINVE 7 8EOIOE[E 6 9GLMOM / 1EXWYFE]EWLM
KVEQQMRK )RZMVSRQIRX [MXL 5YMGO *IIHFEGO 0SST 8YRIH JSVIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
1YPXM7TIGXVEP %WXVSTL]WMGEP 3FWIVZEXMSRW 4VSG SJ %WXVSRSQMGEP (EXE %REP]WMW 7SJX[EVI ERH 7]WXIQW <<:--- :SP
T_  

1 3 &SYWWINVE 6 9GLMOM = 8EOIWLMQE / 1EXWYFE]EWLM
QMRK )RZMVSRQIRX )REFPMRK )RHXS)RH 4VSZIRERGI 1EREKIQIRX JSV XLI %REP]WMW SJ %WXVSRSQMGEP (EXEWIXW :MWYEP -RJSVQEXMGW :SP  2S  TT _  + 'EQIVSR @7TIGMEP *SGYW 1SHYPEV :MWYEPM^EXMSR )RZMVSRQIRXW 1:)W  %'1 'SQTYXIV +VETLMGW :SP  2S  TT_  

QMRK )RZMVSRQIRX JSV 'SQTYXEXMSREP 7XIIVMRK 4VSG SJ XLI %'1-))) 'SRJIVIRGI SR 7YTIVGSQTYXMRK T   ,. &YRKEVX^ % *VERO * 1IMIV @(IWMKR ERH -QTPIQIRXEXMSR SJ E 1SHYPEV )RZMVSRQIRX JSV 'SYTPIH 7]WXIQW 4VITVMRX
7*&  ; .S]I ) 1ERHIP @2I[ *IEXYVIW SJ 7%3-QEKI (7 %W
XVSRSQMGEP (EXE %REP]WMW 7SJX[EVI ERH 7]WXIQW <-- :SP T   & 6 /IRX @:MWYEPM^MRK %WXVSRSQMGEP (EXE [MXL &PIRHIV

7 4IVOMRW . 5YIWXMEY\ 7 *MRRMWW 6 8]PIV 7 &P]XL 1 1
/YXXIP @7GEPEFPI (IWOXST :MWYEPMWEXMSR SJ :IV] 0EVKI 6EHMS%WXVSRSQ] (EXE 'YFIW 2I[ %WXVSRSQ] :SP  TT _ % , ,EWWER ' . *PYOI ( + &EVRIW @-RXIVEGXMZI :MWYEP
M^EXMSR SJ XLI 0EVKIWX 6EHMSEWXVSRSQ] 'YFIW 2I[ %WXVSR
SQ] :SP  2S  TT _  ( 8SH] @8LI -6%* (EXE 6IHYGXMSR ERH %REP]WMW 7]WXIQ -RWXVYQIRXEXMSR MR %WXVSRSQ] :- :SP  TT _ 

 2S  T   
MGW 7]WXIQ %WXVSRSQMGEP (EXE %REP]WMW 7SJX[EVI ERH 7]WXIQW < :SP  T   ) &VE] 8 %PHGVSJX 1 (EZMW % +MRWFYVK % 1 4VMGI;LIPER ; ) /IV^IRHSVJ IX EP @%WXVST] % 'SQQYRMX]
4]XLSR 4EGOEKI JSV %WXVSRSQ] %WXVSRSQ]%WXVSTL]WMGW:SP  T %  
KERW 8 7 =SS @2-,27* :MWYEPM^EXMSR 6IWIEVGL 'LEPPIRKIW6ITSVX  

6IGIMZIH %TVMP  
6IZMWIH %YKYWX  + < 6MXXIV . 2 ;MPWSR . 0 (EZMHWSR @-QEKI %PKIFVE %R
3ZIVZMI[ 'SQTYXIV :MWMSR +VETLMGW ERH -QEKI 4VSGIWWMRK:SP  2S  TT _  

GLEWLQETLXXTWHSGWVWGLEWLQETGLEWLQET  

1 7IVZMPPEX / 6MIFI ' &SMWWSR * &SRREVIP % +EPOMR
1 0SY]W 1 2YPPQIMIV 2 6IREYPX8MREGGM 1 7ERKYMPPSR 3 7XVIMGLIV
@-:3% 4VSZIRERGI (EXE 1SHIP:IVWMSR  -:3% 4VSTSWIH 6IGSQQIRHEXMSR 
LXXT[[[MZSERIXHSGYQIRXW4VSZIRERGI(1
4VSZIRERGI(1THJ  

4 0IEGL 1 1IEPPMRK 6 7EP^ @% 9RMZIVWEPP] 9RMUYI -(IR/ &YRH] IX EP @3ZIVZMI[ SJ XLI 7(77-: 1E2+% 7YVZI]
1ETTMRK 2IEVF] +EPE\MIW EX %TEGLI 4SMRX 3FWIVZEXSV] 8LI%WXVSTL]WMGEP .SYVREP :SP  2S  T   & 'EVVSPP ( 3WXPMI %R -RXVSHYGXMSR XS 1SHIVR %WXVSTL]WMGW 4IEVWSR -RXIVREXMSREP )HMXMSR 4IEVWSR %HHMWSR;IWPI]  [[[VJGIHMXSVSVKVJGTHJVJGVJGX\XTHJ  

126

1EPMO 3PMZMIV &3977).6%,I KVEHYEXIH JVSQ )GSPI 'IRXVEPI HI 2ERXIW
E *VIRGL IRKMRIIVMRK WGLSSP XLIR GSQTPIXIH
E HSYFPI 1EWXIV W (IKVII [MXL /IMS 9RMZIVWMX] MR  -R  LI XLIR TVSGIIHIH XS
GSQTPIXI E 4L( MR GSQTYXIV WGMIRGI EX /IMS
9RMZIVWMX] ,I MW QEMRP] HVMZIR F] GYVMSWMX]
ERH [MPPMRKRIWW XS GSRXVMFYXI

6MOYS 9',-/-

1IQFIV,I VIGIMZIH E & ) MR  JVSQ /IMS 9RMZIVWMX] 'YVVIRXP] LI MW E QEWXIV WXYHIRX EX /IMS
9RMZIVWMX] ,MW VIWIEVGL MRXIVIWXW MRGPYHI EWXVSTL]WMGEP HEXE ZMWYEPM^EXMSR7LYR]E 8%/)/%;%,I VIGIMZIH E 4L( HIKVII JVSQ /IMS 9RMZIVWMX] MR  'YVVIRXP] LI MW E TVSNIGXZEXSV] 2EXMSREP %WXVSRSQMGEP 3FWIVZEXSV] SJ.ETER ,MW VIWIEVGL MRXIVIWXW MRGPYHI VEHMSEWXVSRSQ] ERH SFWIVZEXMSREP WXYHMIW SJ XLIKEPEGXMG GIRXIV/E^Y]E 1%879&%=%7,EX 3OE]EQE 3FWIVZEXSV] /]SXS 9RMZIVWMX]
,I VIGIMZIH LMW 4L( HIKVII 7GMIRGI MR JVSQ /]SXS 9RMZIVWMX] ,MW VIWIEVGL XSTMGWEVI EWXVSRSQMGEP MRWXVYQIRXEXMSR ERH KEPE\MIW=YVMOS 8%/)7,-1%

1IQFIV
7LI MW GYVVIRXP] E 4VSJIWWSV EX 7GLSSP SJ 1I
HME 7GMIRGI 8SO]S 9RMZIVWMX] SJ 8IGLRSPSK]
7LI VIGIMZIH LIV & 7G 1 7G ERH 4L (HIKVIIW MR 'SQTYXIV 7GMIRGIW JVSQ 3GLERSQM^Y 9RMZIVWMX] MR   ERH 
VIWTIGXMZIP] ,IV VIWIEVGL MRXIVIWXW MRGPYHIZSPYQI ZMWYEPM^EXMSR XSTPSK]FEWIH ZMWYEP
M^EXMSR ERH ZMWYEPM^EXMSR IRZMVSRQIRXW1EOSXS 9)196%,I MW ER EWWSGMEXI TVSJIWWSV EX ,MVSWLMQE %W
XVSTL]WMGEP 7GMIRGI 'IRXIV ,MVSWLMQE 9RMZIVWMX] ,I VIGIMZIH LMW 4L( HIKVII JVSQ/]SXS 9RMZIVWMX] MR  ,MW VIWIEVGL MRXIVIWXW MRGPYHI STXMGEP ERH RIEVMRJVEVIH EWXVSRSQ]

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）-WWIM *9.-7,-63,SRSVEV] 1IQFIV,I MW GYVVIRXP] E 4VSJIWWSV EX (ITEVXQIRX SJ-RJSVQEXMSR ERH 'SQTYXIV 7GMIRGI *EGYPX]SJ 7GMIRGI ERH 8IGLRSPSK] /IMS 9RMZIVWMX]
,I VIGIMZIH LMW & ) ERH 1 ) MR MRJSVQEXMSR WGMIRGIW ERH IPIGXVSRMGW MR  ERH 
FSXL JVSQ 9RMZIVWMX] SJ 8WYOYFE ERH LMW (SGXSV SJ 7GMIRGI MR MRJSVQEXMSR WGMIRGIW JVSQ XLI
9RMZIVWMX] SJ 8SO]S MR  &IJSVI NSMRMRK/IMS 9RMZIVWMX] MR  LI [SVOIH EW E JEGYPX] QIQFIV JSV XLI 9RMZIVWMX] SJ 8SO]S 9RMZIVWMX] SJ 8WYOYFE 3GLERSQM^Y 9RMZIVWMX]
ERH 8SLSOY 9RMZIVWMX] ,MW VIWIEVGL MRXIVIWXW MRGPYHI QSHIPMRK TEVEHMKQW ERH WLETIVITVIWIRXEXMSRW ETTPMIH ZMWYEPM^EXMSR HIWMKRERH PMJIG]GPI QEREKIQIRX ERH WQEVX EQFM
IRX QIHME [MXL QYPXMQSHEP HMWTPE]W ,I MW E
QIQFIV SJ 7GMIRGI 'SYRGMP SJ .ETER E JIPPS[SJ XLI .ETER *IHIVEXMSR SJ )RKMRIIVMRK 7S
GMIXMIW E LSRSVEV] QIQFIV SJ --)). ERH E
WIRMSV QIQFIV SJ -RJSVQEXMSR 4VSGIWWMRK 7S
GMIX] SJ .ETER

127

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
IIEEJ

Paper
An Eﬃcient Entropy Coding of Sparse Coeﬃcients Based on Sparsity
Adaptation and Atom Reordering for Image Compression
Ji WANG† , Yoshiyuki YASHIMA† (Member)† Graduate School of Information and Computer Science, Chiba Institute of Technology, Narashino, Japan<Summary> Sparse coding is a technique that represents an input signal as a linear combination of asmall number of atoms in the dictionary. When sparse coding is applied to image compression, it is necessary
to perform eﬃcient code assignment taking into account the statistical properties of weighting factors foreach atom. In this paper, we analyze in detail the position indices and magnitude of non-zero coeﬃcientsin a dictionary designed by K-SVD. Based on the analyzed results, we propose an eﬃcient entropy coding
method introducing sparsity adaptation and atom reordering. Simulation results show that the proposed
method can reduce the amount of generated bits by up to 6.2% compared to the conventional methods.
Keywords: image compression, sparse coding, entropy coding, K-SVD

1.

Introduction

With the recent increase in image resolution and increase in the amount of images by SNS, eﬃcient image
coding technology is indispensable in the ﬁeld of communication services and many home appliances. As image compression coding standards that are widespreadin the world, there are JPEG1) for still pictures andH.264/AVC2), H.265/HEVC3) for moving pictures. In addition, studies of the next-generation video coding standard VVC, including further improvement of coding eﬃciency, have already been started4). In most conventional
international standards, DCT (Discrete cosine transform)
is mainly used as transformation methods for eﬃciently
representing images. The DCT gives a good image representation for various images, but is not the best transformfor each individual image. In order to solve this problem,
sparse coding is attracting attention. Sparse coding candesign a dictionary by training the local features of theimage to be coded and can represent the target image
eﬃciently. In sparse coding, a dictionary can be designed
to minimize the reconstruction error when approximating
the original image using a linear sum of a predetermined
number of atoms. Since sparse coding allows most of theweight coeﬃcients to be zero and image representation
can be performed with very few nonzero weight coeﬃcients, its application to image compression can be expected5),6). When the sparse coding is applied to image
compression, the problem is how to encode the nonzero

128

coeﬃcients distributed in sparse. The statistical properties of sparse nonzero coeﬃcients have been analyzed
in some previous studies. In Reference 7), it has been
experimentally reported that the atom indices to indicate the occurrence position of nonzero coeﬃcient canbe approximated by uniform distribution, and nonzerocoeﬃcient levels can be approximated by Laplacian distribution. However, it is not clear how the atom indicesand the nonzero coeﬃcient levels in a block are related
to the number of nonzero coeﬃcients in the block. Also,a detailed analysis of the relationship between a nonzerocoeﬃcient level and its corresponding atom’s feature has
not been performed. For more eﬃcient entropy coding
design, it is necessary to analyze statistical properties ofnonzero coeﬃcients in more detail.
In this paper, we analyze the statistical properties ofnonzero coeﬃcients in detail from theoretical and experimental viewpoints, and propose an eﬃcient entropy coding method of sparse coeﬃcients based on the analysis.
Section 2 reviews the dictionary design method by KSVD and describes the application method to image coding using the designed dictionary. In addition, we describe typical conventional code assignment methods totransform coeﬃcients (weight coeﬃcients), and point out
their problems when applying to sparse coeﬃcients encoding. Section 3 analyzes the statistical properties ofthe sparse coeﬃcients in detail. First, we measure the occurrence probability of atom indices and coeﬃcient levels
for nonzero coeﬃcients, and clarify the distribution char-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

acteristic of zero run length between nonzero coeﬃcients.

to the samples to be represented. Therefore, it is desir-

Based on the distribution characteristics, we propose aable to co-optimize both dictionary D and coeﬃcient A.

context adaptive code assignment method to zero run

This problem can be formulated as

length and nonzero coeﬃcient level based on the number of nonzero coeﬃcients in the block. Next, we showthat the distribution characteristics of nonzero coeﬃcientlevels diﬀer depending on features of atoms, and clarify
that context adaptive coding to nonzero coeﬃcient levels
based on feature of atoms is eﬀective. Furthermore, weshow that the zero run length can be coded eﬃciently
by rearranging the atoms by their features. In Section
5, we summarize the results obtained and discuss further
works. The main focus of this paper is a research on symbol generation for eﬃcient entropy coding, rather thanactual code design method itself such as variable lengthcode tables or arithmetic coding. Therefore, the amountof generated information is discussed mainly based on entropy.

2.
2.1

Related Works

Review of K-SVD

In this section, we review the dictionary learning procedure based on K-SVD8). The set of sample vectors
for dictionary learning is indicated as the matrix Y,
and each column of Y corresponds to N sample vectors
yi (i = 1, 2, · · · , N ). For image representation, yi is often
set as a vector whose elements are the pixel values in thei-th small block obtained after dividing the image. Let
dk (k = 1, 2, · · · , K) be the k-th atom vector, and let dictionary D be a matrix in which these atoms are arranged
as columns. The dimension of these atoms equals thatof yi . We represent signal yi as a linear combination ofthese atoms as expressed by Eq.(1).
yi =

K


aik dk(1)

k=1

aik , which denotes the k-th element of vector ai , isthe representation coeﬃcient of the sample yi . Using
coeﬃcient matrix A = (a1 , a2 , · · · , aN ), Eq.(1) can be
written as
Y ≈ DA .

(2)

We consider an optimization problem with sparsity constraints that expresses input vector yi with as few atoms
as possible. Approximations with greater sparsity andsmaller error can generally be obtained by using a dictionary learned from samples having characteristics similar

129

min ||Y − DA||2F
D,A

subject to∀i, ||ai ||0 ≤ T0 . (3)Here, notation  · F stands for the Frobenius norm and
T0 is the sparsity constraint threshold.
K-SVD solves Eq.(3) by iterating two stages, sparse
coding stage and dictionary update stage. The former applies orthogonal matching pursuit (OMP)9)to determine
ai for each yi while ﬁxing D; and the latter updates D
together with the nonzero coeﬃcients of ai . The algorithmic description of K-SVD is detailed in Reference 7).
2.2 Image compression by multiclass K-SVD
A dictionary designed by K-SVD under the constraintthat the number of nonzero coeﬃcients falls to T0 or lesscan minimize the reconstruction error when the image
used to design the dictionary is represented by T0 or
fewer nonzero coeﬃcients. However, the characteristicsof K-SVD derived dictionaries are highly dependent on
the feature of the images used in training. A dictionary
trained for a speciﬁc image is optimum for that image,but not necessarily for other images. When applying KSVD to image coding, the decoder has to use the same
dictionary as the encoder, so the designed dictionary itself
must be encoded and transmitted to the decoder. However, the coding and transmission of dictionaries for each
image every time incurs large overheads for informationtransmission, and is not practical from the viewpoint ofrate distortion performance.
To solve this problem, we consider the use of the multiple dictionary approach. That is, ﬁrst, the image isdivided into small blocks to calculate local features, thena set of blocks having similar feature is created as aclass, and ﬁnally, K-SVD is executed for each class so
as to design multiple dictionaries. Local features suchas DSIFT10), intraframe/interframe prediction residual
power in H.264/AVC and H.265/HEVC7),11) have beenutilized for classiﬁcation. A set of dictionaries designed
for each class is shared in advance by the encoder anddecoder, and these are adaptively switched when encoding. This eliminates the need to encode dictionaries, andmakes it possible to represent more images eﬃciently.
In this paper, we calculate the local features of theblock based on the idea of classiﬁcation shown in Reference 9). After classifying the blocks used for training
based on the local feature, a dictionary is designed forIIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）each class. DSURF is used instead of DSIFT in this study
for high speed processing. Figure 1 shows some examples of dictionaries designed by multiclass K-SVD. It is
important to note that the atoms in a dictionary designedby K-SVD are not necessarily arranged in frequency order
like DCT, and the atoms with diﬀerent properties appear
randomly.
In order to eﬀectively represent the actual image, eachdictionary should contain one DC atom, as has been conﬁrmed1),12). Therefore, one DC atom is included in theinitial dictionary for each class, and the DC atom is notchanged during K-SVD iterative processing. Also, allatoms other than DC are normalized so that the meanvalue is zero and the standard deviation is one.
A block diagram of the encoder and decoder using dictionaries designed by the multiclass K-SVD algorithm isshown in Fig. 2. All dictionaries are prestored in both
encoder and decoder. In the encoding process, an image
to be coded is divided into small blocks of the same sizeas the used in the training process. Then, OMP is performed for each target block ti under sparsity condition

(a)

(b)

(c)

(d)

Fig. 1 Examples of multiclass dictionaries designed by KSVD; (a) overcomplete DCT, (b, c, d) Dictionaries
designed by K-SVD. The upper left corner represents
the DC atom

T0 ; the squared errors ec (c = 1, 2, · · · , C) are calculated
as follows:
ec = ti − Dc xi 2F ,(4)then the class index c and sparse coeﬃcients xi that minimize squared error ec are determined. Quantized coeﬃcients Q(xi ) and class index c are encoded and transmitted.
In the decoding process, the dictionary is adaptively
selected block by block based on the decoded class index,
and pixel values in the block are reconstructed as the sumof atoms weighted by the decoded sparse coeﬃcients.
2.3

Fig. 2 Encoder and decoder conﬁguration

Entropy coding for transform coeﬃcients

In image coding, it is necessary to make symbols to
be coded into binary codes. This procedure is called entropy coding, and various kinds of variable length codes
based on the occurrence probability of symbols are utilized. By assigning fewer bits to encode more frequently
occurring symbols, the total amount of bits used to encode the all symbols can be reduced. The Huﬀman codeand the arithmetic code are typical ones.
Here, we review some code assignment techniques fortransform coeﬃcients. Discrete Cosine Transform (DCT)
is one of the most popular techniques used today in video
compression schemes. Figure 3 (a) shows the 8x8 arrayof atom images for the two dimensional DCT. DCT con-130(a) DCT atom(b) Scan order in
JPEG(c) Scan order in
H.265/HEVC

Fig. 3 DCT atom and scan orderverts a set of pixels in a block into the weighted sumof DCT atoms. The weighting factors are called DCT
coeﬃcients. Statistically, the magnitude of the DCT coeﬃcients for low frequency atoms are greater than thatfor high frequency atoms. Also, by quantizing the coef-IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）ﬁcients, many DCT coeﬃcients for high frequency atoms
will be zero.
This property is used to perform eﬃcient entropy coding by appropriately setting the scan order of DCT coefﬁcients. The order of zigzag scan in JPEG and MPEG-2
is shown in Fig. 3(b). The ﬁrst coeﬃcient of each blockobtained as a result of zigzag scan is called the DC coeﬃcient while the other coeﬃcients are called AC coefﬁcients. For AC coeﬃcients, a variable length code is
assigned for the pair of a nonzero coeﬃcient and its preceding zero run length1),13). An End-of-Block (EOB) at
the end of each block indicates the rest of the coeﬃcientsof the block are all zero, and it enables to represent longconsecutive zeros eﬀectively. In H.265/HEVC, quantized
DCT coeﬃcients are coded as follows. They are scanned
diagonally to form a 1D array as shown in Fig. 3(c). The
context adaptive binary arithmetic coder (CABAC) encodes the last position of nonzero coeﬃcients, a signiﬁcance map indicating the positions of nonzero coeﬃcients,
and the quantized coeﬃcient level values3),14).
In the case of complete DCT, the frequency characteristics of each atom are known, and they are regularly
arranged. The relative relationship between the characteristics of each atom and the magnitude of the transform coeﬃcient corresponding to each atom is also clariﬁed. Therefore, by setting the scan order as shown in
Fig. 3 based on these characteristics in advance, the number of occurred bits can be reduced eﬀectively. Also,
for the atoms based on complete DCT, the international
standard methods H.264/AVC and H.265/HEVC haveadopted a method of switching the code table for eachblock using the number of non-zero coeﬃcients as a context2),3). On the other hand, each atom of the overcomplete dictionary designed by K-SVD does not have regularfrequency characteristics like DCT. Thus, it has not beenclariﬁed what kind of atom has a large non-zero coeﬃcient. Also, it has not been clariﬁed how the coeﬃcientquantization level distribution changes with the numberof non-zero coeﬃcients in the block. Therefore, in order to perform entropy coding for sparse coeﬃcients aseﬃcient as the conventional method, we need to clarify
the statistical properties of the sparse coeﬃcients and toclarify how to reorder the sparse coeﬃcients based on theﬁndings.
Several entropy coding for sparse representations havealso been studied. In image coding using sparse representation, OMP is performed using a dictionary for eachblock to be coded, and at most T0 nonzero coeﬃcients are131

calculated. All other coeﬃcients are zero. For the entropy
coding of the sparse representation, the indices of atoms
corresponding nonzero coeﬃcients after quantization andthe nonzero coeﬃcients levels are encoded. In the conventional studies, it has been clariﬁed that the histogramof the atom s indices is approximated to a uniform distribution, the histogram curve for the quantized coeﬃcientlevels is approximated to the Laplacian distribution7),15).
In Reference 15), it is shown that the nonzero coeﬃcientin the case of sparse representation by overcomplete ICT
becomes Laplacian distribution. Based on these features,
a ﬁxed length code was assigned for the atom index coding, and Huﬀman code or a truncated unary code combined with an Exponential-Golomb code was employed to
encode the quantized coeﬃcient level7),15). On the other
hand, instead of assigning a code directly to an index, amethod of assigning a Huﬀman code to a zero run length(i.e. the number of consecutive zero coeﬃcients between
nonzero coeﬃcients) has also been studied 17).
However, in the conventional researches, the relationship between the atom indices corresponding the nonzerocoeﬃcients in a block and the number of nonzero coefﬁcients of the block has not been clariﬁed. Also, therelationship between the probability distribution of thenonzero coeﬃcient levels and the number of nonzero coeﬃcients of the block has not been clariﬁed. In addition,
the detailed analysis of the relationship between the magnitude of nonzero coeﬃcient level and the feature of thecorresponding atoms has not been conducted. Therefore,
there is room for improving the conventional code assignment procedure by using the number of nonzero coeﬃcients and feature of the atoms as a context. In the next
section, we analyze the statistical properties of nonzerocoeﬃcients in detail from theoretical and experimental
viewpoints for sparse representation of images, and we
propose an eﬃcient entropy coding scheme for sparse coeﬃcients.

3.

Statistical Feature Analysis of Sparse
Coeﬃcients

In this section, we analyze the statistical properties ofthe sparse coeﬃcients in detail for the entropy coding
scheme design. The analysis in Section 3 is carried out
theoretically and experimentally. A set of small blocks extracted from six types of images, “BQTerrace”, “BasketballDrive”, and “Cactus”, “ChristmasTree”, “Kimono1”
and “ParkScene” from the MPEG test sequence are used
for statistical analysis, where these images are also used

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Fig. 4 Sparse coeﬃcients to be coded

as test data for the experiments in Section 4. The sparse
coeﬃcients to be encoded can be illustrated as in Fig. 4.
√First, the image is divided into small blocks of L/2 ×√
L/2. Next, for each small block, OMP is performed on
the dictionary designed by K-SVD to obtain T0 sparse
coeﬃcients. The dimension of a dictionary is L. After
quantization, we obtain sparse coeﬃcients to be encoded
for each small block. Let k be the number of nonzerocoeﬃcients after quantization. Here, all DC coeﬃcientsare quantized by step one and they are always encoded.
Also, nonzero AC coeﬃcients are quantized by the quantization step QP. The number of nonzero AC coeﬃcients
after quantization is at most T0 − 1. Set the number ofblocks in which the number of nonzero coeﬃcients to be
coded becomes k among all blocks of the image as N (k).
N (1) means the number of the blocks represented by DC
coeﬃcients only. The total number of blocks in the whole T0
N (k), and the number of DC coimage, N , is N = k=1
eﬃcients in the image, NDC , is equal to N . In addition,
the number of nonzero AC coeﬃcients in the whole image,NnonzeroAC , is expressed by
NnonzeroAC =T0(k − 1)N (k) .

(5)

k=2

3.1 Syntax of sparse coeﬃcients coding
Figure 5 shows the sparse coeﬃcient coding syntax analyzed in this study. The information required for eachblock to be encoded are, class No.: a class number indicating which of dictionaries is used, k: the number ofnonzero coeﬃcients in the block, coefDC : a weighting factor for DC atom, and coefAC : weighting factors for AC
atoms. Also, the number of nonzero AC coeﬃcients is
k − 1, and it is necessary to encode atom indices andquantized coeﬃcient levels for each nonzero AC coeﬃcient. In this study, in order to perform code alloca-

132

Fig. 5 Bit stream structure for sparse coeﬃcientstion adaptively by the number of nonzero coeﬃcients foreach block, the number of nonzero coeﬃcients in a block,k, is encoded prior to the atom indices of the nonzerocoeﬃcients and the quantized coeﬃcients level. For thesyntax of AC coeﬃcients, the atom indices for nonzerocoeﬃcients and the nonzero quantized coeﬃcients levelare encoded. Regarding the atom indices, we will consider two kinds of methods: direct encoding of indicesand a method of using zero run length between indices ofnonzero coeﬃcients.
3.2

Nonzero coeﬃcients distribution and entropy
In the conventional research7),15), the atom indices fornonzero coeﬃcients after quantization and the nonzero
quantized coeﬃcient levels are coded independently, andany adaptation of code assignment depending on thenumber of sparse coeﬃcients in the block and the featureof the atoms has not been studied. In this subsection,
we ﬁrst unify the symbols of all blocks based on the conventional method and analyze the statistical propertiesof the nonzero coeﬃcients and the amount of generated
bits. In this study, the amount of generated information
is analyzed based on the entropy calculated from the occurrence probability of the symbols to be coded. The
total amount of generated bits for the whole image is expressed as
Bitall = Bitclass + Bitk + BitDC + BitAC .

(6)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

(a)

(b)

Fig. 6 Probability histograms of (a) position index and (b)magnitude of nonzero quantized AC coeﬃcients

Here, Bitclass , Bitk , BitDC and BitAC are the amount ofgenerated bits for class number, the number of nonzerocoeﬃcients, DC coeﬃcient, AC coeﬃcient, respectively.
The amount of each code bits can be calculated as follows.
First, a class number can be expressed as a ﬁxed-lengthcode of log2 C bits per block, where C is the number ofclasses. The amount of generated bits in the whole image,Bitclass , can be calculated as Bitclass = N × log2 C.
Next, to calculate the number of bits for the number ofnonzero coeﬃcients, it is necessary to consider the distribution of the occurrence probability p(k). p(k) changes
with the quantization step QP for the coeﬃcients. When
the QP becomes smaller, the occurrence probability oflarge k increases, and as the QP becomes coarser, the occurrence probability of small k increases. The amount ofbits for the number of nonzero coeﬃcients in the whole
image is calculated as Bitk = Ek × N , where Ek is theentropy of p(k) as shown following equation.
Ek = −T0

p(k) log2 p(k)

(7)

k=1

The amount of bits generated for the DC coeﬃcientis calculated as follows. Since the DC coeﬃcients reﬂect
the average value of the block, there is a high correlation
between the DC coeﬃcients of adjacent blocks. Therefore, DPCM is performed based on the diﬀerence withthe previous block. Since the probability distribution of
the diﬀerence signal is approximated as a Laplacian distribution centered at zero, the total amount of bits is
calculated as BitDC = EDC × N , where EDC is an entropy based on the occurrence probability of diﬀerential
DC values.
The amount of bits generated for nonzero AC coeﬃcient is calculated from the distribution of their atom
indices and coeﬃcient levels. Figure 6 (a) shows a histogram of atom indices for nonzero AC coeﬃcients measured when sparse coding is performed on the test imagesby setting T0 = 7 and QP= 16. From the results, the133

Fig. 7 Number of bits generated

occurrence probability of atom indices for nonzero coeﬃcient is almost uniform. Similar measurements were
performed for various combinations of T0 and QP, anda chi-square test was performed for each case. As a result, we could conﬁrm the uniformity of the probability distribution of atom indices as in the conventional
study7). When uniformity of the occurrence probability
distribution of atom indices can be assumed, log2 L bits
are needed per one atom index theoretically. Therefore,
the total amount of bits for atom indices in the whole image, BitI , is Bit I = log2 L × NnonzeroAC . Also, Fig. 6(b)shows the distribution of nonzero quantized AC coeﬃcient levels, which can be approximated by the Laplacian distribution centered on zero. Note that there isno zero coeﬃcient. Coarse quantization concentrates theoccurrence probability distribution to smaller levels andincreases the number of zero coeﬃcients. The amountof bits for nonzero coeﬃcient levels in the whole image,Bit L , is EL × NnonzeroAC , where EL is the entropy of thenonzero AC coeﬃcient levels. The total amount of bits
for nonzero AC coeﬃcient in the whole image, BitAC , iscalculated as the sum of Bit I and Bit L .
Figure 7 shows the amount of bits generated in thewhole image measured by changing QP. The coeﬃcientlevel becomes smaller when the coarse quantization stepis used, so the amount of bits for AC coeﬃcient levels
decreases. Similarly, when coarse quantization step isused, the number of nonzero quantized AC coeﬃcientsdecreases, so the amount of bits for atom indices decreases. Since the quantization step for DC coeﬃcientsis always one, BitDC is constant regardless of the quantization parameter QP for AC coeﬃcients. Bit k shows aslight increase or decrease because the distribution of thenumber of nonzero AC coeﬃcients changes depending on
the magnitude of QP. Bitclass is constant because it isdetermined only by the number of classes. From Fig. 7,
it is clear that reducing the amount of bits for expressing
the AC coeﬃcient is very signiﬁcant.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
k=2
k=3
k=4
k=5

1.5

0.5

k=2
k=3
k=4
k=5

0.4

Probability

Probability[ 10-2]

2.5

0.3
0.2
0.1

0.5
0

50

100

150

200

250

0
-10

-5

Position index of non-zero AC

(a)

0

5

10

LEVEL

(b)

Fig. 8 Probability histograms of (a) position index and (b)magnitude of nonzero quantized AC coeﬃcients, after
categorizing based on k

In order to reduce the amount of generated bits forthe nonzero AC coeﬃcients, it is possible to divide thenonzero AC coeﬃcients into multiple categories according to the number of nonzero coeﬃcients in the block andperform code allocation suitable for each category. Theoretically, if the symbols can be separated into multiple
categories so that their occurrence probability distributions are as diﬀerent each other as possible, the total codeamount can be reduced. Figure 8 shows the distributionof atom indices for nonzero AC coeﬃcients and the distribution of AC coeﬃcient levels, after categorizing basedon the number of nonzero coeﬃcients in the block. As
shown in Fig. 8, it is clear that the information symbols
separation by k has little eﬀect because the probability
distribution of the atom’s indices corresponding nonzerocoeﬃcients is almost same regardless of the value of k.
On the other hand, since the occurrence probability ofnonzero quantization level numbers show diﬀerent distributions depending on k, it is considered to be signiﬁcantto perform the symbol separation by k.
3.3

Sparsity adaptive sparse coeﬃcient coding

Another way to represent atom indices of nonzero coefﬁcients is to use the number of zero coeﬃcients (i.e. zerorun length) preceding nonzero coeﬃcients17). We analyze
the statistics of zero run length when L coeﬃcients are
divided by k nonzero coeﬃcients as shown in Fig.9. This
problem can be solved theoretically as a consequence ofthe broken stick problem18) which is an analysis problem
concerning the probability distribution of the length fora piece of sub segments when the line segment of length 1
is divided by n − 1 random points. The probability density function of the length r(0 ≤ r ≤ 1) of any divided
segments isg(r) = (n − 1)(1 − r)n−2 .

(8)The probability P (r0 ) that the length of each segment
becomes [r0 , r0 + ) is obtained by integration of equation (8) as

134

Fig. 9 The probability density function of r, the length ofany divided segments

Fig. 10 The theoretical probability distribution of zero run
lengthP (r0 ) =

 r0 +
r0

g(r)dr = [−(1 − r)n−1 ]rr00 +

= (1 − r0 )n−1 − (1 − (r0 + ))n−1(9)

Applying the above analysis to the distribution ofsparse coeﬃcients, the length of the original line segment
corresponds to the dimension L of a dictionary, and thelength of each divided segment corresponds to the zerorun length. Figure 10 shows the theoretical probability distribution of zero run length when the length ofthe original line segment is set to L = 256. The occurrence probability is found to be a distribution based on
an exponential function. In Reference 17), code designis performed by integrating the occurrence probabilities,
that is, without classiﬁcation by the number of nonzerocoeﬃcients. However, from Fig. 10, since the parametersof exponential function clearly diﬀer depending on thenumber of nonzero coeﬃcients in the block, it can be expected that more eﬃcient code assignment for zero run
length becomes possible by categorizing nonzero coeﬃcients by k. The entropy of the zero run length isErun (k) =L−1P (i/L) log2 P (i/L)

(10)

i=0

where P (i/L) = (1 − i/L)k − (1 − (i/L + 1/L))k . The
amount of bits to represent the atom indices in the whole
image is calculated as
Bit I =T0

k=2

Erun (k − 1)N (k)(k − 1)

(11)

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

(a)

(b)

(c)

(d)

(e)

Fig. 11 Power spectrum for atoms; (a) overcomplete DCT (Fig.1(a)), (b) K-SVD(Fig.1(b)), (c) K-SVD(Fig.1(c)),(d) K-SVD(Fig.1(d)), (e) complete DCT

3.4 Adaptive coding by atom features
It is known that the features of the atoms appearing in
the dictionary designed by K-SVD are strongly inﬂuenced
by the features of the training samples, and they are different from general atoms such as DCT. Figure 11 shows
the Fourier power spectrum of each atom for the four dictionaries shown in Fig. 1. The center of each spectral image corresponds to the DC component, and the longer the
distance from the center, the higher the frequency. For
comparison, the power spectrum for complete DCT wasadded as shown in Fig. 11(e). From Fig. 11, in the dictionary consisting of atoms with regular frequency arrangement such as DCT and overcomplete DCT (Fig. 11(a),(e)), each atom complements each other so as to cover
all frequency bands. On the other hand, the overcomplete dictionary designed by K-SVD (Fig. 11(b), (c), (d))
does not necessarily consist of atoms that cover all frequency bands. It can be seen that it is composed of
atoms that can express a speciﬁc frequency band in moredetail. When expressing images using a training-based
dictionary, there have been no studies investigating thedependency between the characteristics of the weighting
factors and the features of the atoms. If there is a correlation between some features of atom and weighting
factors, it is possible to reduce the amount of generated
bits by adaptively changing the code assignment to theweighting factors with the atom’s features as the context.
Therefore, we ﬁrst investigated the relationship between some features of atoms and the magnitude of theweighting factors. Let b(i, j) be an M × M atom in a dictionary designed by K-SVD. The following four features
are investigated as the features of each atom,• Fourier transform:
F1 (th) =



(a)

(b)

(c)

(d)

Fig. 12 Correlation between atom feature and magnitude ofnonzero coeﬃcients• Discrete cosine transform:
|u|+|v|≤th |B(u, v)|
2u,v |B(u, v)|

where B(u, v) is the Fourier power spectrum ofb(i, j), −M/2 ≤ u ≤ M/2, −M/2 ≤ v ≤ M/2.
135,

u ≤ M − 1, 0 ≤ v ≤ M − 1.
• Total variation:
F3 =


i

(|b(i+1, j)−b(i, j)|+|b(i, j +1)−b(i, j)|)j

• Number of strong edge:
F4 (th) =i(mH (i, j) + mV (i, j)),

j

where

mH (i, j) =mV (i, j) =,2

where C(u, v) is the DCT coeﬃcients of b(i, j), 0 ≤2(u+v)≤th |C(u, v)|
2u,v |C(u, v)|

F2 (th) =1, if |b(i + 1, j) − b(i, j)| > th0,otherwise
1, if |b(i, j + 1) − b(i, j)| > th0,otherwise

Figure 12 shows the correlation between each featureof atoms and the magnitude of the nonzero AC coeﬃcient. Figure 12 also shows the average and the standard

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a) Before reordering(b) Scan orderFig. 14 Probability distribution of level after reordering(c) Reordered by F1 (2)(e) Reordered by F2 (2)(d) Reordered by F1 (4)(a)k=2(b)k=3(c)k=4(d)k=5(f) Reordered by F2 (4)Fig. 15 Probability distribution of zero run before and after
reordering(g) Reordered by F3

coeﬃcients at the start of the scan, so the probability of
having a short zero run length becomes high. This resultsin more eﬃcient code assignment to zero run length.
Figure 14 shows the occurrence probability of nonzero(h) Reordered by F4 (10)Fig. 13 Examples of the atoms reordered by their features

deviation of the absolute value of nonzero AC coeﬃcients
generated for the atoms in each section after dividing thefeature quantity into 16 sections. In Fig. 12, the results
show the case where the parameter th for each feature
value is set so that the correlation coeﬃcient becomes
the highest. There is a signiﬁcant correlation between
these four feature values and the magnitude of nonzero
AC coeﬃcients. Therefore, if we adapt the code assignment to the nonzero AC coeﬃcient levels according to
the feature of their corresponding atoms, the amount ofgenerated bits can be reduced. Also, from the observation in Fig. 12, we can consider that more eﬃcient code
assignment for the length of zero runs is performed byreordering the atoms so that the coeﬃcients with large
absolute values are scanned ﬁrst. Figure 13 shows theexamples of the atoms reordered by their features. Because the reordering of atoms concentrates nonzero AC

136

AC coeﬃcient levels for the feature value of atom. Here,the feature value of atom utilized is F3 which showed thestrongest correlation from the measurement results shownin Fig. 12. After deﬁning p = int(16 × F3 /max(F3 )),we measure the probability distribution for each p. Thismeasurement is performed under the condition of T0 = 5
and QP = 32. It is clear that the probability distribution
is diﬀerent depending on the feature value of atoms. In
addition, Figure 15 shows the comparison between occurrence probability of the zero run length under originalorder and that after reordering the atoms using the feature value F3 . We can ﬁnd that the zero run length has
a distribution that concentrates on smaller values for all
k compared to before reordering the atoms. Therefore,
it was veriﬁed that the adaptive code assignment by considering the atom feature is very signiﬁcant for reducing
both the amount of nonzero AC coeﬃcient level and zero

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）
Table 1 Simulation conditions
Training data
1.2M 8 × 8 blocks from 18)
Feature for classiﬁer
DSURF
Number of classes C
16, 32, 64, 128
Initial dictionary
16 × 16 overcomplete DCT
T03, 5, 7(a) Dictionary
by K-SVD

designed(b) Complete DCT

Fig. 16 Atom feature F3 according to scan orderrun length.
Figure 16 (a) shows the measured feature value F3 foreach atom in the dictionary designed using K-SVD. The
blue line in Fig. 16 is the result by the conventional zigzag
scan order, and the red line is the result by scanning in ascending order of F3 . When arranged in the conventional
zigzag scan order, the feature value F3 ﬂuctuates drastically. As a result, the probability that a coeﬃcient with alarge magnitude and a coeﬃcient with a small magnitude
will randomly occur becomes higher, and code assignment to zero runs becomes ineﬃcient. If the coeﬃcientsare scanned in ascending order of F3 , the probability thatcoeﬃcients with large magnitude will be concentrated at
the beginning of the scan becomes higher, and eﬃcientcode assignment can be realized. On the other hand, themeasurement results for complete DCT under the same
conditions in Fig. 16(a) are shown in Fig. 16(b). We
can see that even if the proposed method is applied tocomplete DCT, the scan order is almost unchanged fromthe zigzag scan used in the conventional method, and theeﬀect of increasing the coding eﬃciency is small. The
reason why the scan order hardly changes even when
the proposed method is applied is that zigzag scan itself is already setting eﬀectively for complete DCT whose
atom features are already known. Note that the resultsin Fig. 16 was conﬁrmed to be the same when not only
the feature value F3 but also other feature values F1 , F2 ,and F4 are used.

4.

Experiments

4.1 Experimental conditions
In this section, based on the analysis in section 3, simulation experiments are performed under various conditions to verify the coding eﬃciency. The experimental
conditions are shown in Table 1. In order to design the
dictionary, a total of 1.2 million 8 × 8 blocks were137

extracted from the images of the ITE/ARIB HDTV test
materials database19) as training data, and they wereclassiﬁed by DSURF. The multiclass dictionaries was designed under the number of classes of 16, 32, 64 and 128.
In each class, a dictionary is designed by K-SVD withan overcomplete DCT of dimension 16 × 16 as the initialdictionary. The sparse constraint parameter T0 was set
to 3, 5, and 7.
If the number of samples used for training is too small,the image representation performance by the designed
dictionary will be degraded, and as a result meaningfulexperiments for this study will not be possible. If thenumber of samples used for training is large enough andvarious features of general images are well-balanced inthem, the dictionary created by learning will converge toa versatile optimal solution. In this study, the ITE test
image database was used for training. This is because it
is composed of images with various features targeted forcodec evaluation, etc., and it is considered that the features of images that are generally used can be captured
suﬃciently by using all these images for training. In theconventional studies, training is performed using samplesof tens of thousands of blocks ( for example, about 68000
blocks in Reference 7) and one hundred thousand blocks
in Reference 17) ). On the other hand, the number of 1.2
million blocks used in this study is suﬃciently large compared to the number of blocks used in the conventional
studies. So, it is considered that an appropriate dictionary is designed for entropy coding research, which is thefocus of this paper.
When encoding, ﬁrst, the image to be encoded is divided into 8 × 8 small blocks, and their class number
are determined by the k-means method according to the
DSURF feature of the small block. Next, using OMP and
the selected class dictionary, we obtain T0 sparse coeﬃcients that approximate the small block to be encoded.
In OMP, DC atom is always used. Therefore, the numberof AC coeﬃcients is T0 − 1. The obtained DC coeﬃcientsare quantized with quantization step 1 (i.e., rounded to
the nearest integer). On the other hand, AC coeﬃcientsare quantized with quantization step QP. In this experiment, we set QP = 16, 24, 32, 40. Under these pa-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

rameters, the eﬀectiveness of introducing zero runs, theeﬀectiveness of adaptation with the number of nonzerocoeﬃcients, and the eﬀectiveness of adaptation by feature of atoms feature are examined, in comparison withconventional entropy coding. Note that, in this research,
the image quality does not change in case the same QP
is used, so the eﬀectiveness can be veriﬁed only based on
the amount of generated bits. As shown in Section 3, theamount of generated information is calculated by the entropy based on the occurrence probability of the symbols
to be coded. In the original documents ( References 7)
and 17) ) of the conventional methods to be compared,
Huﬀman codes and Golomb-Rice codes are assigned to
the generated symbols. However, for the conventional
methods in this experiment, instead of actually assigninga code bit, the amount of information is calculated basedon the entropy of the generated symbol in order to makea fair comparison.
4.2

Experimental results and discussion

First, under the conditions ﬁxed at T0 = 5 and
QP=24, we measured the eﬀectiveness of k-adaptation,i.e. the adaptive encoding by the number of nonzero coeﬃcients. Figure 17 (1) shows the result of the conventional method based on Reference 7), in which theindex of the atom corresponding to the nonzero coeﬃcient is directly encoded. Figure 17(2) shows the resultof the conventional method based on Reference 17), in
which the zero run length between nonzero coeﬃcientsis encoded. The k-adaptation is not performed in bothFig. 17(1) and Fig. 17(2). On the other hand, Fig. 17(3)shows the result of applying k-adaptation to Fig. 17(2).
We found from Fig. 17 that the introduction of zero run
length can reduce the amount of information generated
for the indices, so the total amount of information decreases accordingly. However, it should be noted thatthe amount of information for the quantized level number of the nonzero coeﬃcients has not been reduced. On
the other hand, introduction of k-adaptation can not only
reduce the amount of information for zero run length but
also reduce the amount of information for level number,as a result it is possible to reduce the total amount of
information up to 11.0% compared to Reference 7) andup to 4.7% compared to Reference 17). These characteristics were also found to be similar when using diﬀerent
T0 and QP.
Next, we veriﬁed the eﬀectiveness of adaptation basedon the feature of the atoms. The experiment was per-138

Fig. 17 Number of generated bits by (1) conventional7),(2) conventional(zero run)17) and (3) k-adaptive,for (a) BQTerrace, (b) BasketballDrive, (c) Cactus,(d) ChristmasTree, (e) Kimono1 and (f) ParkScene.
T0 = 5, QP=24

formed under the condition that the zero run length andthe nonzero AC coeﬃcient level were classiﬁed according
to the number of nonzero coeﬃcients in each block, andthey were encoded independently. We created a dictionary in which the atoms were reordered using the fourfeatures deﬁned in section 3, and compared the amountof bits using the new dictionary with the amount of bits
using the original dictionary. The measured results areshown in Table 2. The column (a) in Table 2 shows
the amount of information generated by the conventional
method shown in Reference 17), the column (b) in Table 2 shows that generated when k-adaptation is applied
to the conventional method, and the column (d) of Table 2 shows the amount of generated information whenatom reordering is performed in addition to k-adaptation
method. The column (c) and (e) in Table 2 show the reduction rate of the amount of generated information forthe column (b) and (d) based on the column (a), respectively. Table 2 shows that reordering of atoms by any of
the four features makes it possible to reduce the amountof generated information compared to before reordering.
In particular, it can be conﬁrmed that the amount ofgenerated bits can be minimized when using the atomfeature value F3 . The reason is that, the feature value F3
is highly correlated with the nonzero AC coeﬃcient level
as described in section 3. As a result, the amount of bits
nonzero AC coeﬃcient levels can be reduced by adopting
diﬀerent code assignment rules for them according to thefeature of atoms. Also, the reordering of atoms can concentrate the distribution of zero run length closer to zero,and leads to a reduction the amount of bits for zero run
length.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Table 2 Number of generated bits (kbit); (a) conventional17), (b) k-adaptation, (c) bit saving ratio(%): ((b)-(a))/(a),(d) k-adaptation+atom reordering with the feature F1 , F2 , F3 , F4 , (e) bit saving ratio(%): (F3 -(a))/(a)
(d)
Image
(a)
(b)
(c)
(e)
F1
F2
F3
F4
Index 117.2 115.8 -1.2 115.7 115.3 115.1 113.8-1.8
BQTerrace
Level
50.5
47.6 -5.7
46.3
44.0
44.2
45.5 -12.5
Total 167.7 163.4 -2.5 162.0 159.3 159.2 159.3
-5.0
Index
79.8
76.9 -3.6
77.6
73.4
73.6
73.6
-7.7
BasketballDrive Level
17.7
16.0 -9.7
15.1
12.9
12.9
13.4 -27.1
Total
97.5
92.9 -4.7
92.7 86.4
86.5
87.0 -11.3
Index 108.4 106.5 -1.8 104.9 104.0 103.4 103.7-4.6
CactusLevel
29.9
28.7 -3.8
27.3
24.5
24.7
25.8 -17.5
Total 138.3 135.2 -2.2 132.2 128.5 128.1 129.5-7.4
Index 136.4 134.8 -1.1 134.2 134.5 134.0 133.7-1.7
ChristmasTree Level
55.5
52.0 -6.3
51.0
49.6
49.4
50.4 -11.0
Total 191.9 186.8 -2.6 185.2 184.1 183.5 184.2
-4.4
Index
63.6
61.6 -3.2
60.1
58.6
58.4
59.3
-8.2
Kimono1
Level
13.1
12.3 -6.2
11.5
9.5
9.6
10.5 -26.9
Total
76.7
73.9 -3.7
71.5
68.1
68.0
69.8 -11.4
Index 104.6 102.3 -2.2 100.0
99.5
98.8
99.7
-5.5
ParkScene
Level
24.0
23.0 -4.2
21.7
18.9
18.9
20.3 -21.2
Total 128.6 125.3 -2.6 121.7 118.4 117.7 120.0-8.4
Table 3 BD-rate[%] between proposed method and Reference 17)
Number of class
Image
T0
16
32
64
128
3
-1.43 -1.35 -1.37 -1.31
BQTerrace
5-5.50 -6.27 -5.56 -6.10
7-4.27 -4.21 -4.16 -4.11
3-2.03 -2.07 -3.45 -1.33
BasketballDrive 5-0.48 -0.72 -1.08 -0.81
7
0.44
0.23
0.17 -0.35
3-2.37 -3.73 -2.94 -3.67
Cactus5-3.12 -3.27 -3.53 -3.26
7-3.72 -3.36 -3.41 -3.28
3-1.13 -0.83 -0.83 -1.06
ChristmasTree
5-1.26 -4.97 -4.27 -1.17
7-3.02 -3.00 -3.26 -2.41
3 -3.13 -3.08 -2.22 -2.70
Kimono1
5-3.00 -2.90 -2.86 -2.61
7-3.02 -2.72 -2.62 -2.58
3 -4.90 -1.94 -3.87 -3.01
ParkScene
5-3.30 -3.80 -3.41 -3.37
7-3.53 -3.57 -3.43 -3.50

Finally, we measured the overall performance under setting the feature value used to reorder the atoms to F3
which is the most eﬀective to reduce the amount of bits.
The class number C was set to four types of 16, 32, 64
and 128. For each C, the sparse parameter T0 was set
to 3, 5 and 7, and the quantization parameter QP was
set to 16, 24, 32 and 40. The total amount of bits generated was measured as the sum of the amount of bits
for the class number, for the number of nonzero coeﬃcients, for coeﬃcients of DC atom and for coeﬃcients of
AC atom. The measured average performance gain, BDrate, between the proposed method and the conventional

139

(a)

(b)

(c)

(d)

Fig. 18 Image quality comparison (0.30 bit/pel); (a) original, (b) enlargement of partical original image, (c)decoded image of (b) (conventional17)), and (d) decoded image of (b) (proposed)

method is shown in Table 3. Table 3 shows that the
proposed method can reduce the total amount of bits up
to 6.2% compared to the conventional method.
K-SVD is a block based processing similar to DCTbased coding, so the block noise occurs when the compression ratio becomes high. Using the proposed entropy
coding method, a smaller quantization step can be used incomparison with the conventional entropy coding methods under the same compression ratio. As a result, block
noise can be reduced as shown in Fig. 18.
In the experiments by combining the number of classes(C = 16, 32, 64, 128), quantization step (QP = 16, 24,
32, 40) and sparsity (T0 = 3, 5, 7) as experimental pa-

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

rameters, we clariﬁed that the proposed entropy coding
is eﬀective at any bit rate from high compression to low
compression. When K-SVD is applied to actual compression coding, multiple parameters of the number ofdictionary classes, the quantization step and the sparsity
parameter must be controlled in order to keep the amountof generated bits within a predetermined compression ratio. It is considered that the proposed entropy coding
method can be utilized for the rate-distortion optimization control for image compression with K-SVD, it will
be addressed as a future work.

5.

Conclusions

In this paper, we focused on an eﬃcient entropy coding
for sparse coeﬃcients when sparse coding is applied to image coding. First, the statistical properties of the sparse
coeﬃcients under various sparsity parameter and quantization step were analyzed in detail. Next, based on theanalysis, we proposed two methods of the adaptive code
assignment with the number of nonzero coeﬃcients in theblock and reordering of atoms by their features. The proposed methods enable to encode the indices and quantized levels of the nonzero sparse coeﬃcients eﬃciently.
Finally, by experiments using various sparsity parametersand quantization width, it was clariﬁed that the amountof generated bits can be reduced up to 6.2% comparedwith the conventional method. The application to sparse
coding for intra-frame/inter-frame prediction error, andthe application to ultra-high deﬁnition video such as 4K
and 8K will be studied as interesting researches in the
future.

References
1) ISO/IEC 10918-1 | ITU-T Recommendation T.81: “Information Technology - Digital Compression and Coding ofContinuous-tone Still Images: Requirements and Guidelines”(1994).
2) ISO/IEC 14496-10: “Information Technology - Coding of
Audio-Visual Objects - Part 10: Advanced Video Coding”(2014).
3) ISO/IEC 23008-2: “Information Technology – High Eﬃciency
Coding and Media Delivery in Heterogeneous Environments –Part 2: High Eﬃciency Video Coding” (2017).
4) Versatile Video Coding (VVC) | JVET, https://jvet.hhi.
fraunhofer.de (2019).
5) O. Bryt, M. Elad: “Compression of Facial Images Using the KSVD Algorithm”, Journal of Visual Communication and Image
Representation, Vol.19, No.4, pp.270–282 (2008).
6) M. Kalluri, M. Jiang, N. Ling, J. Zheng, P. Zhang: “Adaptive
RD Optimal Sparse Coding with Quantization for Image Compression”, IEEE Trans. on Multimedia, Vol.21, No.1, pp.39–50(2019).
7) Je-Won Kang, M. Gabbouj, C.C.J. Kuo:

“Sparse/DCT

140(S/DCT) Two-Layered Representation of Prediction Residuals
for Video Coding”, IEEE Trans. on Image Processing, Vol.22,
No.7, pp.2711–2722 (2013).
8) M. Aharon, M. Elad, A. Bruckstein: “K-SVD: An Algorithm
for Designing Overcomplete Dictionaries for Sparse Representation”, IEEE Trans. on Signal Processing, Vol.54, No.11,
pp.4311–4322 (2006).
9) Y.C.C. Pati, R. Rezaiifar, P.S.S. Krishnaprasad: “Orthogonal Matching Pursuit: Recursive Function Approximation with
Applications to Wavelet Decomposition”, Proc. of 27th Asilomar Conference on Signals, Systems and Computers, pp.1–5(1993).
10) A. Vedaldi, B. Fulkerson: “Vlfeat: An Open and Portable Library of Computer Vision Algorithms”, Proc. of the International Conference on Multimedia, p.1469 (2010).
11) J.W. Kang, C.C.J. Kuo, R. Cohen, A. Vetro: “Eﬃcient Dictionary Based Video Coding with Reduced Side Information”,
IEEE International Symposium of Circuits and Systems (ISCAS), pp.109–112 (2011).
12) B. Olshausen, D. Field: “Natural Image Statistics and Eﬃcient
Coding”, Network: Computation in Neural Systems, Vol.7,
No.2, pp.333–339 (1996).
13) ISO/IEC 13818-2: “Information Technology – Generic Codingof Moving Pictures and Associated Audio Information” (2000).
14) J. Sole, R. Joshi, N. Nguyen, T. Ji, M. Karczewicz, G. Clare,
F. Henry, A. Duenas: “Transform Coeﬃcient Coding in
HEVC”, IEEE Trans. on Circuits and Systems for Video Technology, Vol.22, No.12, pp.1765–1777 (2012).
15) N. Pati, A. Pradhan, L.K. Kanoje, T.K. Das: “An Approach to Image Compression by Using Sparse Approximation
Technique”, Procedia Computer Science, Vol.48, pp.769–775(2015).
16) A. Budillon, G. Schirinzi: “Low Bit Rate SAR Image Compression Based on Sparse Representation”, Digital Image Processing, pp.51–70 (2012).
17) R. Vinith, A. S. Aswani, K. Govindan: “Medical Image Compression Using Sparse Approximation”, International Journal of Advanced Computer and Mathematical Sciences, Vol.6,
No.2, pp.30–39 (2015).
18) D. Webb: “The Statistics of Relative Abundance and Diversity”, Journal of Theoretical Biology, Vol.43, No.2, pp.277–291(1974).
19) HDTV Test Materials for Assessment of Picture Quality, https://www.ite.or.jp/content/test-materials (2018).

(Received May 31, 2019)(Revised August 27, 2019)

Ji

WANG

He received the B.E. and M.E. degrees from
Chiba Institute of Technology, Chiba, Japan,
in 2012 and 2014, respectively. Currently heis studying in Graduate School of Information and Computer Science, Chiba Instituteof Technology. His research interests include
next generation video coding, image coding.

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Yoshiyuki YASHIMA (Member)He received the B.E., M.E., and Ph.D degrees from Nagoya University, Nagoya, Japan,
in 1981, 1983 and 1998, respectively. In
1983 he joined the Electrical Communications Laboratories, Nippon Telegraph andTelephone Corporation (NTT), Kanagawa,Japan, where he has been engaged in the
research and development of high quality
HDTV signal compression, MPEG video coding algorithm and standardization. He wasalso a visiting professor of Tokyo Institute ofTechnology during 2004-2007. In 2009, hemoved to Chiba Institute of Technology. Currently, he is a professor at the Faculty of Information and Computer Science, Chiba Institute of Technology. His research interests
include next generation video coding, preand post-processing for video coding, processing of compressed video, compressed video
quality metrics and image analysis for videocommunication system. He was awarded
the Takayanagi Memorial Technology Prize in2005, and received the FIT Funai Best Paper
Award in IEICE in 2008. Professor Yashima
is a Fellow member of the IEICE, a senior
member of the IEEE, and a member of ACM,IPSJ and ITE.

141

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Value Estimation of SpO2 Using a Non-Contact Method: Comparison with a Contact Method
Yoshimitsu NAGAO†‡ (Member), Yanan GAO†‡, Jiang LIU†‡ (Member), Shigeru SHIMAMOTO†‡†

Department of Computer Science and Communications Engineering, WASEDA University‡ Graduate School of Fundamental Science and Engineering, WASEDA University<Summary> It is impossible to estimate arterial oxygen saturation (i.e., SpO2) for individuals by using conventional approaches
unless the given sensor of the pulse oximeter is attached to an individual’s finger. This study introduces a novel method to solve
this problem. This study has focused on realizing SpO2 measurements by using non-contact space measurements, and thesuccess of the approach is validated through experiments. Finally, despite a few problems including the susceptibility of the
proposed approach to other light interference, the study offers an initial method to utilize laser wavelengths for thefore-mentioned purposes. As the characteristic of elderly individuals involves the hardening of the fingertips’ skin, it is difficult
for the light of a probe to enter the same. Therefore this study can be applied to medical care, elder care, and other related
fields. .Additionally, there are cases that symptoms are unmeasurable. The light receiving property to other light interference inspace constitutes a problem for the fore-mentioned method.
Keywords : SpO2, saturation pulse, non-contact space measurement, light sensing, biological information, LMM,
pulse oximeter, blood oxygen saturation

fingers and measures it by using a non-contact methodwith two different wavelengths, and it will be makepossible to calculate the values of SpO2. Thus, the study
proposes a completely different approach to measure thetarget SpO2 values. Light is measured by detecting
transmitted light in the red and infrared spectra.

1. Introduction1)-4)
1.1 Background of starting this research
Initially, a trigger of this research is our following
experience. When the author visited a considerable
elderly in the hospital, the nurse started a measurement
by attaching a probe to the old man’s finger to measure
SpO2 (Oxygen saturation from the pulse). Unfortunately,
the saturation value was not able to be read. The reason
was that as the human beings get aged, the wall thickness
increases and becomes harder, so it is difficult for thelight emitted into the probe to enter the finger, whichmakes it impossible to measure the value. This kind ofthing is likely to happen often. Under such circumstances,
if we cannot know biological information as to whether
the necessary oxygen has been brought into the humanbody or not, it is considered to be a problem related to lifeand death. Knowing this kind of situation, we started thisresearch, looking for a non-contact measurement method
to read the SpO2 values.

1.3 Measurement principles and types of SpO2 values
The light-receiving element transmits light to themeasurement object medium and extracts the componentof light that is not absorbed after penetrating themeasurement object, with respect to the molar extinction
coefficient of the molecule and wavelength of the light,
the artery, vein and transmittance wavelengthcharacteristics. The transmittance exhibits a wide
absorption band range from 640 [nm] to 1600 [nm] or
more. The standard deviation corresponds to the numberof received data and is in the interval between the peak of
the heartbeat’s waveform and the time of the peak. It is
crucial to obtain information with respect to the amountof oxygen that is supplied to the blood to measure thestate of the living body. The index is termed as the arterial
blood oxygen saturation or SpO2. The device that iscurrently used can continuously and non-invasively
measure SpO2 and is called a pulse oximeter. The SpO2
values are calculated by sandwiching the finger between
irradiance devices to irradiate two types of lights withdifferent wavelengths, and then by measuring the amountof transmitted light to get the result. Although themethods used in extant studies correspond to the first
method that uses laser light, they are facing an inherent
problem as they are susceptible to interferences fromother light sources in the environment. This is considered
as a primary difficulty in realizing contactless approaches
today. The human body is mainly composed of water. The

1.2 Spatial measurements of blood oxygen saturation
SpO2
Existing probes are equipped with excited individual
light-emitting elements of red and infrared LEDs. They
use the different absorption rate of red light and infrared
light due to the binding of oxygen and hemoglobin inhuman blood to measure blood oxygen saturation degree(i.e., SpO2 of the artery (oxy-hemoglobin) and vain
(Deoxy-hemoglobin) values) by using contact-based
methods. Furthermore, only contact-based methods havebeen used for these types of measurements.
Only a few impractical approaches were proposed byextant studies, and there is paucity of studies examining
contactless approaches. Therefore, in this research, a lightemitting element transmits light to penetrate human142

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）(a)Fig. 1 Characteristics of the transmittance wavelength ofcontact sensors(b)Fig. 3 A pulse oximeter; (a): product, (b):the principle ofthe contact SpO2 value measurement

Fig. 2 H2O spectrum as obtained by near-infrared spectroscopy

does not exist in nature. When a stimulus such as highintensity light is introduced into the laser cavity, the laser
is radiated and light is generated. Although the output islight, the laser and amplified light are greatly different
from sunlight and bulb light. Laser and amplified lighthave special characteristics, so there is a danger for theoperation of the equipment, during servicing. A laser oramplified light source generates visible light or invisible
light of monochromatic light with very high intensity.
Laser and amplified light are coherent, meaning that lightwaves are in phase with each other.
At present, standards for correctly manufacturing andusing laser products are defined globally by IEC 60825-1.
EU countries also comply with this (EN-60825-1), Japan
complies with IEC 60825-1 compliant JIS C 6802-1:2005"Radiation safety standard of laser products"7) (revised
January 20, 2005) of radiation safety standards.
Among them, from the viewpoint of safety, it is aclass defined for a visible laser and visible light with asingle pulse (pulse width 10-7[s]) of 1 [mW] or less, and 2[W] or less in He - Ne, against a blinking time of 0.25[s].
For the safety to ensure that the retina is not damaged inthis research, we are taking into consideration within thescope of Class 27).

percentage of water corresponds to approximately 90%
for fetuses, 75% for newborns, 70% for children, 60% to
65% for adults, and 50 to 55% for elderly individuals.
This indicates that the moisture ratio reduces with aging,and thus it is difficult to measure the SpO2 values withaging. The wavelength and absorbance of blood andwater are shown in Fig. 1 and Fig. 2.

2. System Architecture (PPG and LMM)
2.1 Current SpO2 value measurement method formeasuring blood oxygen saturation
Currently, contact sensors comprise of photoplethysmography (PPG) that uses conventional
photoelectric volume pulse waves. These types of sensors
are incorporated into a probe and clip-on device thatshould be attached to an earlobe or a fingertip. As shownin Fig.1, and Fig.2, there are two measured wavelengths,
namely a wavelength that approximately corresponds to650 [nm] in the red light range and another wavelengththat approximately corresponds to 940 [nm] in the infrared
range. As shown in Fig. 3, using a pulse oximeter, it will
be performed that the finger interposes between twoprobes as shown later. Then the measurement value andthe output value of the acquired data are compared. The
size of these devices approximately corresponds to 6.6[cm] × 2.9 [cm] (Fig. 3 (a)).

2.2.2 A mechanism for performing space measurementwith two wavelengths
In this study, two lights with different wavelengths are
emitted in a non-contact manner by a SpO2 spatial
measurement transmission method that is proposed basedon the laser transmission method. Specifically, as depicted

2.2 Laser beam measurement method (LMM)
2.2.1 Fear of danger during operation by using laser
Lasers and amplifier light sources generate light that143

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）beam of 980 [nm] (1[mW]) and a laser light of 640 [nm](1[mW] or less). Recent studies confirmed thatnear-infrared light (i.e., 700 [nm] to 1,500 [nm]) exhibitshigh permeability in living tissues.
Oxygen concentrations in living tissues are measured
by using light in this region. Thus, advancements are
expected in terms of new non-invasive measurement
techniques (as shown in Table 1).
LMM is radiated by red and infrared diode laser
excited elements (as shown in Fig. 5). In the study, due tonon-contact, the difference in light transmitted through thefinger (which is the medium over which light of two
different wavelengths of output is radiated) is measured,and the target SpO2 value is also measured. An infraredlaser beam of 980 [nm] (1 [mW]) is used as the laser lightof 640 [nm] (1[mW] or less). Recent studies indicated thatnear-infrared light (from 600 to 1500 [nm]) has a highpermeability with respect to living tissues and that oxygen
concentration in living tissue is measured by using light inthis region. Thus, extant studies endeavored to use
technological advancements to establish a newnoninvasive measurement technique. In this study, LMM
is used to realize the measurement in space (as shown in
Fig. 6). Experiments indicate that the light receiving
element is significantly affected by the surrounding lightenvironment. Additionally, the study involved developinga structure that reduces the influence of the surrounding
light environment by attaching a special cover andfabricating a light receiving element module to theelement8).

Fig. 4 Illustration of the manner in which SpO2 value ismeasured in the proposed approach
Table 1 Measurement principles of different lightemitting systems
Conventional method
Photoelectric volume pulse wave
sensor
PPG: (Photo plethysmography)
Contact and transmissive(mainstream), reflective

Proposed method (the present
study)
Laser beam measurement
method(LMM)
Non-contact and transmissive

Emission
systemRed LED(R)IR LED(IR)Red laser
LM-101-A2
Diode-pumped
solid-state laser

Infrared laser
PPLS
Diode-pumped
solid-state laser

The light-receiving
element sensitivity
Sensitivity wavelength range
Maximum sensitivity
wavelength

Phototransistor
1000~4300μA

Phototransistor
145~580μA

Si PIN
Photodiode320~1100nm
960nm

Si PIN
Photodiode320~1100nm
960nm

Emission
wavelength
outputPower supply

660±3nm900~940±10nm 640nm
1mW or less
DC3V
40mA or less

980nm
5mW or lessDC3~5V
40mA or less

Measurement
objectarteryvein

vein

artery3. Deriving SpO2 Values by Irradiance Passing
through a Medium
3.1 Measurements of laser lightFirstly, SpO2 is defined as the SpO2 value that isderived by a calculation formula if the measured value isobtained. It is necessary to accurately analyze waveforms
and numerical values via measurements of laser light. The
principle of deriving SpO2 value is as follows: A fingertip
is irradiated with two lights, i.e., Oxy-hemoglobin andDeoxy-hemoglobin, that correspond to red light andinfrared light, respectively, based on the absorption
characteristics of light and from the ratio of the magnitudeof the pulse wave of transmitted light to arterial blood andvein to calculate absorption characteristics. Theoretically,
oxygen saturation corresponds to 100% when the totalhemoglobin contains oxygen and changes to
Oxy-hemoglobin. The SpO2 value, an indicator of theamount of oxygen present in blood is then determinedfrom the amplitude ratio of the pulse wave due to light atwavelengths of 640 [nm] and 980 [nm] that are irradiatedbased on the oxygen saturation of arterial blood andvenous blood, respectively.
With respect to the oxygen saturation of arterial bloodand venous blood, 70% of the gas in venous blood consistsof red darkened with oxygen. It should be noted that thenumerical value of SpO2 is expressed as a percentage. The

Fig. 5 Conceptual diagram of our non-contact
SpO2 measurement system

in Fig. 4, light is transmitted through a finger that serves asa medium and is measured by the light-receiving element,
and the target SpO2 value is obtained without using aprobe. In this method, the oxygen saturation of arterial
blood and venous blood is measured by utilizing thedifferent absorption rates between red light and infrared
light due to the binding of hemoglobin in the oxygen to theoxygen in the blood to detect the transmitted light. Given
the non-contact approach, by measuring two lights withdifferent wavelengths that penetrated after the finger, it
will help to measure the target SpO2 value.
Compared with the conventional methods, this study
proposes a non-contact approach that uses an infrared laser

144

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）reference value typically ranges from 97% to 99% for ahealthy individual, and it is largely considered that
pulmonary function declines if the reference value of anindividual equals or is lower than 90%.
3.2 Mechanism of derivation of SpO2 value using
Lambert-Beer's law4), 9), 10)
This section compares incident light to transmitted
light in a specific concentration of a solution. If theabsorption coefficient of the given solution at a specificwavelength is determined in advance, then it is possible
to obtain the concentration of the solution by measuring
the incident light, transmitted light, and the distance ofthe solution. Thus, arterial blood oxygen saturation of
SpO2 is expressed by B will be given as follows:‫=ܤ‬⊿஼௢௫௬・௅ುభషುమ
⊿஼௢௫௬・௅ುభషುమ ＋⊿஼ௗ௘௢௫௬・௅ುభషುమ

Fig. 6 Vital signs obtained from the density of the given liquid

A’= — log (I out /I in) = ε C L ＋ S,

. (1)where, S represents the attenuation of the light intensity
due to scattering.
When the concentration of the solution changes fromC to (C + ΔC), the quantity of transmitted light changes to
Iout + ΔIout, and the relation is given as follows:This is followed by obtaining the maximum amplitude ofeach hemoglobin change within a heartbeat:M=⊿‫ܮ・ݕݔ݋ܥ‬௉ଵି௉ଶ ＋⊿‫ܮ・ݕݔ݋݁݀ܥ‬௉ଵି௉ଶ

,(2)where, ⊿Coxy・L୔ଵ-୔ଶ represents the red laser and amountof the change, and ⊿‫ܮ・ݕݔ݋݁݀ܥ‬௉ଵି௉ଶ represents the infraredlaser and amount of the change.

Ar=— log[ (I out + ΔI out) / I in] = ε (C + ΔC) L＋S , (5)The attenuation (S) of the light intensity due toscattering is not changed in Eq. (2) and Eq. (3), and thisfact results in the following expressions:

Furthermore, a state in which hemoglobin protein
which is present in erythrocytes, a protein consisting offour peptide chains and heme is bound, oxygen is boundto the iron atom at the center of heme.
The Deoxyhemoglobin estimation Hb (vein) involves
checking the waveform with an oscilloscope by using redlaser light and infrared laser light that are not coupled withoxygen.
Additionally, two changes, namely ΔCoxy and
ΔCdeoxy are observed in the waveform of the oscilloscope
when a finger is held between the light-emitting element
and the light-receiving element. In the fore-mentioned
case, the two components are transmitted normally, andthus detection is possible.
As illustrated in Fig. 6, the Lambert-Beer law is used
to obtain the incident light Iin to penetrate a solution withthe constant concentration to determine the transmitted
light by measured absorbance of the solution. This isexpressed as follows:
A = — log (I out / I in) = ε C L,

(4)

R=— log[ (I out + ΔI out) / I in ] = ε ΔCL,R(λ)=— log[ (I out(λ) + ΔI out(λ)) / I in(λ) ]= (ε oxy(λ)･ ΔCoxy+ ε deoxy(λ)･ΔCdeoxy)･L .

(6)
(7)

From the incident light that illuminates living tissue ofa specific wavelength λ, it is possible to know the amountof absorption due to scattering in the living body and theamount of permeation to the outside of the living body.
In Eq. (6) as shown above, the concentration variation
C oxy of OxyHb determines the concentration amount ofchange ΔCdeoxy of DoxyHb. Specifically, given that twovariables are used to determine ΔCdeoxy, with respect toΔCoxy the red-infrared extinction coefficient is used at thetwo wavelengths corresponding to 640 [nm] and 980[nm], and this results in the following expressions of Eq.
(8) and Eq. (9), respectively :(3)

R(980)=— log[ (I out(λ980) + ΔI out(λ980) ) / I out(λ980) ]= (ε oxy (λ980) ･C oxy+ ε deoxy (λ980) ･ΔC deoxy) ･L , (8)where Iin is the initial light intensity, Iout is the lightintensity after passing through the solution, A represents
the absorbance, ε represents the extinction coefficient of
the solution, C represents the concentration of thesolution, L represents the distance (i.e., the average optical
path length). The extinction coefficient of the solution ispreviously determined at a specific wavelength epsilon.
Subsequently, Iin is used to determine concentration Cof the solution by measuring Iout via L. the modified
Lambert-Beer Law, if Lambert-Beer law is applied to thescattered medium, will be as follows:

R(640)=— log[ (I out(λ640) + ΔI out(λ640) ) / I out(λ640) ]= (ε oxy (λ640) ･ΔC oxy+ε deoxy (λ640) ･ΔC deoxy) ･L .(9)If optical path length L cannot be set, L is left as L·ΔCoxyor L·Δcdeoxy, for D·ΔCoxy and L·ΔCdeoxy, i.e.,
ΔCoxy+L·ΔCdexoy=L·ΔCtotal, and the concentration
variation of the total hemoglobin is given as ΔCtotalHb.
Oxygen is present in blood in a form that is bonded tohemoglobin. Oxy-hemoglobin (artery), which is termed

145

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

1.0

Infrared laser

Red laser

980

640

0.5

Hb
HbO2

600

700

Absorption coefficient α(λ)

800

900

1000

Wave length (nm)Fig. 7 Ttransmittance wavelength characteristics of red lightand infrared ray

as HbO2, is bound to oxygen. Arterial blood that contains
significant amount of oxygen exhibits a bright red color.
Conversely, deoxy-hemoglobin (vein) that consists of Hbthat is not bound to oxygen, and venous blood after
releasing oxygen into the body exhibits a dark color. A
conventional method involves superimposing theemission spectrum of an LED (which is a typical lightemitting diode) and the relative light absorption spectrumof oxygenated and deoxygenated hemoglobin. The
transmittance wavelength characteristics of red light andinfrared ray are shown in Fig.7.

Fig. 8 Oxygen dissociation curve

reduced. The oxygen partial pressure corresponding to 60[mmHg] and the Oxygen saturation degree below 90%
results in a condition that necessitates oxygen inhalation,
which is termed as respiratory insufficiency. Thus, 40[mmHg] released venous blood contains approximately
23% of oxygen although 75% of the oxygen is bound.
Hemoglobin binds tightly to oxygen in places with highoxygen partial pressure and is released immediately inplaces in which oxygen partial pressure is low (peripheral
tissues). An increase in CO2 due to body movements
tends to acidosis. Increased hyper metabolism that
corresponds to an increase in body temperature due todiseases leads to the production of 2-3 DPG in hypoxia(increased number of erythrocytes due to glycolysis).
This increases the demand for oxygen, and oxygen isused in large quantities. With respect to 40 [mmHg], thesaturation level significantly decreases from 75% to 60%.
The released oxygen gradually increases. With respect toP50 (saturation degree: 50%), which corresponds to thehalf-life of oxygen saturation, it causes a state in which
oxygen hypoxemia is created and lasts long. It is assumed
that the dissociation curve moves to the right and thecurve He moves oxygen to the right by increasing theoxygen.

3.3 Arterial blood oxygen saturation (SaO2 value)11)
The 50% saturated blood spectrum is considered, andthe relative red absorbance is compared with the IR
indicated by the black circle. The broken line shows the
spectrum of 50% saturated blood and the relative
absorbance of red. Additionally, R is indicated by a circle.
Based on the absorption characteristics of light, twofingertips of Oxy-hemoglobin and Deoxy-hemoglobin red
light and infrared light are irradiated to the fingertip. The
absorption characteristic of only arterial blood is
calculated from the ratio of the magnitude of the pulse
wave of the transmitted light. An oxygen saturation
corresponding to 100% is obtained when the totalhemoglobin contains oxygen and it changes to
Oxy-hemoglobin. Arterial blood oxygen saturation (SaO2)
is determined by the oxygen saturation of arterial bloodfrom the amplitude ratios of the pulse waves due to thetwo irradiated lights of 640 [nm] and 980 [nm].
In the oxygen dissociation curve (Fig. 8), the arterial
blood oxygen saturation (SaO2) coupled with hemoglobin
is plotted on the vertical axis, and the arterial oxygen
partial pressure (PaO2) is plotted on the horizontal axis. In
the solid line of the bell, the oxygen partial pressure inside
the alveolar becomes normal at 100 [mmHg], and theoxygen saturation corresponds to approximately 98%.
With respect to an oxygen partial pressure of 60 [mmHg],
oxygen saturation reaches approximately 90%. Oxygen
saturation is maintained despite a slight decrease in theoxygen partial pressure. Furthermore, the oxygen carrying
capacity with respect to the periphery is high. A large
amount of oxygen is used in peripheral tissues from amixed blood level of oxygen partial pressure of 60[mmHg] or less, and thus the oxygen saturation levelsignificantly reduced. Oxygen carrying capacity is

4. Production of SpO2 Value Spatial Instrument
Prototype12), 13)
4.1 Sensor module installed in equipment14)
Initially, the study involved measuring SpO2 data uses
a non-contact method through irradiating light byattaching light-emitting elements of two wavelengths to agantry (Fig. 9).
4.2 Equipment15)-17)
As depicted already, a device is used to detect SpO2
values by using red laser (i.e., artery) and infrared laser(i.e., vein). The device is used to detect normal values in astable state. A relationship exists between the oxidized

146

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）hemoglobin concentration of the artery at 640 [nm] andthe reduced hemoglobin concentration of the vein at 980[nm] in relation to the molecule (i.e., molar extinction
coefficient) and wavelength. Therefore, the standard
deviation is a method of grasping the peak of the
waveform and averaging the number of received data at
each time interval by a fixed number. This is used to
identify the quality of the data and by using a numericalvalue of 25 or less. Specifically, the value of the ratio
between the AC component and the DC component iscalculated from the visible light and infrared light as the
R-value of the SpO2 value. With respect to the study
specifications, the light-receiving surface
size
corresponded to 5.5 [mm] × 4.8 [mm] (Fig. 10), theeffective light-receiving area corresponded to 26.4[mm2].
The reverse voltage (RV) displayed an absolute maximumof 35 [V], the sensitivity wavelength range (λ) from 320[nm] to 1100[nm], and the maximum sensitivity
wavelength (λp) receiving a good component and thesignal by 980[nm].
As shown in the signal processing concept in Fig.10,
this constitutes a device for detecting SpO2 value by using
red laser (artery) and infrared laser (vein). A relationshipexists between oxidized hemoglobin concentration,
Oxy-hemoglobin (640[nm] for artery) and reduced
hemoglobin concentration Deoxyhemoglobin (vein980[nm]), the molecule (molar extinction coefficient) andthe wavelength.
This is used as the quality of the data and reliability isdetermined with a numerical value of 25 or less. The valueof the ratio between the AC component and the DC
component is calculated as visible light and infrared lightas corresponding to the R value of the SpO2 value to be
calculated. The SpO2 value is calculated from the R value
and is expressed in terms of a percentage value.
Experiments on the effectiveness of the proposed method
were performed by linking the created program and thefabricated board to the laser beam module (as shown in
Fig. 11). The results indicated that the SpO2 value was
effective and confirmed the success of the spatial
measurement by non-contact method.

Fig. 9 Dimensions of light receiving element

Fig. 10 Conceptual diagram of our signal processing
workflow(a)

4.3 Proposal method of performance evaluation18)-19)
4.3.1 How to check the infrared light hit on the finger
The wavelengths of two kinds of light used here are
640 [nm] (visible light region) and 980 [nm] (infrared
region) as shown in the photograph (Fig.11). Since 640[nm] emits red light, it can be grasped visually so it's easy
to hit light on your fingers. However, 980 [nm] cannot be
grasped visually. You can check the light of 980 [nm],which should not be seen, by looking at the light while
operating the smart camera or the CCD camera.
Two kinds of light are reflected in the photographs(Fig.11) because they have high sensitivity in the same
wavelength region due to the spectral sensitivity
characteristics in the near-infrared region.

(b)

Fig. 11 Prototype of non-contact SpO2 measuring device inoperation and measuring with (a) right and (b) left
index finger

147

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

4.3.2 Can the SpO2 value be estimated even when thefinger is set at any angle?20)
Although the angle of the finger to the light will cause
a problem, in the experiment, when changing the angle ofthe finger hitting the light with the finger, the SpO2 valuechanges. Even if the fingertip is changed by about 30
degrees, even if there is a change in the numerical value
due to reflection or refraction of the light transmitted
through the finger, there are no influences on the final
derived numerical value because the two numerical values
are contrasted.

4.3.3 Can the SpO2 value be estimated when the
distance between the laser and the finger takes any
value
The experimental distance of 110 [mm] is between thelight source and the light receiving part, but the distance tothe finger is 110 [mm] minus finger thickness. In the
waveform data by the oscilloscope, it shows that the
scattering is large when the finger is close to the lightsource, and the scattering becomes smaller as it becomes
farther from the light source and becomes closer to thelight receiving portion.

Table 2 Report data with respect to a commercially available
pulse oximeter

5. Verification of the Prototype Method Proposed
for Contactless Measurement Data21)-22)

Commercial pulse oximeter data report
User Information
Name: yn
Age:
Sex: M
Height/Kg:
Time Length: 00:03:40 Time: 00:01:55
Event Data
SpO2
PR
Total Event
00Time in Event(min)0.0
0.0
Avg. Event Dur.(sec) --------- -----Index(1/hr)
0.0
0.0
Artifact(%)
21.8
21.8
Adjusted Index(1/hr) 0.0
0.0
SpO2 Data
Basal SpO2(%)
98.4
Time (min)<88%
0.0
Events<88%0Minimum Spo2(%)
97
Avg. Low SpO2(%)--------Aglow SpO2<88%--------PR Data
Avg. Pulse Rate(bpm) 75.9
Low Pulse Rate(bpm) 68

5.1 Comparison of a traditional contact-based method
and the proposed non-contact method

Date(Y/M/D):2017/02/07
Nationality:
Comments:%SpO2 Level
99-95
94-90
89-85
84-80
79-75
74-70
69-65
64-60
59-55
54-50
49-45
44-40
39-35
34-30

vents
0
0
0
0
0
0
0
0
0
0
0
0
0

Below(%)
100
95
90
85
80
75
70
65
60
55
50
45
40
35
30

Data comparison was performed by using a
conventional type SpO2 measuring instrument. It should
be noted that the authors served as human subjects in theexperiments. Table 2 and Fig. 12 show the results ofmeasuring SpO2 values by using a commerciallyavailable device. In the figures, the basal SpO2 value
corresponded to 98.4% while the minimum SpO2 value
corresponded to 97%. In the measurement waveform asshown in Fig. 13, the part surrounded by the dotted line
denotes SpO2 from 99% to 97%. Additionally, theprototype SpO2 measurement output is shown in Fig.13(a) and (b) from 52 [s] to 222 [s].
Its waveform was analyzed, and it was determined asstable in saturation from the highest 100% to the lowest
99%.When the interval between 52 [s] and 222 [s] thatserved as the actual measurement time range was
extracted, it resulted in a time range ranging from 0[s] to165[s]. Therefore, the use of an output parameter variablecorresponding to -3 allows the use of the commerciallyavailable one as shown in Table 3. The output numericalvalues in those two figures are combined to show that themeasurements in the space measurement prototype werenormal.

Time(%)
100.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Fig. 12 Measured waveform obtained from a commerciallyavailable pulse oximeter

6. Conclusion
Many reports have been issued on studies of direct
oxygenity and reflectivity of blood oxygen levels. In thispaper, we aim to find the saturation value of SpO2 bycontactless space measurement method for knowing
biological data for applications such as medical care,nursing care, everyday life, field work and so on. We
investigated the effectiveness and characteristics of laser
light and the effectiveness of biometric measurement. The
data measured with the model proposed in this study isalmost the same as the value of SpO2 obtained from acommercially available pulse oximeter (Fig.3), and it was
confirmed that constant and stable numerical display wasproduced.
Furthermore, when considering a wide range ofapplications, you can also use the method proposed in(a) Before correction(b) After correction
Fig.13 Measured waveform obtained from the proposed
prototype SpO2 device

148

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Table 3 Spatial measurement of SpO2 and corrected saturation
value
(i)
124450124810124990125207125599125913126117126591
127240127804128270128973129339129681
129947130160130358130794131174131716131954132323133048134170134613135319135643136456136741137241137614138097138363(ii)(i)
138895139279139885139986140435140581

(ii)

18
17
19
20
77
87
89
83
66
57
55
48
46
48
55
63
71
81
81
73
74
68
55
44
43
41
40
36
40
48
49
53
54
61
58
56
62
63
67

(iii)
1978
1973
2013
2050
1210
77
81
108
154
149
148
161
115
136
147
160
170
85
87
125
128
116
153
293
298
291
275
258
297
191
197
175
184

(iv)
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

(v)
0.37
0.39
0.42
0.39
0.38
0.38
0.39
0.36
0.38
0.39
0.4
0.38
0.35
0.37
0.35
0.36
0.37
0.38
0.37
0.37
0.37
0.36
0.39
0.4
0.34
0.33
0.34
0.42
0.43
0.4
0.33
0.32
0.37

(vi)
101
100
100
100
101
100
100
101
100
100
100
101
101
101
101
101
101
101
101
101
101
101
100
100
102
102
101
100
99
100
102
102
101

(Vii)
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3
-3

(Viii)
98
97
97
97
98
97
97
98
97
97
97
98
98
98
98
98
98
98
98
98
98
98
97
97
99
99
98
97
96
97
98
99
98

(iii)

(iv)
0
0
0
0
0
0

(v)
0.38
0.39
0.43
0.41
0.42
0.45

(vi)
100
100
99
100
99
99

(Vii)
-3
-3
-3
-3
-3
-3

(ix)
97
97
96
97
98
96

105
91
112
170
168
187

(2014).
8) Y, Nagao, M. Hatsuda, J. Liu, S. Shimamoto: “Using Light
Sensing to Acquire SpO2 Biological Information via a
Non-Contact Approach”, Proc. of International Conference on
Mobile and Wireless Technology 2017 (ICMWT2017), pp. 259–
268 (2017).
9) Y. Yoshimura, Determination of chemical species in solution
using light absorption, Second edition,
http://kuchem.
kyoto-ac.jp/ubung/yyosuke/uebung/light_abs03.htm.
(Last accessed December 6, 2019).
10) University of Iowa Health Care , Pulse Oximetry Basic Principles
and Interpretation, pp. 2-3, https://medicine.uiowa.edu/iowaprotocols/pulse-oximetry-basic-principles-and-interpretation (Last accessed December 6, 2019).
11) M. Kanazawa, Pulse Oximeter Handbook, p.20, The JapaneseRespiratory Society (2014).
12) K. Itoh, Body Intelligence System Theory – Control of Movementand Learning with Human Robotics – , pp. 52–78, Kyoritsu
Shuppan Co., Ltd. (2005).
13) Y. Ohba, S. Yamauchi, Edition of the Chemical Society of Japan,
Electron Spin Resonance Spectroscopy, First edition, pp. 106–
150, Kyoritsu Shuppan Co., Ltd. (2017).
14) Y. Ozaki, Invitation to Spectroscopy – New measurement
technology developed by light – , pp.22–35, pp.62–78, Sangyo
Tosho Co., Ltd.(1997).
15) H. Kurosawa, Laser Theory, First edition, pp. 46–53, pp. 58–71,
pp. 233–263, Optronics Inc. (2011).
16) H. Sano, Introduction to Mossbauer Spectroscopy, First edition,
pp.91–102, Kodansha (1972).
17) H. Ushio, Engineering and Laser Basics and Mechanism –Properties and Application of Light –, Second edition, pp. 54–72,
pp. 290–311, Shuwa System (2010).
18) K. Ando, Semiconductor Laser, First edition, pp. 86–89,
Technical Criticism Company (2011).
19) N. Hosimiya, Biometric Measurement, First edition, pp.81–89,
Tohoku Publishing Co., Ltd. (2002).
20) Y. Nagao, Patent Application, International Patent Classification,
October 2019, “Blood Oxygen Saturation Measuring Device”(2019).
21) T. Iijima, et al., Visualization Techniques Biological Information,
First edition, pp.107–134, Biological Information Visualization
Technology Editorial Committee, Corona Publishing Co., Ltd.
(1997).
22) M. Saitoh, Basics of Medical Engineering, First edition, pp.69–87,
pp.140–155, Shokodo Co., Ltd. (1990).

Notes: (i) Observed time, (ii) Heart rate, (iii) Standard deviation(contact type-R), (iv) no data, (v) Non-contact type-R, (vi) The
original S-value, (vii) Parameter, (viii) Output parameter / correctedwith variable / saturation value，(ix) Finger angle about 30 degrees /in refractive index change

this research to obtain SpO2 values for pets and wild
animals, and can also be used for zoo health management.
Therefore, we believe that this research represents animportant and potentially pioneering contribution in thefield of biological information.

(Received September 5, 2017)(Revised November 14, 2017)

References

Yoshimitsu NAGAO (Member)

1) C. Andrews, R. L. Phillips, Laser Beam Propagation throughRandom Media, Second edition, pp. 57–73, The International
Society for Optical Engineering (2005).
2) Y. Ozaki, S. Kawata, Near Infrared Spectroscopy, 2nd edition,
Measurement Method Series 32, Academic Publication Center,
The Spectroscopically Society of Japan, pp. 11–21 (1998).
3) S. Hashimoto, Introduction to Biomedical Engineering, pp.153–
160, Corona Publishing Co., Ltd. (2013).
4) Y. Ozaki, H. Iwahasi, Introduction to Biomolecule Spectroscopy,First edition, pp.13–17, pp. 45–51, Kyoritsu Shuppan Co., Ltd.
(1992).
5) T. Sawada, S. Oda, K. Washio, Photoacoustic Spectroscopy and Its
Application-PAS, Measurement Method Series 32, Academic
Publication Center, The Spectroscopically Society of Japan,
pp.170–178 (1982).
6) H. Ohnishi, I. Tsukahara, Absorptiometry Inorganic, Japan Society
for Analytical Chemistry, Second edition, pp. 15–36, pp. 39–47,
Kyoritsu Shuppan Co., Ltd. (1988)
7) JIS 6802: 2014 (IEC 60825-1: 2014), Safety of Electrical
Instrumentation, Control and Laboratory Equipment – General
Requirements：Safety of Laser Products – Safety of OpticalWireless Communication Systems for Information Transmission

He received the Master degree from WASEDA
University in Graduate School of International
Information and Telecommunications in 2003, andis with doctoral course from 2015. He has been
Invited researcher at GITS research center from2005, at GITS faculty of Science and Engineering
Information Technology．His research field is tomeasure non-contacting space such as blood
oxygen concentration by light sensing andinformation analysis technology in non-contact
high precision human respiration monitoring by
DRM using radio wave to support healthy
longevity society ． He is a staff of Building
Monitoring Application Subcommittee engaged inlog house design and construction. He is a memberof Architectural Institute of Japan, IEICE, IET, andIEEE. He has been Chairperson of common area
study group on architecture and image electronics(AIM) of IIEEJ from 2015．He was IIEEJ financial
director in 2018．

149

IIEEJ Transactions on Image Electronics and Visual Computing Vol.7 No.2 （2019）

Yanan GAO
She received the Bachelor degree in informationengineer from Qingdao University of Science andTechnology, China, in 2012. She received Master
degree in signal and information process from The
Communication University of China and inwireless communication fieldfromWasedaUniversity, Japan, in 2015 and 2018 respectively.
Her research interests include glucose measurement,machine learning and human body communication
for body sensor network, near field communicationand some applications about human healthcare andso on.

Jiang LIU (Member)
She received the M.S. and Ph.D. degrees in
information and telecommunications from Waseda
University in 2006 and 2012, respectively. In theacademic years 2009 to 2012, she was a research
associate at Waseda University. In 2012 she joined
Faculty of Science and Engineering at Waseda
University as an assistant professor, and since 2017she has been an associate professor affiliated with theInternational Center for Science and Engineering
Programs. Her research focuses on the opticalwireless communications, wireless network systems,
near field communication, and their applications on
5G network design and healthcare industry. She is amember of the Institute of Image Electronics
Engineers of Japan (IIEEJ), and the Institute ofElectronics, Information and Communication
Engineers (IEICE). She also serves as the secretary of
the Japan Division for the Institution of Engineeringand Technology (IET).

Shigeru SHIMAMOTO
He received the B.E. and M.E. degrees from the
University of Electro-Communications, Tokyo, Japan,
in 1985 and 1987, respectively. He received the Ph.D.
degree from Tohoku University, Sendai, Japan, in
1992. He was with NEC Corporation from 1987 to
1991. From 1991 to 1992, he was a research associate
in the University of Electro-Communications, Tokyo,
Japan. He was a research associate in Gunma
University, Gunma, Japan, from 1992 to 1993, and wasan associate professor, from 1994 to 2000. In 2000, he
was an associate professor in the Graduate School of
Global Information and Telecommunication Studies(GITS), Waseda University, Tokyo, Japan. Since 2001,
he has been a professor in the Graduate School of
GITS, Waseda University. He was a visiting professorof E.E. at Stanford University in 2008. Currently, he isa professor of department of Communication andComputer Engineering in Waseda University. His main
research interests include satellite communications,mobilecommunications,
opticalwireless
communications, ad-hoc networks, sensor networks,
and body area networks.

150

IIEEJ Transactions on

Image Electronics and Visual Computing

Call for Papers

Special Issue on the 6th IIEEJ International Conference
on Image Electronics and Visual Computing (IEVC2019)(Vol.8, No.1, 2020)

Editorial Committee of IIEEJThe 6 th IIEEJ International Conference on Image Electronics and Visual Computing(IEVC2019) will be held in Bali, Indonesia, on August 21-24, 2019. The aim of the conference
is to bring together researchers, engineers, developers, and students from various fields inboth academia and industry for discussing the latest researches, standards, developments,
implementations and application systems in all areas of image electronics and visual computing.
The Conference already solicited Journal Track (JT) Papers to be included in December
2019 issue of Transactions on Image Electronics and Visual Computing (Vol.7, No.2), andthe deadline of the submission has been announced as September 13, 2019. The editorialcommittee plans to publish the June 2020 issue of the IIEEJ Transactions on Image Electronics
and Visual Computing as the special issue on ``Extended Papers Presented in IEVC2019".
The editorial committee will widely ask for submissions for the papers of the following area byextending the material presented in all sessions of IEVC2019. We hope you would submit yourhigh-quality original papers, after checking the electronic submission guidelines on our site.

Topics covered include but are not limited to:
Image and Video Coding, Transcoding, Coding Standards and Related Issues, Image andVideo Processing, Image Analysis, Segmentation and Classification, Image Recognition,
Image Restoration, Super-Resolution, Color Restoration and Management, Computer Vision,
Motion Analysis, Computer Graphics, Modeling, Rendering, Visualization, Animations,Interaction, NPR, Virtual Reality and 3D Imaging, Data Hiding, Watermarking andSteganography, Content Protection, Bioinformatics and Authentication, Computer Forensics,
Image database, Image and Video Retrieval, Digital museum, Digital Archiving, Content
Delivery, Image Assessment, Image Quality, Printing and Display Technologies, Imaging
Devices, Digital Signage, Electronic Paper, Visual Communication; Human Interfaces
and Interactions, Mobile Image Communication, Networking and Protocols, OpticalCommunication, Hardware and Software Implementation, Image Related Applications, LSI,
Understanding of Human Vision and/or Human Tactile Sense, Web-Related Techniques,
Personalization Technique, Interaction between Human and Computer, Usability,
Accessibility, Image Processing Technique Considered Emotion, Other Fundamental andApplication Technique, International Standardization.

Paper Submission Due Date:

January 8, 2020

The Institute of Image Electronics Engineers of Japan151

Call for Papers
Special Issue on CG & Image Processing Technologies
for Automation, Labor Saving and Empowerment
IIEEJ Editorial Committee
A declining birthrate and a rapid aging population are common problems in developed countries.
Particularly in Japan, the working-age population is declining continuously, after reaching its peak in 1995,
resulting in a shortage of labor. On the other hand, per capita labor productivity in Japan is very low, ranking
21st out of 36 OECD countries in 2018. In order to solve the labor shortage, it is necessary not only to
expand the workforce but also to improve labor productivity.
Low productivity may be due to Japan-specific personnel systems and social customs, but improvements
in productivity can also be expected through automation and labor saving technologies such as autonomous
driving that has been actively studied in recent years and white collar work automation using RPA (Robotic
Process Automation). Specifically, automation applications can be expected to expand into a wide range offields in the future by using evolving image recognition technology, increasing information from IoT devices,and less expensive robots.
Productivity can also be improved by adding value to the product or empowering workers, such as byusing the Cloud. A wide range of image processing technologies can contribute to productivity, such asdecision making support with image analysis, time saving with remote processing using IoT devices, andimprovement of customer service through interactive video processing systems.
Based on this background, we look forward to receiving your papers, system development papers, anddata papers in this special issue.
1.

Topics covered include but not limited to
Image Processing, Image Recognition, Image Detection, Pattern Recognition, Machine Learning,
Computer Vision, IoT, Ubiquitous, Big Data, Autonomous Driving, RPA, Automation, Robotics,
Usability, Interface, Interaction, Other related fundamental / application / systemized technologies.

2.

Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary
contributed paper. If the number of accepted papers is less than the minimum number for the special
issue, the acceptance paper will be published as an ordinary contributed paper. We ask for yourunderstanding and cooperation.

3.

Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.8, No.2 (December 2020)

4.

Submission Deadline:

Friday, May 29, 2020
5.

Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org

6.

Online Submission URL: http://www.editorialmanager.com/iieej/

152

Call for Papers
Special Issue on
Image-Related Technologies for the Realization of Future Society
IIEEJ Editorial Committee
There is a great expectation for an advanced and comfortable society brought about by economic development
and the solution of social issues through the introduction and spread of ICT technology. For this expectation, thegovernment advocates and promotes Society 5.0 as a new future society, following hunting society (Society 1.0),
agricultural society (Society 2.0), industrial society (Society 3.0), and information society (Society 4.0). It isclearly stated that this purpose is to build a system that coalesces cyber space (virtual space) and physical space(real space) at a high level, and integrates drones, AI devices, medical / nursing care, smart work, smart
management and autonomous driving etc.
Not only image recognition and visualization, but also XR that integrates virtual reality (VR), augmented reality(AR), mixed reality (MR) is also necessary to make cyber space more familiar. In addition to visual effects,
cross-modal sensory presentation that appeals to the human senses is emphasized. Therefore, technologicalinnovation in computer graphics, computer vision, user interface, user experience, etc., which form thesetechnological foundations, is important, and practical application of technology that appeals not only to vision but
also to other senses through images and video.
In this special issue, we look forward to receiving your papers, system development papers, and data papers thatwill realize a future society through images and video.
1. Topics covered include but not limited to
VR, AR, MR, Computer graphics, Image processing, Interaction, Realtime processing, Cross-modal sensory,
Computer vision, Machine learning Image analysis, Object detection, Image recognition, User interface, User
experience
2. Treatment of papers
Submission paper style format and double-blind peer review process are the same as an ordinary contributed
paper. If the number of accepted papers is less than the minimum number for the special issue, the acceptancepaper will be published as an ordinary contributed paper. We ask for your understanding and cooperation.
3. Publication of Special Issue:
IEEJ Transactions on Image Electronics and Visual Computing Vo.9, No.1 (June 2021)
4. Submission Deadline
Friday, October 30, 2020
5. Contact details for Inquires:
IIEEJ Office E-mail: hensyu@iieej.org
6. Online Submission URL: http://www.editorialmanager.com/iieej/

153

Revised: January 6, 2017
Revised: July 6, 2018

Guidance for Paper Submission
1.

Submission of Papers(1) Preparation before submission
・ The authors should download “Guidance for Paper Submission” and “Style Format” from the“Academic Journals”, “English Journals” section of the Society website and prepare the paperfor submission.
・ Two versions of “Style Format” are available, TeX and MS Word. To reduce publishing costs
and effort, use of TeX version is recommended.
・ There are four categories of manuscripts as follows: Ordinary paper: It should be a scholarly thesis on a unique study, development orinvestigation concerning image electronics engineering. This is an ordinary paper topropose new ideas and will be evaluated for novelty, utility, reliability andcomprehensibility. As a general rule, the authors are requested to summarize a paper
within eight pages.
 Short paper: It is not yet a completed full paper, but instead a quick report of the partial
result obtained at the preliminary stage as well as the knowledge obtained from the said
result. As a general rule, the authors are requested to summarize a paper within four
pages.
 System development paper: It is a paper that is a combination of existing technology or it
has its own novelty in addition to the novelty and utility of an ordinary paper, and thedevelopment results are superior to conventional methods or can be applied to othersystems and demonstrates new knowledge. As a general rule, the authors are requested
to summarize a paper within eight pages.
 Data Paper: A summary of data obtained in the process of a survey, product development,
test, application, and so on, which are the beneficial information for readers even though
its novelty is not high. As a general rule, the authors are requested to summarize a paper
within eight pages.
・ To submit the manuscript for ordinsry paper, short paper, system development paper, or data
paper, at least one of the authors must be a member or a student member of the society.
・ We prohibit the duplicate submission of a paper. If a full paper, short paper, systemdevelopment paper, or data paper with the same content has been published or submitted toother open publishing forums by the same author, or at least one of the co-authors, it shallnot be accepted as a rule. Open publishing forum implies internal or external books,
magazines, bulletins and newsletters from government offices, schools, company
organizations, etc. This regulation does not apply to a preliminary draft to be used at anannual meeting, seminar, symposium, conference, and lecture meeting of our society or othersocieties (including overseas societies). A paper that was once approved as a short paper andbeing submitted again as the full paper after completion is not regarded as a duplicate
submission.
(2) Submission stage of a paper・ Delete all author information at the time of submission. However, deletion of reference
information is the author’s discretion.
・ At first, please register your name on the paper submission page of the following URL, andthen log in again and fill in the necessary information. Use the “Style Format” to upload yourmanuscript. An applicant should use PDF format (converted from dvi of TeX or MS Word154

format) for the manuscript. As a rule, charts (figures and tables) shall be inserted into themanuscript to use the “Style Format”. (a different type of data file, such as audio and video,
can be uploaded at the same time for reference.)
http://www.editorialmanager.com/iieej/
・ If you have any questions regarding the submission, please consult the editor at our office.
Contact:
Person in charge of editing
The Institute of Image Electronics Engineers of Japan
3-35-4-101, Arakawa, Arakawa-Ku, Tokyo 116-0002, Japan
E-mail: hensyu@iieej.org
Tel: +81-3-5615-2893, Fax: +81-3-5615-2894

2.

Review of Papers and Procedures(1) Review of a paper・ A manuscript is reviewed by professional reviewers of the relevant field. The reviewer willdeem the paper “acceptance”, “conditionally acceptance” or “returned”. The applicant is
notified of the result of the review by E-mail.
・ Evaluation method
Ordinary papers are usually evaluated on the following criteria:
 Novelty: The contents of the paper are novel.
 Utility: The contents are useful for academic and industrial development.
 Reliability: The contents are considered trustworthy by the reviewer.
 Comprehensibility: The contents of the paper are clearly described and understood by
the reviewer without misunderstanding.
Apart from the novelty and utility of an ordinary paper, a short paper can be evaluated byhaving a quickness on the research content and evaluated to have new knowledge withresults even if that is partial or for specific use.
System development papers are evaluated based on the following criteria, apart from thenovelty and utility of an ordinary paper.
 Novelty of system development: Even when integrated with existing technologies, thenovelty of the combination, novelty of the system, novelty of knowledge obtained fromthe developed system, etc. are recognized as the novelty of the system.
 Utility of system development: It is comprehensively or partially superior compared tosimilar systems. Demonstrates a pioneering new application concept as a system. The
combination has appropriate optimality for practical use. Demonstrates performance
limitations and examples of performance of the system when put to practical use.
Apart from the novelty and utility of an ordinary paper, a data paper is considered novel if
new deliverables of test, application and manufacturing, the introduction of new technology
and proposals in the worksite have any priority, even though they are not necessarily
original. Also, if the new deliverables are superior compared to the existing technology andare useful for academic and industrial development, they should be evaluated.
(2) Procedure after a review
・ In case of acceptance, the author prepares a final manuscript (as mentioned in 3.).
・ In the case of acceptance with comments by the reviewer, the author may revise the paper inconsideration of the reviewer’s opinion and proceed to prepare the final manuscript (as155mentioned in 3.).
・ In case of conditional acceptance, the author shall modify a paper based on the reviewer’s
requirements by a specified date (within 60 days), and submit the modified paper for
approval. The corrected parts must be colored or underlined. A reply letter must be attachedthat carefully explains the corrections, assertions and future issues, etc., for all of theacceptance conditions.
・ In case a paper is returned, the author cannot proceed to the next step. Please look at thereasons the reviewer lists for the return. We expect an applicant to try again after reviewing
the content of the paper.
(3) Review request for a revised manuscript
・ If you want to submit your paper after conditional acceptance, please submit the reply letterto the comments of the reviewers, and the revised manuscript with revision history to thesubmission site. Please note the designated date for submission. Revised manuscripts
delayed more than the designated date be treated as new applications.
・ In principle, a revised manuscript willl be reviewed by the same reviewer. It is judged either
accceptance or returned.
・ After the judgment, please follow the same procedure as (2).

3.

Submission of final manuscript for publication(1) Submission of a final manuscript
・ An author, who has received the notice of “Acceptance”, will receive an email regarding thecreation of the final manuscript. The author shall prepare a complete set of the final
manuscript (electronic data) following the instructions given and send it to the office by thedesignated date.
・ The final manuscript shall contain a source file (TeX edition or MS Word version) and a PDF
file, eps files for all drawings (including bmp, jpg, png), an eps file for author’s photograph(eps or jpg file of more than 300 dpi with length and breadth ratio 3:2, upper part of the body)for authors’ introduction. Please submit these in a compressed format, such as a zip file.
・ In the final manuscript, write the name of the authors, name of an organizations,introduction of authors, and if necessary, an appreciation acknowledgment. (cancel macros in
the Style file)・ An author whose paper is accepted shall pay a page charge before publishing. It is theauthor’s decision to purchase offprints. (ref. page charge and offprint price information)(2) Galley print proof
・ The author is requested to check the galley (hard copy) a couple of weeks before the paper ispublished in the journal. Please check the galley by the designated date (within one week).
After making any corrections, scan the data and prepare a PDF file, and send it to our officeby email. At that time, fill in the Offprint Purchase Slip and Copyright Form and return thescanned data to our office in PDF file form.
・ In principle, the copyrights of all articles published in our journal, including electronic form,
belong to our society.
・ You can download the Offprint Purchase Slip and the Copyright Form from the journal on our
homepage. (ref. Attachment 2: Offprint Purchase Slip, Attachment 3: Copyright Form)(3) Publication
・ After final proofreading, a paper is published in the Academic journal or English transaction(both in electronic format) and will also be posted on our homepage.

156

IIEEJ Trans. on Image Electronics and Visual Computing©2019 IIEEJ

Vol. 7, No. 2 2019

Editor in Chief: Mei Kodama
The Institute of Image Electronics Engineers of Japan
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan