IIEEJ Transactions on
IIEEJ Transactions on
Image Electronics and Visual Computing

Image Electronics andVisual Computing

Special Issue on Journal Track Papers in IEVC2024

Vol. 12, No. 1 2024

I.I.E.E.J

Editorial Committee of IIEEJ
Editor in Chief
Osamu UCHIDA (Tokai University)

Vice Editors in Chief
Naoki KOBAYASHI (Saitama Medical University)
Yuriko TAKESHIMA (Tokyo University of Technology)
Masahiro ISHIKAWA（Kindai University）

Advisory Board
Yasuhiko YASUDA (Waseda University Emeritus)
Hideyoshi TOMINAGA (Waseda University Emeritus)
Kazumi KOMIYA (Kanagawa Institute of Technology)
Fumitaka ONO (Tokyo Polytechnic University Emeritus)
Yoshinori HATORI (Tokyo Institute of Technology)
Mitsuji MATSUMOTO (Waseda University Emeritus)
Kiyoshi TANAKA (Shinshu University)
Shigeo KATO (Utsunomiya University Emeritus)
Mei KODAMA (Hiroshima University)

Editors
Yoshinori ARAI (Tokyo Polytechnic University)
Chee Seng CHAN (University of Malaya)
Naiwala P. CHANDRASIRI (Kogakuin University)
Chinthaka PREMACHANDRA (Shibaura Institute of Technology)
Makoto FUJISAWA(University of Tsukuba)
Issei FUJISHIRO (Keio University)
Kazuhiko HAMAMOTO (Tokai University)
Madoka HASEGAWA (Utsunomiya University)
Ryosuke HIGASHIKATA (FUJIFILM Business Innovation Corp.)Yuki
IGARASHI (Ochanomizu University)
Takashi IJIRI（Shibaura Institute of Technology）
Mitsuo IKEDA（Shikoku University）
Tomokazu ISHIKAWA（Toyo University）
Naoto KAWAMURA (Canon OB)
Shunichi KIMURA (FUJIFILM Business Innovation Corp.)
Shoji KURAKAKE (NTT DOCOMO)
Kazuto KAMIKURA (Tokyo Polytechnic University)
Takashi KANAI (The University of Tokyo)
Tetsuro KUGE (NHK Engineering System, Inc.)
Takafumi KOIKE（Hosei University）
Koji MAKITA (Canon Inc.)
Tomoaki MORIYA (Tokyo Denki University)
Paramesran RAVEENDRAN (University of Malaya)
Kaisei SAKURAI (DWANGO Co., Ltd.)
Koki SATO (Shonan Institute of Technology)
Syuhei SATO（Hosei University）
Masanori SEKINO (FUJIFILM Business Innovation Corp.)
Kazuma SHINODA (Utsunomiya University)
Mikio SHINYA (Toho University)
Shinichi SHIRAKAWA (Aoyama Gakuin University)
Kenichi TANAKA (Nagasaki Institute of Applied Science)
Yukihiro TSUBOSHITA (Fuji Xerox Co., Ltd.)
Daisuke TSUDA (Shinshu University)
Masahiro TOYOURA (University of Yamanashi)
Kazutake UEHIRA (Kanagawa Institute of Technology)
Yuichiro YAMADA (Genesis Commerce Co.,Ltd.)
Hiroshi YOSHIKAWA (Nihon University)
Norimasa YOSHIDA (Nihon University)
Toshihiko WAKAHARA (Fukuoka Institute of Technology OB)
Kok Sheik WONG (Monash University Malaysia)

Reviewer
Hernan AGUIRRE (Shinshu University)
Kenichi ARAKAWA (NTT Advanced Technology Corporation)
Shoichi ARAKI (Panasonic Corporation)
Tomohiko ARIKAWA (NTT Electronics Corporation)
Yue BAO (Tokyo City University)
Nordin BIN RAMLI (MIMOS Berhad)
Yoong Choon CHANG (Multimedia University)
Robin Bing-Yu CHEN (National Taiwan University)
Kiyonari FUKUE (Tokai University)
Mochamad HARIADI (Sepuluh Nopember Institute of Technology)

Masaki HAYASHI (UPPSALA University)
Takahiro HONGU (NEC Engineering Ltd.)
Yuukou HORITA (University of Toyama)
Takayuki ITO (Ochanomizu University)
Masahiro IWAHASHI (Nagaoka University of Technology)
Munetoshi IWAKIRI (National Defense Academy of Japan)
Yoshihiro KANAMORI (University of Tsukuba)
Shun-ichi KANEKO (Hokkaido University)
Yousun KANG (Tokyo Polytechnic University)
Pizzanu KANONGCHAIYOS (Chulalongkorn University)
Hidetoshi KATSUMA (Tama Art University OB)
Masaki KITAGO (Canon Inc.)
Akiyuki KODATE (Tsuda College)
Hideki KOMAGATA (Saitama Medical University)
Yushi KOMACHI (Kokushikan University)
Toshihiro KOMMA (Tokyo Metropolitan University)
Tsuneya KURIHARA (Hitachi, Ltd.)
Toshiharu KUROSAWA (Matsushita Electric Industrial Co., Ltd. OB)
Kazufumi KANEDA (Hiroshima University)
Itaru KANEKO (Tokyo Polytechnic University)
Teck Chaw LING (University of Malaya)
Chu Kiong LOO (University of Malaya) F
Xiaoyang MAO (University of Yamanashi)
Koichi MATSUDA (Iwate Prefectural University)
Makoto MATSUKI (NTT Quaris Corporation OB)
Takeshi MITA (Toshiba Corporation)
Hideki MITSUMINE (NHK Science & Technology Research Laboratories)
Shigeo MORISHIMA (Waseda University)
Kouichi MUTSUURA (Shinsyu University)
Yasuhiro NAKAMURA (National Defense Academy of Japan)
Kazuhiro NOTOMI (Kanagawa Institute of Technology)
Takao ONOYE (Osaka University)
Hidefumi OSAWA (Canon Inc.)
Keat Keong PHANG (University of Malaya)
Fumihiko SAITO (Gifu University)
Takafumi SAITO (Tokyo University of Agriculture and Technology)
Tsuyoshi SAITO (Tokyo Institute of Technology)
Machiko SATO (Tokyo Polytechnic University Emeritus)
Takayoshi SEMASA (Mitsubishi Electric Corp. OB)
Kaoru SEZAKI (The University of Tokyo)
Jun SHIMAMURA (NTT)Tomoyoshi SHIMOBABA (Chiba University)
Katsuyuki SHINOHARA (Kogakuin University)
Keiichiro SHIRAI (Shinshu University)
Eiji SUGISAKI (N-Design Inc. (Japan), DawnPurple Inc.(Philippines))
Kunihiko TAKANO (Tokyo Metropolitan College of Industrial Technology)
Yoshiki TANAKA (Chukyo Medical Corporation)
Youichi TAKASHIMA (NTT)Tokiichiro TAKAHASHI (Tokyo Denki University)
Yukinobu TANIGUCHI (NTT)Nobuji TETSUTANI (Tokyo Denki University)
Hiroyuki TSUJI (Kanagawa Institute of Technology)
Hiroko YABUSHITA (NTT)
Masahiro YANAGIHARA (KDDI R&D Laboratories)
Ryuji YAMAZAKI (Panasonic Corporation)IIEEJ OfficeOsamu UKIGAYA
Rieko FUKUSHIMA
Kyoko HONDA

Contact Information
The Institute of Image Electronics Engineers of Japan (IIEEJ）
3-35-4-101, Arakawa, Arakawa-ku, Tokyo 116-0002, Japan
Tel : +81-3-5615-2893 Fax : +81-3-5615-2894
E-mail : hensyu@iieej.org
http://www.iieej.org/ (in Japanese)
http://www.iieej.org/en/ (in English)
http://www.facebook.com/IIEEJ (in Japanese)
http://www.facebook.com/IIEEJ.E (in English)

IIEEJ Transactions on
Image Electronics and Visual Computing
Vol.12 No.1 June 2024
CONTENTS

Special Issue on Journal Track Papers in IEVC2024
1

Upon the Special Issue on Journal Track Papers in IEVC2024

Osamu UCHIDA

Contributed Papers
2Classification of White Blood Cells Using YOLOv7: Single and Cascade
Classification Approaches Applied to Images Segmented by CellaVisionTM DM96

Thalita Munique COSTA, Yoko USAMI, Mai IWAYA,
Yuka TAKEZAWA, Yuika NATORI, Hernan AGUIRRE,
Kiyoshi TANAKA

15

VigNet: Semiautomatic Generation of Vignette Illustrations from Video

Mayu NAMAI , Issei FUJISHIRO

23

Adaptive Selection of Auxiliary Tasks Using Deep Reinforcement Learning
for Video Game Strategy

Hidenori ITAYA, Tsubasa HIRAKAWA,
Takayoshi YAMASHITA, Hironobu FUJIYOSHI

32

Real-Time Intuitive Interaction and Realistic Illumination for CT VolumeRendering

Kousuke KATAYAMA, Toru HIGAKI, Kazufumi KANEDA,
Bisser RAYTCHEV,Wataru FUKUMOTO, Hidenori MITANI40

An Examination of Motion Analysis During Weight Illusion by Impression
Change of the Own Body

Joichiro MURAOKA, Kosei TOMIOKA, Yusei MURAISHI,
Naoki HASHIMOTO, Mie SATO

48

Smartphone-Based Continuous Authentication Based on Flick Input Features
Using Japanese Free Text

Shuto KINOSHITA, Yasushi YAMAZAKI

Announcements
56

Call for Papers : Special Issue on Image Electronics Technologies Related to
VR/AR/MR/XR

Guide for Authors
57

Guidance for Paper Submission

Published two times a year by the Institute of Image Electronics Engineers of Japan （IIEEJ）
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan
Tel：+81-5615-2893 Fax：+81-5615-2894 E-mail：hensyu@iieej.org http://www.iieej.org/

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Upon the Special Issue on
Journal Track Papers in IEVC2024
Editor: Osamu UCHIDA(Tokai University)
The 8th International Conference on Image Electronics and Visual Computing(IEVC2024) was held in Tainan City, Taiwan, on March 11-14, 2024, as the international
academic event of Image Electronics Engineers of Japan (IIEEJ). It was based on the great
success of the previous seven workshops/conferences in 2007 (Cairns, Australia), 2010
(Nice, France), 2012 (Kuching, Malaysia), 2014 (Koh Samui, Thailand), 2017 (Da Nang,
Vietnam), 2019 (Bali, Indonesia), and 2021(Online). The conference aims to bring together
researchers, engineers, developers, and students from various fields in academia andindustry to discuss the latest studies, standards, developments, implementations, andapplication systems in all image electronics and visual computing areas.
As in previous IEVCs, IEVC2024 has two paper categories: general paper and latebreaking paper (LBP). Also, as in the past two IEVCs, the general paper category has two
tracks: Journal track (JT) and Conference track (CT). The journal track offers authors theadvantage of having their papers publish in the special issue on Journal Track Papers in
IEVC2024, scheduled for the June 2024 issue of IIEEJ Transactions on Image Electronics
and Visual Computing.
This special issue contains 6 papers that were accepted within the publication scheduleof the June 2024 issue, and the second special issue on JT papers in IEVC2024 is alsoplanned for the coming issues. All the papers are of high-level content and will contributeto further developing the field of image electronics and visual computing.
Before concluding, I would like to express my heartfelt appreciation to all the reviewers
and editors. Your dedication and expertise have been instrumental in improving the qualityof the papers. I would also like to extend my deepest gratitude to the members of theeditorial committee of IIEEJ and the staff at the IIEEJ office for their invaluable support.

1

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）-- Special Issue on Journal Track Papers in IEVC2024 --Classification of White Blood Cells Using YOLOv7: Single and Cascade Classification
Approaches Applied to Images Segmented by CellaVisionTM DM96
Thalita Munique COSTA†, Yoko USAMI††, Mai IWAYA††, Yuka TAKEZAWA††, Yuika NATORI††,
Hernan AGUIRRE‡, Kiyoshi TANAKA‡ (Honorary Member)† Graduate School of Medicine, Science and Technology, Shinshu University, Japan,†† Department of Laboratory Medicine, Shinshu University Hospital, Japan,‡ Academic Assembly (Institute of Engineering), Shinshu University, Japan<Summary> A common task executed in the medical routine is the identification, classification, quantification, and analysisof white blood cells from peripheral blood, which is commonly done with the help of automatic counters. Some of the mostpopular machines present low accuracy and commit relevant mistakes in the classification of the cells. In this work, we proposeand discuss the use of the deep learning architecture YOLOv7 in the reclassification of blood cell images segmented by themachine CellaVision TM DM96 into 11 classes, i.e., Band Neutrophil, Segmented Neutrophil, Basophil, Eosinophil,
Erythroblast, Thrombocyte, Lymphocyte, Lymphocyte Variant, Metamyelocyte, Monocyte, and Myelocyte, in single andcascade classification methods. The classification made by CellaVision TM DM96 achieved an accuracy of 76.20%, precision of
80.93%, recall of 92.87%, and F1-Score of 86.49%. The single classification method presented a mean accuracy of 93.59%,
precision of 96.16%, recall of 97.23%, and F1-Score of 96.69%. The Cascade method resulted in a mean accuracy of 93.85%,
precision mean of 96.81%, a recall of 97.23%, and F1-Score of 96.83% for the same evaluation database. Both methods proved
effective in increasing the performance in blood images classification and, mainly the cascade method, reduced the rate of more
relevant mistakes.
Keywords: digital pathology, white blood cells, YOLOv7, deep learning

1.

Autoimmune diseases and immune deficiencies areexamples of diseases diagnosed by the WBC count2), buteach pathological agent modifies the number of each WBC
type differently. Because of that, counting the number ofcells of each type and identifying abnormal cells is also
crucial for diagnosis. Metabolic disorders, bacteria, andfungi are examples that increase the number of neutrophils.
Hepatitis, viruses, and leukemia are situations that increase
the number of lymphocytes while HIV rubeola, andchickenpox reduce the quantity. Listeriosis, malaria,
bacterial and viral infection can change the number ofmonocytes. Allergic diseases and parasites are examples thatchange the number of eosinophils and malignant
myeloproliferative diseases, and inflammation can change
the number of basophils found in the bloodstream2).
Because of their shape, some other cells are commonly
wrongly classified as WBCs by automated counters.
Erythroblasts and Giant Thrombocytes are examples ofthese cells. Even though they are not considered WBCs, theanalysis of thrombocytes and erythroblasts is also a step
performed in the diagnosis of diseases.

Introduction

In clinical laboratories, the identification andclassification of blood cells, mainly white blood cells (i.e.,WBCs) is a common task. The number of WBCs in asample, the quantity of each type of WBC, and theidentification of abnormal cells are important information
used in the diagnosis 1). WBCs are the group of nucleatedblood cells that play the most significant role in the defenseof the human body against diseases, acting on immunity andeliminating viruses, bacteria, parasites, and fungi2).
There are three main types of WBCs: monocytes,
lymphocytes, and granulocytes. Granulocytes are further
divided into neutrophils, eosinophils, and basophils.
Neutrophils can be classified, based on maturity level, into
band neutrophils, segmented neutrophils, myelocytes, andmetamyelocytes. Metamyelocytes and myelocytes are rarely
seen in the bloodstream but can be found in cases of severe
infections. WBC disorders can be quantitative or qualitative.
In qualitative defects, abnormal cells are present incirculation. Variant lymphocytes (i.e., variant) are abnormal
lymphocyte cells1).

2

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig. 1 Cell images obtained from CellaVision: a) Basophil, b) Band Neutrophil, c) Eosinophil, d) Erythroblast e) Thrombocyte, f) Lymphocyte,g) Metamyelocyte, h) Monocyte, i) Myelocyte, j) Segmented Neutrophil, k) Variant Lymphocyte, l) Image outside the proposed classification

The human classification of WBCs is a task that
consumes a lot of time and energy for blood-specialized
professionals and, even a very well-trained professional, cancommit mistakes in a very busy and tiring routine. For thesereasons, automated counters are widely used for theidentification and classification of WBCs since they bringhigher accuracy, consistency in results, and speed overmanual techniques but, due to the biologically complex
nature of cells, there isn't a single optimal method
supporting the diagnosis in the laboratories3).
CellaVision DM96TM 4) (i.e., CellaVision) is an
automated image analysis system for peripheral bloodsmears, widely used in laboratories, which scan glass slides,
identifies potential WBCs, take digital images with highmagnification, and, using a neural network algorithm,
outputs images of the segmented WBCs and a class label ona customizable computer display for confirmation orreclassification. Figure 1 shows images obtained by
CellaVision. CellaVision can identify not just healthy WBCs
but also abnormal WBCs and nucleated Red Blood Cells5).
As also happens in other automated counters, the classes
attributed to the cells by this system are often wrong. The
machine offers important support in the blood analysis,segmenting each WBC in a different image, but, due to thehigh error rate in the classification, blood specialists must
reassess all the segmented images and perform thereclassification of these cells, which is a heavy burden in
medical routine. To solve this problem, deep learningalgorithms can be a useful tool in the reclassification ofthese images with a higher accuracy so that manual
reclassification is no longer necessary.
Neural Networks and Deep Learning merged trying tosimulate the human brain neural network process of learningand memorizing. Deep Learning algorithms consist of amulti-layered neural network architecture and, after being

trained with a large amount of data, can perform
identifications and classifications traditionally performed
manually6). YOLOv77) is a deep learning-based object
detection model that stands out, among other reasons, for its
accuracy and detection speed.
This work studies the use of the deep learning
architecture YOLOv7 as a tool to support the classificationof WBC images segmented by CellaVision to increase theaccuracy of the system and reduce the number of more
relevant mistakes to support the diagnosis of diseases. In thiscontext, we propose and evaluate the single classification
method and the cascade classification method to classify theimages into 11 classes (i.e., Basophil, Band, Eosinophil,
Erythroblast, Thrombocyte, Lymphocyte, Metamyelocyte,
Monocyte, Myelocyte, Segmented and Variant) separating
the images in respective folders and creating an extra folder
called Unclassified for the images that could not beclassified with a high class-confidence value.

2. Literature Review
Kratz et al.5) evaluated the clinical performance ofCellaVision against manual microscopy. In the experiments
performed, CellaVision achieved 82% accuracy. The correctclassification rate was higher for mature than for immature
and abnormal cells and the highest coefficients wereobtained for segmented neutrophils and lymphocytes while
the lowest were obtained for basophils and eosinophils. The
motivation of our work is to propose a solution for
CellaVision’s low accuracy, so the specialists don’t have toreclassify all the images segmented by the system.
A paper about the WBC classification was written by
Hedge et al.3). The authors compare the classification madeby traditional image processing and using convolutional
neural networks trained for 1,000 epochs. The authors used
1,418 cropped images to build tools to classify images into 6

3

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

perform the classification of 5 types of WBCs. Only thenormal types of WBCs are used in the database and theclassification as well. The paper's main discussion is theconsequence of using two different neural networkarchitectures combined to make the classification. The
approach used by Akalin obtained 98% accuracy in the
hybrid approach and 83.3% and 94.66% using YOLOv5
with 2 different types of activation functions. Akalin’s paper
doesn’t use a different evaluation dataset, the metrics
discussed are the ones for the images used during thetraining of the neural network, so the method cannot be
precisely evaluated.
To the best of our knowledge, no previous work wasdone with the goal of classifying WBCs from CellaVision
into 11 classes, including abnormal cells. In the literature, it
is most common the classification by 5 classes of normalWBCs, however, it is crucial to consider the abnormal cells
in the database and classify into more precise classes foraiding the diagnosis in the laboratories. Also, since the deeplearning architecture YOLO has been developing day by day,the authors solve this problem using the latest version,
YOLOv7, launched in 2022, in this work. Furthermore, we
evaluate the obtained results not only in terms of accuracy
but also from a clinical point of view analyzing the impactof the mistakes for the diagnosis, while other works don’t
evaluate the results from this point of view.

classes, i.e., lymphocytes, monocytes, neutrophils,
eosinophils, basophils, and abnormal cells. As
pre-processing the images were cropped to segment just thecell. They obtained 99% accuracy with full training CNN
and got even better results, 99.8% using the classifier on
hand-crafted features.
Kutlu et al.2) evaluated the use of neural networks for
WBC classification into 5 classes (i.e., Lymphocytes,
Monocytes, Eosinophils, Basophils, and Neutrophils). Using
the neural network architecture ResNet50 with a transfer
learning method, and a database with 5,000 images (1,000
images of each class), an accuracy of 97%, recall of 99%,
and precision of 97% was obtained for the threshold of 0.3,
from training the neural network 3,000 epochs.
Shahin et al.8) used two different transfer learning
methods, fine-tuning AlexNet neural network and deep
activation features for training 2,551 images, including 5
healthy WBC types. The accuracy obtained was 96.1%.
Some versions of the YOLO series were used in the
WBC classification. Abass et al. 9) used YOLOv2 to identify
and classify 3 types of WBC. The goal was to support theleukemia diagnosis. The neural network was used in two
steps to first segment and later classify the images, and theyobtained an accuracy of 94.3%. The deep learning
architecture used, YOLOv2, was launched in 2017.
Praveen at. al. 10) used YOLOv3 to classify 4 types ofWBCs. The process was divided into detection andclassification. The method achieved an accuracy of 90%.
Their dataset doesn’t consider abnormal cases. The deeplearning architecture used, YOLOv3, was launched in 2018.
Akalin et al.11) used YOLOv5, launched in 2021, to

3. Deep Learning and YOLOv7
Artificial neural networks simulate the human brain’s
process of learning. An artificial neural network, like thehuman neural network, is composed of connected neurons.

Fig. 2 YOLOv7 architecture

4

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

The information passed from one neuron to another istransformed according to a parameter called “weight”. These
weights are modified in the training process. Deep learning
architectures are neural networks with multiple layers6).
Convolutional neural networks are deep learning
architectures inspired by visual perception, capable of object
detection and classification in images12). The use of
convolutional neural networks stands out compared to
traditional image-processing methods mainly because of thehigh performance, but also for the characteristics of theprocess. For the neural network training it's not necessary
any pre-processing or pre-feature extraction and selection.
The features are extracted by the convolutional layers2).
The deep learning architecture used in this work was
YOLOv7 that is the 7th version of the YOLO algorithm,
released in 2022, and increased accuracy, speed (can achieve
600fps in the detection), and ability to detect a wider rangeof objects (i.e., high performance detect either small,medium or big objects) compared to the previous versions of
YOLO and some of the most used deep learning algorithms.
YOLOv7, like some other object detectors, performs image
recognition by predicting bounding boxes and classprobabilities for the objects in the image, using a
convolutional neural network to extract features from theimage. YOLOv7 architecture is represented in Fig. 2.
YOLO frameworks have three main components, thebackbone, the head, and the neck. The backbone mainly
extracts essential features of an image and feeds them to thehead through the neck. The neck collects feature maps
extracted by the backbone and creates feature pyramids.
Finally, the head consists of output layers that have final
detections. The computational block used in YOLOv7
architecture is called Extended Efficient Layer Aggregation
Network (i.e., E-ELAN) and allows better learning of themodel by expanding, shuffling, and merging cardinality.
The training process is optimized with an architecture called“Bag of Freebies”. Another optimization is Deep
Supervision Label Assignment which adds auxiliary heads
in the middle layers to help the model converge well7).
YOLOv7 comprehends the desired characteristics for thisstudy as high accuracy and speed compared to other object
detectors, prediction of bounding boxes, the possibility of
training many classes at the same time, and training images
even bigger than the original size (i.e., 360×363pixels).

Fig. 3 Schematic Diagram of the methodology

Fig. 4 Single classification method. The images are classified into
11 classes and separated into respective folders. An extra
folder called Unclassified is also created for unclassified
imagesinto the CellaVision machine that outputs WBC images
segmented but containing many misclassifications.
Considering the low classification accuracy by CellaVisionand the importance of classifying normal and abnormal cells,
we propose the reclassification of these images using
YOLOv7 to increase the accuracy in single and cascadeclassification methods. A detailed description of bothmethods is presented in the following subsections.
4.1 Single classification method

4. Proposed Methods

In the single classification method, represented in Fig. 4,
only one folder is prepared with all the images for the
training dataset, subdivided into train (i.e., 80%) andAn overview of the methodology is shown in the
Schematic Diagram in Fig. 3. The blood samples are input

5

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

validation (i.e., 20%). The images in the subfolders arechanged for each training in the five-fold cross-validation.
The deep learning architecture YOLOv7 was trained for theclassification of the images into 11 classes (i.e., band,
segmented, basophil, eosinophil, erythroblast, thrombocyte,
lymphocyte, variant, metamyelocyte, monocyte, andmyelocyte). With the trained weights, the neural network
can classify WBCs in images never seen by the neuralnetwork before, drawing a bounding box around theseobjects, and presenting the class with its class confidence
value. If there is more than one detection in an image, only
the detection with higher class confidence is considered. The
classification algorithm creates 11 folders for each class andthe images with the cells already classified and tagged, aresaved in the respective folders. An extra folder called“Unclassified” is also created to store the images that haveno detections with a high class confidence value.
The official training scripts of YOLOv7 were notmodified but for the detection and classification, a modified
script had to be prepared. In the case of two cells identified
in the same image, the scripts consider only the detection
with a higher class confidence value. The single
classification script creates 12 folders (i.e., one for eachclass and one for unclassified images), runs the detection,and saves the images into the respective folders, sending theimages with no cell detection or detection with classconfidence value lower than 0.7 into the folder“Unclassified”.

images are classified in two phases using 4 different trained
weights set. The training dataset for cascade classification iscomposed of 4 folders, one for the first phase of theclassification and 3 for the second phase. First, the cells will
be classified between the 5 classes of WBCs (i.e., basophil,eosinophil, lymphocyte, monocyte, and neutrophil) and an
extra class called "Not WBCs" that represent images withother types of cells present in the database. The
classification algorithm creates separate folders and saves
the classified images into different folders. In each one ofthese folders, the classification of subclasses is performed.
The images classified as lymphocyte are reclassified into
lymphocyte and variant, the images classified as neutrophils
are reclassified into band, metamyelocyte, myelocyte, andsegmented, and in the images classified as not WBCs,
erythroblast, and thrombocyte are also reclassified. If there
is more than one detection in an image, only the detection
with higher class confidence is considered. 11 folders are
created to store images from each one of the classes, and the“Unclassified” folder is created to store the images that haveno detections with high class confidence values.
The flowchart for the cascade classification detection
script is shown in Fig. 6. The script first creates 6 folders(i.e., one for each WBC type and a not WBC folder), andinside these folders create the respective subfolders. The
images are first classified into the 6 groups using the 1st
layer trained weights, then all the images stored in thefolders with subclassifications are reclassified by the 2nd
layer trained weights for their cell type and the images are
moved to the subclass folder.

4.2 Cascade Classification Method
For the cascade classification, represented in Fig. 5, theFig. 5 Cascade method. First, the images are classified into 6 classes representing the 5 types of white blood cells images and another filefor other images. In the second phase, images are reclassified into subclasses and separated into other folders, resulting in 11 classfolders plus one extra folder for images outside the 11 defined classes.

6

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

and Erythroblast cells are not WBCs but have clinicalrelevance and the number of these images segmented by
CellaVision is very high, so it's also necessary to classify
and separate these images into different folders inside thegroup of cells not classified as WBCs so these cells also
don’t have to be reassessed by the professionals.

5.

Experimental Setup

5.1 Implementation
The computer used for the experiments has an Intel(R)Core (TM) i9-10900F processor, 32GB RAM, and a video
card NVIDIA GeForce RTX 3090. The application codes
were written in Python 3.9.
The images obtained from CellaVision have 360×363
pixels so, for the training, the image size set in thehyperparameters was 384 ×384, which was the closest
possible value. The weights of the neural network weretrained separately for each classification for 3,000 epochs.
The weights were saved for each epoch, so the best weights
set, i.e., the ones with higher accuracy, precision, mAP@0.5,
and mAP@0.5:0.95 scores, were selected.
After the training, an F1-Score graph was plotted withthe class confidence values. By the graph, it is possible toanalyze the F1-Score result for each possible classconfidence value. We set a threshold value of 0.7 for all
detections. This value got high results in the F1-Score curve
for all the trainings and was chosen also because for theapplication is important that the cells are classified only ifthey have a high class confidence to guarantee the reliabilityof the system. For each method, five-fold cross-validation
was performed, resulting in information about average andhighest accuracy.

Fig. 6 Flowchart of the detection algorithm in cascadeclassification method

4.3 “Unclassified” folder
Sometimes, in the medical routine, abnormal cells very
different from the ones more commonly seen are found in
the blood samples. These cells for their different aspects areoften not correctly classified by the machines, but theiranalysis is very important once it can happen in cases of rarediseases. Cases of problems in the sample preparation or theimage scanning also can make the classification process by
the systems difficult. It is important in the development of
the system to consider that only images with high classconfidence value are classified so these images don’t have tobe reassessed by specialists. A folder called “Unclassified”
was created to store these images that could not be classified
with the minimum class confidence value set for no of the
WBC types, so the specialists only must access this folder to
identify the unusual abnormal cases. Giant Thrombocytes

5.2

Database

The database includes healthy WBCs, abnormal WBCs,
and other cells with clinical interest commonly mistaken forWBCs. The dataset used comprehends 5850 blood cell
images of size 360 × 363 pixels, from blood samples
collected between 2018 and 2019 in Shinshu University
Hospital. The data comprehends only blood images and theclass of each cell. No information from the patients wascollected, respecting the patient’s privacy. The images were
downloaded from the CellaVision system and classified by
two excellent blood analysis specialists. Each one of thespecialists analyzed the images alone and on different dates.
Only the images classified equally for both specialists wereused in this study.

7

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Images of 11 classes were obtained, i.e., band neutrophil,segmented neutrophil, basophil, eosinophil, erythroblast,
lymphocyte, variant lymphocyte, metamyelocyte, monocyte,
myelocyte, and thrombocyte. From these 5,850 images,
2,000 were separated for the evaluation of the method and3,850, comprehending a similar number of each class, wereused as training dataset.
For each image in all the training datasets, a text file wascreated, in a specific model for YOLO training, containing a
number that corresponds to the class, the position, and thesize of the bounding box that involves the cell presented in
the image. The same images were used for each neuralnetwork training in the single and the cascade classification
methods.
The 2,000 images validation dataset comprehends all theclasses but not with equal quantities. In this evaluation
dataset, specialists classed 269 (i.e., 13.45%) images as
neutrophil band, 576 (i.e., 28.8%) images as neutrophil
segmented, 101 (i.e., 5.05%) images as basophil, 97 (i.e.,
4.85%) images as eosinophil, 106 (i.e., 5.3%) images as
erythroblast, 117 (i.e., 5.85%) images as lymphocyte, 94 (i.e.,4.7%) images as lymphocyte variant, 143 (i.e., 7.15%)
images as metamyelocyte, 272 (i.e., 13.6%) images as
monocyte, 95 (i.e. 4.75%) images as myelocyte, and 130(i.e., 6.5%) images as thrombocyte. A blood sample contains
different amounts of each cell (e.g., the segmented
neutrophils amount is higher than the other blood cell types)
so the evaluation dataset has an adequate amount of eachclass but having a greater number of segmented neutrophils
represents better the performance that would be obtained ina real situation.

with the wrong classification, and false negatives (i.e., FN)
are the cells not classified or classified with a confidence
value below 0.7, i.e., images in the file “Unclassified”. As
the objective is to classify cell images already segmented by
CellaVision, all images used contain a cell and therefore thesystem does not have true negatives. The system only aims
to reclassify and separate the images into their respective
folders so the limits of the bounding boxes that identify thecells are not evaluated.

(1)
(2)
(3)
(4)

5.4 Five-fold Cross Validation
A five-fold Cross Validation13) based method was usedin the training of the neural network. The method appliedconsists of 5 trainings of the neural network. The training
dataset is randomly divided into 5 parts, containing a similar
number of images from each class, and, for each training, 1
different part (i.e., 20% of the dataset) is used for thevalidation of the training while the other 4 parts, (i.e., 80%of the dataset) is used in the training. With this method, it ispossible to obtain the mean error rate of the trainings, anerror rate value more consistent than the one obtained inonly one training. The database division is represented inFig. 7.

5.3 Metrics for evaluation
For the neural network training evaluation, the metrics
used are precision (i.e., P), which represents the percentage
of correct classifications, recall (i.e., R), which reflects thecapability of the neural network in finding correct items,
mAP@0.5, which is the mean average precision and can becalculated as the area under a precision-recall curve, andmAP@0.5:0.95, that considers the IoU (i.e., Area of overlap
divided by area of union) values from 50% to 95% at step of
5%. For the evaluation of the proposed method, besides P
and R, the accuracy (i.e., A), which represents the
proportion of the correct predictions, and F1-Score (i.e., F1),
that is the harmonic mean between precision and recall, wasalso used.
True positives (i.e., TP) represent the images correctlyclassified, False Positives (i.e., FP) are the cells identified

Fig. 7 Database representation for the 5 experiments foreach method

8

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

5.5 Transfer learning
Transfer learning is a method used in the training ofneural networks that consists of using weights trained for
different applications as a starting point for training thedesired application. This technique reduces training time andcan bring better results, as the neural network would alreadyhave a basic understanding of relevant objects andcharacteristics, such as edges, textures, and colors, among
others. The pre-trained weights used for Transfer Learningin this work are provided by the YOLOv7 authors in theofficial Yolov7 repository14) and were trained to detect 80
different classes (e.g., person, airplane, umbrella, orange, ...).

6.

Fig. 9 F1-Score graph for single classification method

Results and Discussion

The deep learning architecture YOLOv7 showedsatisfactory results in WBC classification. Figure 8 shows
an example of a WBC image classified and tagged by
YOLOv7. We show the details in the following.
In the training phase, the cascade method showed better
results compared to the single classification method, as we
can see comparing Fig. 9 and Fig. 10 obtained in training 1.
While the F1-Score for the single classification was 96%,
the F1-Score obtained in the cascade method was 97.7%.
Analyzing F1-Score we are, at the same time, analyzing thecapability of the neural network in finding correct items andanalyzing the classifications made, because of that F1-Scoreis the most important metric to be analyzed in the training.
Table 1 shows the metrics results in CellaVisionclassification, the results for each one of the trainings, andthe mean results for single and cascade classification. For all
the metrics, the proposed methods obtained significantly

Fig. 10 F1-Score graphs for each classification layer in cascadeclassification method
Table 1 Results for CellaVision classification (i.e., CV), each oneof the trainings and mean for single classification (i.e.,SC) and cascade classification (i.e., CC) methods
Train
CV

Fig.8 WBC image classified by YOLOv7

9

A

P

R

F1

76.20%

80.93%

92.87%

86.49%SC

Train 1

94.7%

96.63%

97.93%

97.28%SC

Train 293.85%

96.45%

97.2%

96.83%SC

Train 394%

97.41%

96.41%

96.91%SC

Train 491.85%

95.18%

96.33%

95.75%SC

Train 593.55%

95.12%

98.27%

96.67%

SC

Mean

93.59%

96.16%

97.23%

96.69%CC

Train 194.55%

95.55%

98.90%

97.20%CC

Train 294.20%

97.01%

97.01%

97.01%CC

Train 393.20%

97.13%

95.83%

96.48%CC

Train 493.80%

97.96%

95.67%

96.80%CC

Train 593.5%

96.40%

96.89%

96.64%

CC

Mean

93.85%

96.81%

96.86%

96.84%IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Table 2 Accuracy results by class and mean of accuracies for
CellaVision (i.e., CV) classification, single classification(i.e., SC), first layer of cascade classification (i.e., CC l1),
and second layer of cascade classification (i.e., CC l2).

better results compared to the CellaVision ones, and thecascade classification obtained the mean and maximum best
results. The accuracy of the CellaVision classification for thevalidation database is 76.20%, the precision is 80.93%, therecall is 92.87% and the F1-Score is 86.49%. In CellaVisionclassification, 5.85% of the images were wrongly classified
outside the proposed classes as “Artefact” and “Smudgecell”. In single classification, we obtained a mean accuracyof 93.59%, a mean precision of 96.16%, a mean recall of
97.23%, and a mean F1-Score of 96.69%.
The maximum values obtained in five-foldcross-validation were accuracy of 94.7 % in 1st training,
precision of 97.4% in the 3rd training, recall of 98.27% in the
5th training, and F1-Score of 97.28% in 1s training. A meanof 2.67% of the images were classified as “Unclassified” inthis method. In the cascade classification method, we
obtained a mean accuracy of 93.85%, mean precision of
96.81%, mean recall of 96.86%, and mean F1-Score of
96.83%. The best values obtained in five-foldcross-validation were an accuracy of 94.55% in 1st training, aprecision of 97.96% in the 4th training, recall of 98.90% in
the 1st training, and F1-Score of 97.20% in the 1st training. A
mean of 3.05% of the images was classified as“Unclassified”.
Compared to the CellaVision results, the average
accuracy increased by 17.39 percentage points using thesingle classification method and 17.65 percentage points
using the cascade method, the average precision increased
by 15.23 percentage points using the single classification
method and 15.88 percentage points using the cascademethod, the mean recall increased 4.36 percentage points
using the single classification method and 3.99 percentage
points using the cascade method, and the average F1-Score
increased 10.79 percentage points using the single
classification method and 10.35 percentage points using thecascade method. The metrics for the two methods haveapproximate results, accuracy increased by 0.26, precision
increased by 0.65, recall decreased by 0.37 and F1-Score
increased by 0.15 percentage points from single to cascadeclassification methods.
Table 2 shows the accuracy results, for each class andthe mean of the accuracies, for CellaVision classification,
single classification, first layer of cascade classification, andsecond layer of cascade classification. The biggest
difference between the accuracy values obtained between
single classification and cascade classification occurred inthe "Monocyte" class where cascade classification had abetter accuracy by 10.01 percentage points. If we consider

Class

CV

SC

CC l1

CC l2

Basophil

92.07%

100%

99.41%

99.41%

Eosinophil

98.96%

100%

100%

100%

Lymphocyte

85.47%

92.99%

97.27%

90.28%

Variant

31.91%

96.80%

92.98%

91.63%

Monocyte

92.27%

83.45%

93.46%

93.46%

Band

38.88%

95.85%

99.85%

95.20%

Meta

50%

75.63%

99.30%

71.76%

Myelocyte

38.94%

96.42%

99.16%

95.81%

Segmented

91.84%

96.21%

99.76%

95.15%

Erythroblast

77.35%

99.05%

99.43%

99.06%

Thrombocyte

100%

100%

100%

99.38%

Mean

72.52%

94.22%

98.24%

93.74%

the mean of accuracies by class, which represents theaccuracy if the sample has the same number of cells for eachclass, accuracy for single classification is 0.48 percentage
points higher than cascade classification, but it is importantto analyze not just the mistakes but also the kind of thesemistakes.
Table 3, Table 4 and Table 5 shows the error rate
between the classes and Fig.11 shows the most common
mistakes (i.e., higher than 3% error rate) in CellaVision,
single classification, and cascade classification. We noticethat in CellaVision classification, different WBC types (e.g.,
Monocyte and Myelocyte), cells that would be easily
identified by human classification (e.g., Eosinophil andErythroblast), and WBC cells with not WBC (e.g.,
Erythroblast and Lymphocyte) are commonly mistaken. We
also notice that in single classification there are stillcommon mistakes between different types of WBC (i.e.,
Lymphocyte Variant and Monocyte), and in the cascadeclassification method, mistakes between different classes are
avoided and the images are sent to the “Unclassified” folder
instead of being mistake with another wrong class.
Examples of misclassifications are shown in Fig. 12.
Here, images a) to c), were misclassified by CellaVision but
correctly classified by both single and cascade classification
methods, while image d) was misclassified by CellaVisionand single classification but correctly classified by cascadeclassification. In the image a) there is an image misclassified
with a different type of WBC, which compromises theanalysis of quantity of each WBC type. Images in b) and c)show erythroblasts. In both cases, it was mistaken by WBCs

10

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig.11 Common mistakes between classes for a) CellaVision, b) Single classification and c) Cascade classification. Intersections represents
more than 3% of error rate between the classes.
Table 5 Errors rate between classes in Cascade classification for
Fig. 11 c).

Table 3 Error rates between the classes in CellaVision for Fig. 11 a)
CV Bas Met Mye Ban Seg Eos Lym Var

Mon

Ery

Thr

Oth

Bas 92 0.99 1.98 00.99 1.98 0

0

0

0

0

1.98

50 3.52 31.7 4.23 0.70 0

0

7.04

0

0

2.82

1.05 7.37

0

18.9

0

1.05 9.47

00 38.8 48.52 3.70 0

0

0

0

0

8.89

Seg 0.35 0

0 3.47 91.8 2.78 0

0

0

0

0

1.56

Eos

Met 0

Mye 2.11 15.8 38.9 5.26 0
Ban 0

0

0

0

0

0

98.9 0

0

0

0

0

1.03

Lym 0

0

0

0

0

0 85.4 13.7

0

0

0

0.85

Var

0

0

0

0

0

0 3.19 31.9

1.06

0

0

63.8

Mon 0

0

0

0

0

0

92.2

0

0

1.10

0 6.62

Ery 1.89 0

0 2.83 0.94 4.72 7.55

0

0.94

77.3

0

3.77

Thr

0

0

0

0

100

0

0

0

0

0

0

0

CC Bas Met Mye Ban

Seg Eos Lym Var Mon Ery Thr Unc

Bas 99.4 0

0

0

0

0

0

0

0

0

0 0.59Met 0 71.7 14.5 3.36

0.70 00

0 0.420

0 9.25

Mye 0 0.62 95.8 0

0

0

0

0 3.35

Ban 0 0.67 0 95.2

1.77 000

0

0

0 2.36Seg

0

0

0 2.9100

0

0 0.21 000

0 1.9495.1 0

0

0

Eos 0

0

0

0 100 0

0

Lym 0

0

0 0.5100 90.2 4.82 0.17

0

0 4.21

Var 0

0

0

0

0

0 0.99 91.6 1.49

0

0 5.89Mon 0

0

0 0.370

0 2.06 093.4

0

0 4.12

Ery

0

0

0

0

0

0 0.57 00

99.0 0 0.37

Thr

0

0

0

0

0

0

0

0 99.3 0.62

0

0

0

0

Table 4 Errors rate between the classes in single classification for
Fig. 11 b).
SC Bas Met Mye Ban Seg Eos Lym Var

Mon

Ery

Thr

Unc

Bas 100

0

0

0

0

0

0

0

0

Met 0 75.6 13.2 1.97 0.28

0

0

0

0

0.28

0

8.59

Mye 0 1.47 96.4 0

0

0

0

0.21

0

0

0

1.89

Ban 0 0.30 0 95.8 1.93

0

0

0

0

0

0

1.93

Seg 0.14 0

0 1.35 96.2

0

0

0

0

0

0

2.29

Eos

0

0

0

0

0

0

0

100

0

0

0

0

0

Lym 0

0 0.34 00

0 92.9 3.76

0

0.17

0

2.74

Var

0

0

0

0 1.28 96.8

0

0.21

0

1.70

Mon 0.29 0.07 0.29 000 0.37 9.56

83.4

0

0

5.96

Ery

0

0

0

0

0

0 0.19 0.75

0

99.0

0

0

Thr

0

0

0

0

0

0

0

0

100

0

0

0

0

0

0

0

Fig. 12 Example of images misclassified: a) the class is
Myelocyte. It was misclassified as Monocyte by
Cellavision and correctly classified by single and cascadeclassification, b) the class is Erythroblast. It was
misclassified as Eosinophil by CellaVision and Correctlyclassified by Single and cascade classification, c) the
correct class is Erythroblast but it was misclassified as
Lymphocyte by CellaVision and correctly classified bysingle and cascade classification, d) the class is Monocyte.
It was misclassified as Variant by Cellavision and single
classification but correctly classified by cascadeclassification.

11

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Table 6 Comparison with related works
Reference

No. ofClasses

Cell types

Deep
Learning

Database

Shahin5

NormalAlexNet*3 public databases

Single classification + 2(2,551 images)

Transfer Learning methods

et al.

8)

Hedge

6

et al. 3)
Kutlu

Normal +AlexNetAbnormal**

Praveenet al.

5

NormalAlexNet4

NormalYOLOv3

3

Leukemia

YOLOv211)

Abass

Proposedmethod410 images from BCCD +242 images from
LISC open datasets
364 images fromBCCD Dataset

5

Normalet al. 10)
11

Normal +Abnormal

Deep Learning +Accuracy
92.9%
99.6%

Traditional Image Processing
Single classification

97%

Segmentation and90%

Classification in different steps
Classification in different steps

YOLOv5,2,700 images diagnosed
with Leukemia fromVIN Hospital
1,000 images from

Detectron2

Raabin Health Databasecombined

YOLOv75,850 images from Shinshu
University Hospitalobtained with CellaVision

et al. 9)
Akalin1,418 cropped images(hand crafted features)

et al. 2)

Main Features

Segmentation and2 neural networks
Single and cascadeclassification

94.3%

98%
93.85%(98.24%***)* AlexNet modified by the authors** All kinds of abnormal cells are considered just one class*** Result of the first layer of cascade classification into 6 classes

by CellaVision, which also compromises the total WBC
count. Besides, these images would be easily classified byhumans. The image d) shows an example where single
classification method misclassified an image, compromising
the counting of each WBC type, which is a problem in the
single classification method.
If we consider that the images present in the
Unclassified folder are all correctly classified by an expert,
the new accuracy would be 96.26% for single classification
and 96.9% for cascade classification and the images
rechecked would be, on average, 54 for single classification
and 61 for cascade classification what represents,
respectively, 2.67% and 3.05% of the images that the
professionals check without the methods. Although themethodologies still have errors, the health professionals who
worked on this study consider that the error is clinicallyirrelevant, that is, the errors presented would not make any
difference in the medical routine. Finally, Table 6 compares
related works including the proposed methods, in which wecompare the number of classes to be distinguished, cell
types (normal and abnormal), deep learning architectures,
database and images used, main features, and accuracy
performed by each method. Note that the figures of accuracy
in this table are only for reference since a fair comparison
cannot be possible.

7.

Conclusions

This work studies the improvement of the quality of theclassification, using the deep learning architecture YOLOv7
in single and cascade classification methods, of WBC
images segmented by the system CellaVision DM96. Bothevaluated methods presented significantly better results thanthe ones obtained by CellaVision, not just in terms of
metrics but also reducing the criticality of the mistakes. The
CellaVision detections had an accuracy of 76.20%, precisionof 80.93%, recall of 92.87%, and F1-Score of 86.49%. The
single classification method showed an mean accuracy of
93.59%, precision of 96.16%, recall of 97.23%, andF1-Score of 96.69%. The cascade method resulted in an
mean accuracy of 93.85%, precision of 96.81%, recall of
96.86%, and F1-Score of 96.83% for the 0.7 threshold. The
methods separates the images into files representing theirrespective classes and a different file with the images that
could not be classified with the chosen threshold and classes(i.e., “Unclassified”). The methods, mainly the cascadeclassification, facilitate the medical routine because, withoutthem, all the images obtained had to be reassessed due to the
low accuracy of the CellaVision classification, with thediscussed methods, with the increase in accuracy and the
reduction of the most serious errors (e.g., errors between
WBC and non-WBC cells, errors between different WBC

12

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

tipes and errors that would not be made by humanclassification), the professionals already have the cells
separated and classified and they only need to reassess theimages present in the folder “Unclassified”, reducing
drastically the heavy burden (i.e., time, energy) in the
medical routine.
In future work, we will work on defining the maturity
level of neutrophils to support the diagnosis of bacterial
infection. We will also implement the proposed method in a
medical laboratory and compare the performance using
other new neural network architectures.

1645–1656 (2018). https://doi.org/10.1007/s11277-017-5224-x.
7)“YOLOv7: Trainable bag-of-freebies sets new state-of-the-art
for real-time object detectors”, Jul 2022.ArXiv./abs/2207.02696
8)

2)

Convolutional Deep Neural Learning Networks”, Computer
Methods and Programs in Biomedicine, Volume 168, 2019, pp.
69–80, ISSN 0169-2607,
https://doi.org/10.1016/j.cmpb.2017.11.015.
9)

for the Detection and Classification of Leukocytes in Leukemia",

Harold Schumacher, William Rock, Sanford Stass, Handbook ofIndonesian Journal of Electrical Engineering and Computer

Hematologic Pathology, CRC Press; 1 edition (2000).

Science 25.1, 200–213 (2022).

Hüseyin Kutlu, Engin Avci, Fatih Özyurt: "White Blood Cells

10) Fatma Akalin, Nejat Yumusak. "Detection and Classification of

Detection and Classification Based on Regional Convolutional

White Blood Cells with an Improved Deep Learning-Based

Neural Networks", Medical Hypotheses, Volume 135, 2020,

Approach." Turkish Journal of Electrical Engineering and

109472, ISSN 0306-9877,

Computer Sciences 30.7, 2725–2739 (2022).
11) Nalla Praveen, et al. "White Blood Cell Subtype Detection andRoopa B. Hegde, Keerthana Prasad, Harishchandra Hebbar, BrijClassification", 2021 8th International Conference on Electrical

Mohan Kumar Singh: "Comparison of Traditional Image

Engineering, Computer Science and Informatics (EECSI). IEEE

Processing and Deep Learning Approaches for Classification of

4)(2021).

White Blood Cells in Peripheral Blood Smear Images",

12) Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir

Biocybernetics and Biomedical Engineering. Volume 39, Issue

Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang,

2, 2019, pp. 382–392, ISSN 0208-5216.

Jianfei Cai, Tsuhan Chen: “Recent Advances in Convolutional

https://doi.org/10.1016/j.bbe.2019.01.005.

Neural Networks”, Pattern Recognition, Volume 77, 2018, pp.

CellaVision DM96 –Automated Digital Cell Morphology–,354–377, ISSN 0031-3203,

2009. Sysmex America Inc. Document Number MKT10-1033

https://doi.org/10.1016/j.patcog.2017.10.013.

06/2009 2M.

5)Shakir Mahmood Abass, Adnan Mohsin Abdulazeez, Diyar
Qader Zeebaree. "A YOLO and Convolutional Neural Networkhttps://doi.org/10.1016/j.mehy.2019.109472.
3)

Ahmed Ismail Shahin, Yanhui Guo, Khalid M. Amin, Amr A.
Sharawi: “White Blood Cells Identification System Based on

References
1)

Chien-Yao Wang, Alexey Bochkovskiy, Hong-Yuan Mark Liao:

13) Chang-Xue Jack Feng, Zhi-Guang Samuel Yu, Unnati Kingi, M.

https://www.sysmex.com/US/en/products/hematology/cellimage

Pervaiz Baig: “Threefold vs. Fivefold Cross Validation in

analysis/documents/brochure_dm96.pdf. Accessed 31 AugustOne-hidden-layer and Two-hidden-layer Predictive Neural2023.

Network Modeling of Machining Surface Roughness Data”,

Alexander Kratz, Hans-Inge Bengtsson, Jeanne E. Casey, Joan

Journal of Manufacturing Systems. Volume 24, Issue 2, 2005,

M. Keefe, Gail H. Beatrice, Debera Y. Grzybek, Kent B.

pp. 93–107, ISSN 0278-6125,

Lewandrowski,

Elizabethhttps://doi.org/10.1016/S0278-6125(05)80010-X.

Evaluation

the

of

M.

CellaVision

Van

Cott:

DM96

“Performance
System:WBC14) Yolov7 repository. Available at

Differentials by Automated Digital Image Analysis Supported

https://github.com/WongKinYiu/yolov7 on September 2023.

by an Artificial Neural Network”, American Journal of Clinical6)

Pathology, Volume 124, Issue 5, November 2005, pp. 770–781,(Received Oct 15, 2023)

https://doi.org/10.1309/XMB9K0J41LHLATAY.

(Revised Dec 20, 2023)

Yu-chen Wu, Jun-wen Feng: “Development and Application of
Artificial Neural Network2, Wireless Pers Commun102, pp.

13

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Thalita Munique COSTA

Hernan AGUIRRE

She received her B.S. and M.E. degrees inElectronic Engineering, and in Science, Area ofConcentration: Biomedical Engineering, fromFederal University of Technology - Paraná(UTFPR), Brazil in 2018 and 2020, respectively.
She is now a Dr. Eng. Candidate in
Telecommunication System at Shinshu
University, Nagano, Japan. Her researchinterests include Digital Pathology, Artificial
Intelligence, and Image Processing.

He received an engineering degree incomputer systems from Escuela Politécnica
Nacional, Ecuador, in 1992 and a Ph.D. from
Shinshu University, Japan, in 2003. He is a
Professor at Shinshu University. His researchinterests include evolutionary computation andlearning,
multidisciplinary
designoptimization, computational intelligence, andsustainability. He has published over 220
international journal and conference research
papers on evolutionary algorithms, focusing on
the working principles of single-, multi-, many-,
and any-objective evolutionary optimizers,
landscape analysis, epistasis, algorithm design,
and real-world applications. He received the best
paper award at GECCO in 2011, 2015, and 2020.
His students have received the best student paper
award at EMO 2019 and ECTA 2023. He served
as the Editor-in-Chief for the Genetic andEvolutionary Computation Conference 2018 andas a Program Co-Chair for Parallel Problem
Solving from Nature 2022. He serves on the
ACM SIGEVO Executive Board, is Associate
Editor of the ACM Transactions on Evolutionary
Computation and Learning, and is a member ofthe Editorial Board of the Evolutionary
Computation Journal, MIT Press. He is amember of ACM, IEEE, IEICE, and IPSJ.

Yoko USAMI
She received her Ph.D. degree from Tokyo
Medical and Dental university, Tokyo, Japan in
2013. She joined the Department of Laboratory
Medicine at Shinshu University Hospital, Japan
where she is currently a vice tech supervisor.
Her research interests include Laboratory
Medicine, Clinical Chemistry, and ClinicalLaboratory Immunology.

Mai IWAYA
She received her M.D. degree from Shinshu
University in 2005. She completed anatomicalpathology residency at Shinshu University
Hospital in 2011 and completed gastrointestinal
pathology fellowship at University of Toronto,
Canada in 2018. She received the Ph.D. degreefrom Shinshu University in 2019. She joined thedepartment of laboratory medicine at Shinshu
University in 2018 and currently, works as an
associate professor/lecturer. Her researchinterests include artificial intelligence in thepathology field.

Kiyoshi TANAKA (Honorary member)He received his B.S and M.S. degrees inElectrical Engineering and Operations
Research from the National Defense Academy,
Yokosuka, Japan, in 1984 and 1989,
respectively. In 1992, he received the Dr. Eng.
Degree from Keio University, Tokyo, Japan. In
1995, he joined the Department of Electricaland Electronic Engineering, Faculty ofEngineering, Shinshu University, Nagano,
Japan, and currently, he is a full professor in
the Academic Assembly (Institute ofEngineering) of Shinshu University. He is theformer Vice-president of Shinshu University incharge of international affairs. His researchinterests include image and video processing,
3D point cloud processing, information hiding,human visual perception, evolutionary
computation, multi-objective optimization,
smart grid, and their applications. He is ahonorary member and a fellow of IIEEJ, amember of IEEE, IEICE, IPSJ, JSEC, and soon.

Yuka TAKEZAWA
She joined the Department of Laboratory
Medicine at Shinshu University Hospital, Japan
in 2007. She received her Ph.D. degree from
Shinshu University in 2014 and currently, works
as a chief. Her research interests include
Laboratory Medicine and Clinical Hematology.

Yuika NATORIShe received her bachelor's degree from
Teikyo University. She joined the Department ofLaboratory Medicine at Shinshu University
Hospital in 2014. She is currently in charge ofblood and urine tests. Her research interests
include laboratory medicine and clinicalhematology.

14

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2024 -VigNet: Semiautomatic Generation of Vignette Illustrations from Video
Mayu NAMAI† , Issei FUJISHIRO† (Honorary Member)† Keio University<Summary> A variety of summarization techniques has recently been proposed to manage the growing volume of media data, but most are oriented toward homogeneous media conversion, resulting in a limited compression ratio. In thisstudy, we focus on the creation of a vignette illustration that represents the story in an animation or game brieﬂy and thatallows the viewer to understand its world perspective at a glance. If video can be converted into vignette illustration, it isexpected to provide a more highly compressed summary of the media information. This paper proposes VigNet, a systemthat semiautomatically converts an input video into vignette illustrations so they reﬂect users’ preferences.
Keywords: video summarization, media conversion, machine learning, world perspective visualization

1.

The conversion from an input video to vignette illustrations
may be considered one of the most important topics in visual
intelligence. Most existing video summarization techniques
focus primarily on selecting frames, whereas the synthesis ofvignette illustrations requires not only the selection of frames
but also the placement of elements, with subtle adjustments
to their color tones and more, which is far more challenging.
Generative models that convert natural language expressions
to images have recently emerged, but only with conventional
generative models, making it hard to generate vignette illustrations adaptively that arrange viewers’ target contents from thevideo. While large language models are excellent at generating a variety of images, they are ineffective at generating consistent images robustly because they are susceptible to slightchanges in input parameters and training data. Therefore, generating vignette illustrations from video requires a judicious
combination of various visual computing methods, including
video processing and image synthesis.

Introduction

A variety of summarization techniques has been proposed to
manage the growing volume of media data1). However, mostapproaches are aimed at compressing information of the same
modality, resulting in a limited compression rate. To achieve
an efﬁcient summarization, it is essential to explore the potential of heterogeneous media conversion. Thus, this study
focuses on vignette illustrations, miniature diorama-like illustrations that represent the condensed story content of animatedﬁlms and games for entertainment purposes. The modality ofillustration remains visual, but it clearly differs from video, asshown in Fig. 1.
As a result of our own preliminary analysis of approximately 20,000 examples on Pinterest11) and other related websites, we found that a vignette illustration should consist of thefollowing ﬁve elements:• Characters: The main character and others• Stage: Base for positioning characters• Background: Background scene with vague outlines• Supporters: Objects supporting characters• Effects: Visual effects that enhance characters
Figure 1 gives an example of a vignette illustration and the ﬁveelements. In vignette illustrations, the characters and stage are
essential elements, while the other three are ancillary elementsthat may be combined as needed. Furthermore, a typical characteristic of vignette illustrations is that they often depict the
full-body images of characters. In addition, it is a notable factthat some parts of the background may be omitted.

Fig. 1 An example of a vignette illustration with the ﬁveelements. This example includes all elements,
but some vignette illustrations may not include
the Background, Supporters, or Effects.

15

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig. 2 Example execution of the VigNet system. Input video6) (a) is semiautomatically converted to an output illustration (b)Fig. 3 VigNet processing framework
This paper proposes a system called video image generative network (VigNet), which generates semiautomatically vignette illustrations from a video, reﬂecting user preferencesby means of semantic segmentation and the additional learning of image synthesis models. An example of the system’s
execution is shown in Fig. 2.

2.

3. System
As shown in Fig. 3, the processing framework of VigNet
is based on the preliminary analysis mentioned in Chapter 1.
Note in the current framework that non-character elements areaggregated into a stage without distinguishing among them.
Hereafter, the image area for characters, or the stage, is referred to as the character or stage material, respectively. Bothmaterials are generated independently in the process outlined
in Section 3.1 and Section 3.2 and are combined in the process of Section 3.3 to yield a vignette illustration. In addition,
Fig. 4 and Fig. 5 use the video Alike6) to illustrate the intermediate processes of the system.

Related Work

Composable diffusion14) is one of the latest generative models for heterogeneous media conversion, enabling various
combinations of different media formats. By mediating thenoise between different input and output media, it can handle many combinations of input and output modalities. However, the paper, which focuses on composable diffusion models, does not consider conversion from video to static images,
nor does it provide any examples of results for this speciﬁctype of conversion.
Meanwhile, NewsThumbnail8) is a system that generates
automatic thumbnails from news videos by screening contents
that are semantically similar to the user’s query and combining them. However, its results are displayed only in the form ofmultiple snapshots arranged with keywords, so NewsThumbnail does not have as high a condensed impression as VigNet.
To the best of our knowledge, there are no known reported
studies on the conversion of a video to a single still image thatis not merely an enumeration of the contents.

3.1 Character material generation
We refer to frames capturing the full bodies of characters
as candidate character frames, which are extracted using
the following procedure. First, for frames extracted at regular intervals from the input video, we use the Grounded Segment Anything Model (Grounded SAM)4) 9) to detect characters in the frames by specifying “character” as the prompt.
Then, only frames in which detected bounding boxes do nottouch the image boundary are selected, because vignette illustrations typically portray the full bodies of characters. In addition, frames in which characters overlap with other objects
are manually excluded, by which process we obtain candidate
character frames, as shown in Fig. 4(a).

16IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig. 4 Character material extraction. (a) Candidate character
frame examples and (b) Clipped results
Second, candidate character frames are displayed to users,
where each detected character is enclosed in a red frame of
the Grounded SAM. Then, the users are prompted to selecttheir favorite frame, referred to hereafter as the representative
character frame.
Third, the character material is extracted from the representative character frame using the mask generated by the
Grounded SAM, the actual results of which are shown in
Fig. 4(b). If there is more than one character in the frame, therelative positions of the characters in the original frame arealso stored and referenced, as will be shown in Section 3.3.
This relative positioning effectively reﬂects the relationships
between characters. In many cases, the distance between characters within a frame image represents their psychological distance. When there is a problem in their relationship, characters
are often depicted physically apart in the frame. Conversely,
when characters harbor intimate feelings for each other, they
are typically shown close together. It can be seen in the animation Alike, as illustrated in Fig. 4, that characters stand apartin scenes of disagreement (second from the left), and are remarkably close in scenes of agreement (fourth from the left).

Fig. 5 Stage material generation. (a) Candidate stage frame
examples and (b) Process of converting a representative
stage frame into stage materialof a speciﬁc prompt and the caption to be assessed, into theAPI. The content of the prompt is, “I will input the text now.
Please respond only with the number．In the following texts,
answer 1 if the ﬁrst noun is a character, such as a person or
animal, and otherwise answer 2.” Based on the instruction, if
the response returned from the ChatGPT API is ‘2,’ it is determined that the caption pertains to the background context.
Initially, the accuracy of a single assessment by the ChatGPTAPI was approximately 80%. To achieve higher accuracy, we
conducted 10 rounds of evaluations, designating the ﬁnal candidate stage frame only if the caption was judged as ‘2’ ﬁve or
more times. This approach improved the accuracy up to about
90%. However, there remains room for exploration to reﬁnethe prompts entered into ChatGPT and improve the accuracyof this method.
Second, candidate stage frames are presented to users forselection of the representative stage frame, deﬁned as theframe the users ﬁnd most appealing. If the second candidate
in Fig. 5(a) is selected as the representative stage frame, theprocessing described later is executed based on the frame, as
shown in Fig. 5(b).
Third, a stable diffusion model12) is additionally trained by
Low-Rank Adaptation (LoRA)3) using training data frames
extracted from an input video at regular intervals. We also
employed the Counterfeit-V2.5 model2), which was tailored to
produce anime-style images, as the pre-ﬁne-tuned model.
Fourth, with the following stable diffusion models, stage
material is generated from the representative stage frames via
two-step img2img, as shown in Fig. 5(b). In the ﬁrst step, using the pre-ﬁne-tuned model, an image is generated by taking
as inputs the caption of a representative stage frame and an
existing vignette illustration to use as a template image. The
output image is currently deﬁned as the base image. In the second step, the same caption and base image are used as inputs
to generate the stage material using the post-ﬁne-tuned model.
In the initial phase of generation, the base image may fail to

3.2 Stage material generation
We refer to frames capturing stages as candidate stage
frames, and they are extracted by the following procedure.
First, by comparing the average absolute error of consecutive
frame images, cuts in the input video are detected. Then, aframe located in the middle of each detected cut is extracted.
For these frames, the Grounded SAM is once again applied
to detect characters and to select only frames in which a relatively small proportion of the scene is occupied by the characters. Next, captions for each frame are obtained using the
BILP Image Captioning Large Model7). From these captions,frames in which the stage is mentioned are selected using
the GPT-3.5-turbo model10) of the ChatGPT API. Throughthis process, we obtain candidate stage frames, as shown in
Fig. 5(a).
We detail a method for leveraging ChatGPT to determine
whether a caption describes the background context. This
procedure involves the input of concatenated text, consisting17

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）(a) Frames of the input video, extracted at 1,000-frame intervals(b) Representative frames and output results. The representative character frames are enclosed in blue, the representative stage frames ingreen, and the vignette illustrations generated by their combination are encompassed in red

Fig. 6

Frame images and examples of output results with Alike6) as the input video

capture the color scheme and ambience of the video. However,in the subsequent phase, the stage materials accurately reﬂect
the video’s color palette and atmosphere.

Table 1 Detailed statistics for additional training. M4.1 isdeﬁned as the model used in Section 4.1, and M4.2a,
M4.2b, and M4.2c as the models used in Section 4.2.
The video (a) is Alike6), (b) Grump in the Night 5),and (c) Flow13)

3.3 Image synthesis
Initially, the character material is automatically resized toensure appropriate scaling within the vignette illustrations. As
described in Chapter 1, the preliminary analysis revealed thatthe number of characters typically present in a single vignette
illustration ranges from one to two. Furthermore, it was empirically determined that the total area occupied by the charactersin an illustration should be approximately 0.1 × the number ofcharacters. Based on this empirical rule, the character assets
were resized accordingly.
Second, the Grounded SAM was used to detect the characters’ foothold on the stage material and to place the charactersby specifying “ﬂoor surface” as the prompt. If one characteris present, they are placed at the center; otherwise, they aresuperimposed on the stage material, reﬂecting their positionin the representative character frame. In this way, the character(s) and stage material are merged into a vignette illustration.

4.

Model

M4.1

M4.2a

M4.2b

M4.2c

Video
#learning frames#total learning steps
Training time
Loss

(a)
1,021
204,200
24h 17m
0.0851(a)
103
20,6003h 31m
0.103

(b)
61
12,2001h 50m0.14(c)50
10,0001h 41m0.146

ment the current prototype system of VigNet. For the experiments, we exclusively used modiﬁable CG animation videos
that are licensed under Creative Commons license to ensure
their appropriateness for publication. As vignette illustrations
are often derived from anime and game artworks, we evaluatedwhether VigNet could generate vignette illustrations from actual anime productions.
4.1 Execution example
The short movie Alike6) was used as an input to the system, some frames of which are shown in Fig. 6(a). The work
consists of 10,208 frames in total, and it is 6 minutes and 48
seconds long. We prepared a model trained with frame images
extracted at 10-frame intervals from the input video, which

Results

We used a personal computer with CPU: 12th Gen Intel(R) Core(TM) i9-12900H @2.30GHz, RAM: 64 GB, GPU:NVIDIA GeForce RTX 3080 Ti, and Python 3.8.16 to imple-

18

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）(a) Alike6)(b) Grump in the Night 5)(c) Flow13)

Fig. 7 Frames of input videos, and examples of representative stage frames and their output results. Frames of input videos are
extracted at 1,000-frame intervals
candidate stage frames, which were extracted automatically.
Thus, the number of intermediate frames did not confuse theuser when making their choice. Several representative frames
and output results are shown in Fig. 6(b), and the ﬁve output
results cover the user’s selection of representative frames fromthe candidate frames in various combinations. In all cases,
output illustrations were generated so they would convey thefeatures of the frame images selected by the ﬁrst author andsummarize the content of an input video.

took 24 hours and 17 minutes to train. The parameters include
a repetition count of 20, epoch number of 10, training batch
size of 10, network rank of 8, and no normalized images. The
detailed statistics of the additional training are shown in Table 1. The procedure in Section 3.1 automatically extracted
41 frame images, 13 of which were manually extracted as candidate character frames, and the procedure in Section 3.2 resulted in 10 candidate stage frames. Meanwhile, candidate
character frames were extracted manually, as opposed to the

19IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

For VigNet to function effectively with input videos, weconsider two necessary conditions. The ﬁrst is the inclusion offrames that display the full body of the character without anyother objects overlapping it in the foreground. The second is
the inclusion of candidate stage frames in which characters do
not appear. At present, the system requires the manual selection of candidate character frames and representative framesby the user, but we have a plan to make it fully automatic in
the future.
Even when live-action videos are input into VigNet, if theabove two necessary conditions are met, VigNet is expected tofunction effectively. The object detection feature of Grounded
SAM used in VigNet has proven effective not only with animated images but also with live-action images. When using
live-action videos as input, it may be necessary to adjust the
prompts slightly for object detection, but otherwise, it is anticipated that vignette-style live-action images would be generated. Furthermore, some of the stable diffusion models used
for image generation excel at producing photorealistic images.
Therefore, using a model specialized in generating realistic
images rather than the Counterfeit-V2.5 model used in thisstudy might yield better results.

ate base images, the denoising strength was set to 0.75. Conversely, for the generation of stage materials with img2img,
the strength was adjusted to 0.90. These settings were adopted
as a consistent criterion for image generation within our sensitivity analysis.
Figure 7 gives frames of input videos and examples of two
representative stage frames based on which two vignette illustrations were generated. The visual representation of the stage
materials varies signiﬁcantly depending on the chosen representative stage frame. Furthermore, even when using the same
representative stage frame, by merely adjusting the seed value
in img2img, it becomes evident that stage materials with similar yet distinct appearances can be generated.
In the current version of VigNet, elements other than characters are aggregated into the stage. Indeed, Fig. 7(b) extracts
the room as a background, while Fig. 7(c) contains coral asa supporter. Because the input videos used in this experiment
did not contain any effects, it follows naturally that the output illustrations also lacked effects. Moreover, it was easier togenerate a higher quality stage material from a video such as
Flow, with sequences of similar supporters, than from another

4.2 Sensitivity analysis
We investigated how changes to an input video affect theoutput illustrations. In addition to Alike, two other videos,i.e., Grump in the Night 5) by Kris Theorin, Somethings Awry
Production and Flow13) by The Animation School, were selected as inputs. Table 2 compares the three videos in termsof several features. The parameters are consistent across allcases, with a repetition count of 20, epoch number of 10, training batch size of 10, Network Rank of 8, and no normalized
images. The detailed statistics of the additional training areshown in Table 1.
Alike contains both indoor and outdoor scenes, while Grumpin the Night and Flow are composed mostly of indoor scenes
and outdoor scenes, respectively. As a dataset for the learning model, frame images were commonly extracted from each
video at 100-frame intervals. For image generation, denoisingstrength was set as follows: when using img2img to generTable 2

Fig. 8 Examples of Failures

Features of input videos. (a) is Alike6), (b) is
Grump in the Night 5), (c) is Flow13)
Title

(a)

(b)

(c)

Duration#total frames#cuts
Main scenes#characters

6m 48s
10,20876
indoor & outdoor
3

4m 13s6,096
103
indoor4

3m 24s4,924
16
outdoor3

Fig. 9 Examples output by VigNet for different amounts ofmodel training data．(a) was generated using a modeltrained on 103 frame images, while (b) was generated
with a model trained on 1,021 frame images
20

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig. 10

Results of questionnaire

one, such as Grump in the Night, which contains many different supporters. Another reason for the low quality of thegenerated illustrations in Grump in the Night may be the highfrequency of cuts.
Moreover, it was found that an insufﬁcient number of images in the training dataset failed to generate high-quality
stage materials. Figure 8 shows two resulting illustrations
that use stage materials based on two outdoor frames from
Grump in the Night. Because most of this video is composedof indoor scenes, when attempting to generate outdoor scenes,
the training data were insufﬁcient, resulting in stage materials that hardly reﬂect the appearance of the house or weather
conditions.
Conversely, it was found that as the amount of image data
for training increased, VigNet was able to generate highquality stage materials. Figure 9 compares the output resultsof VigNet when the amount of training data for the model
varies. Figure 9(a) was generated using a model trained on
103 frame images, while Fig. 9(b) was generated with a modeltrained on 1,021 frame images. Upon comparison, it is obvious that (b) reproduces the shape and color of the objects in
the input video more accurately. In fact, the model that produced (a) exhibited a loss of 0.103, whereas the model thatgenerated (b) demonstrated a lower loss of 0.0851, indicatingthat the latter achieved a smaller loss value.

Table 3
a
b
c
d

e
f

g
h

Questionnaire items

Did any of the candidate character frames include the characters you imagined?
Did any of the candidate stage frames include the background you imagined?
How much of a difference is there between your imagined
vignette illustration and the output illustration?
Under the assumption that the quality of your imagined illustration is 0, what is the quality of the output
illustration?
Do you think the output image is a vignette illustration?
Which reminds you more of the content of the video, Alike’s
YouTube thumbnail, as shown in Fig. 11, or the
output image?
Does the output image condense the essence of what you
consider important in the video?
Does the output image express the world perspective of thevideo?

Fig. 11

Alike’s YouTube thumbnail

system, we asked each to choose two representative
frames from the candidate frames.
3. We presented each with VigNet’s output illustrations corresponding to their selected combinations.
4. We asked them to answer a questionnaire to evaluate theoutput illustrations while comparing with the vignette illustrations they imagined.

4.3 Evaluation experiments
We conducted an evaluation experiment concerning the output results for Alike as the input video. For a rewarded
viewer evaluation, we collected 26 non-professional participants (50–50 split between genders). The participants rangedin age from their late teens to late twenties. As a reward, each
was offered snacks and beverages. The procedure for the evaluation experiment is as follows.
1. We let them freely watch the movie, Alike, and then asked
them to imagine their own vignette illustration.
2. After brieﬂy introducing the process ﬂow of the VigNet

Table 3 lists eight questions asked in the questionnaire, andFig. 10 shows the results of each. The participants in the evaluation experiment were asked to answer the questions on a
7-point scale from +3 to -3. Positive (+3 to +1) and negative(-3 to -1) results are indicated by the reddish and bluish color
gradations, respectively, except for neutral evaluations (0), indicated in white.

21

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

As a result, 60% of participants answered that the results
generated by VigNet were close to their imagined vignette illustrations, more than 90% stated the results summarized thevideo well, and all participants reported that the results expressed the world perspective of the video.
Many participants commented that they would like various
supporters, including the violin, the stacked books, and thecharacter’s bag, to appear in an output illustration. Each of
the participants had different ideas of what elements should
be included in the output image. Several participants speciﬁcally mentioned that if such supporters could be added to the
VigNet output, the result would be extremely close to theirown vignette illustrations. This suggests that VigNet’s results
reﬂect common ideas shared among participants.
Based on this feedback, we would like to enable VigNet toreﬂect users’ preferences by placing supporters explicitly in
its outputs. We also have a plan to conduct additional experiments to evaluate the system’s usability after a careful redesignof the system’s user interface. Furthermore, future studies willcompare the outputs generated by VigNet with vignette illustration drawn by professional artists.

5.

5) Somethings Awry Production Kris Theorin, Grump in the Night,
2022,
https://www.youtube.com/watch?v=ystQWn3K2GY, License: https://creativecommons.org/licenses/by/4.0/, latest
access 02/04/2024.
6) Daniel Martı́nez Lara and Rafa Cano Méndez,

Alike,

2017,
License:https://creativecommons.org/licenses/by/4.0/,
latest access 02/04/2024.
https://www.youtube.com/watch?v=PDHIyrfMl_U,7) J. Li et al., “BLIP: Bootstrapping language-image pre-training for uniﬁed vision-language understanding and generation,” in Proceedings ofthe 39th International Conference on Machine Learning, PMLR, Vol.
162, pp. 12,888–12,900, 2022.
8) J. Li et al., “Newsthumbnail: Automatic generation of news video
thumbnail,” in 2022 IEEE International Conference on Systems,
Man, and Cybernetics (SMC), pp. 1383–1388, 2022. DOI: 10.1109/
SMC53654.2022.9945444

9) S. Liu et al., “Grounding DINO: Marrying DINO with grounded pretraining for open-set object detection,” 2023. https://arxiv.org/
abs/2303.05499

10) OpenAI, ChatGPT,
02/04/2024.

https://chat.openai.com/,

11) Pinterest, Inc., Pinterest, Vignette Illustration,

latest accesshttps://www.

pinterest.jp/search/pins/?rs=ac&len=2&q=vignette%
20illustration&eq=vignette%20illu&etslf=11462,

latestaccess 02/04/2024.
12) R. Rombach et al., “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 10:684–10:695, June 2022.
13) The Animation School, Flow, 2023, https://www.youtube.com/
watch?v=rO87HFsmfis, License: https://creativecommons.org/
licenses/by/4.0/, latest access 02/04/2024.

Conclusion

Regardless of the kind of metaverse realized, the value ofa still image that provides a bird’s-eye view of the world willremain unchanged. In this paper, therefore, we proposed the
VigNet system for generating vignette illustrations semiautomatically from computer graphics animation videos by combining semantic segmentation and ﬁne-tuned image synthesis
models. The evaluation experiments conducted empirically
proved that the prototype system of VigNet can robustly visualize the world perspective of videos on a small scale, while
reﬂecting users’ preferences.

14) Z. Tang et al., “Any-to-any generation via composable diffusion,” 2023.
https://arxiv.org/abs/2305.11846(Received November. 27, 2023)
Mayu NAMAI
She is currently a master’s student in the Schoolof Science for Open and Environmental Systems,
Graduate School of Science and Technology, Keio
University. She received her bachelor’s degree inInformation and Computer Science from Keio University in 2023. Her major research interest includes semiautomatic generation of vignette illustrations from videos.

Acknowledgments
This work has been in part supported by the JSPS KAKENHI Grant-in-Aid for Challenging Research (Exploratory)
Grant Number 23K18468 and the Research Grant of Keio
Leading-edge Laboratory of Science & Technology.

Issei FUJISHIRO (Honorary Member)
He is currently head professor of the Center for Information and Computer Science at Keio University, Yokohama, Japan. He received his Doctorof Science in information sciences from the University of Tokyo in 1988. His research interests
include modeling paradigms and shape representations, applied visualization design and lifecycle
management, and smart ambient media with multimodal displays. He is an associate member of Science Council of Japan, a fellow of JFES and IPSJ,
and a senior member of IEEE and ACM. He is a
2021 inductee to the IEEE Visualization Academy.

References
1) T. Baltrušaitis et al., “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, No. 2, pp. 423–443, 2019. DOI:10.1109/TPAMI.2018.
27986072) gdgsfsfs, Hugging Face, Counterfeit-V2.5, https://huggingface.
co/gsdf/Counterfeit-V2.5, latest access 31/03/2024.
3) E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,”
2021. https://arxiv.org/abs/2106.09685
4) A. Kirillov et al., “Segment anything,” 2023. https://arxiv.org/
abs/2304.02643

22

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2024 --Adaptive Selection of Auxiliary Tasks Using Deep Reinforcement Learning
for Video Game Strategy
Hidenori ITAYA† (Student Member ), Tsubasa HIRAKAWA† , Takayoshi YAMASHITA† , Hironobu FUJIYOSHI†† Chubu University<Summary> Multitask learning can be utilized to eﬃciently acquire common factors and useful features
among several diﬀerent tasks. This learning method has been applied in various ﬁelds because it can
improve the performance of a model by solving related tasks with a single model. One type of multitask
learning utilizes auxiliary tasks, which improves the performance of the target task by learning auxiliarytasks simultaneously. In the video game strategy task, unsupervised reinforcement learning and auxiliary
learning (UNREAL) has achieved a high performance in a maze game by introducing an auxiliary task.
However, in this method, the auxiliary task must be appropriate for the target task, which is very diﬃcult
to determine in advance because the most eﬀective auxiliary task will change dynamically in accordancewith the learning status of the target task. Therefore, we propose an adaptive selection mechanism called
auxiliary selection for auxiliary tasks based on deep reinforcement learning. We applied our method to
UNREAL and experimentally conﬁrmed its eﬀectiveness in a variety of video games.
Keywords: deep learning, auxiliary task, reinforcement learning

auxiliary tasks7). In video games, game scores have been

1.

Introduction

improved by introducing three diﬀerent auxiliary tasks to
the main task, which is a game strategy based on deep
reinforcement learning (DRL)8). The auxiliary tasks inthese studies were designed to be tasks that were unrelated to the main task but that were intuitively thought
to contribute to the main task. For example, the domainof an in-vehicle camera image is highly dependent on the

Real-world problems are complex mixtures of various
elements, and even seemingly disparate tasks can beclosely related. Solving these diﬀerent tasks simultaneously with a single model can improve the model performance and reduce training time. This learning method,
known as multitask learning, can eﬃciently acquire factors and useful features that are common to several different tasks. Research in the ﬁeld of image recognition

time of day and the weather conditions. Since auxiliarytasks are designed manually, they do not necessarily contribute to solving various main tasks. On the contrary,some of them may actually interfere with learning, depending on the main task. While this problem could potentially be solved by carefully designing auxiliary tasks
that depend on the main task, it is expensive to verify
the eﬀectiveness of auxiliary tasks. In learning methods
that introduce these auxiliary tasks, the loss function isa weighted sum of each task, including the main task.
Therefore, the weight of each task, i.e., the ratio of themixing gradients for each task, is important for learning.
However, it is diﬃcult to set the optimal weights in advance because they depend on the nature of the main task
and the training stage.
In response to these challenges, the purpose of our cur-has shown that model performance can be improved bylearning class classiﬁcation, object detection, segmentation, etc. simultaneously1)–4). In the ﬁeld of natural language, training part-of-speech classiﬁcation, sentence segmentation, and sentence relation values simultaneously isalso known to be eﬀective5),6).
Auxiliary learning is a type of multitask learning that
improves the performance of the main task as a kind of
normalization by adding auxiliary tasks unrelated to themain task to be solved to the learning target task andby learning the auxiliary tasks at the same time. In
automated driving, the accuracy of the main task can
be improved by utilizing depth estimation and semantic
segmentation from in-vehicle camera images as the maintask and introducing time and weather estimation as therent study is to adaptively select an auxiliary task suit-

23

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

information as input and the position of the auxiliary task
in the network structure, veriﬁed the eﬀectiveness of theable for the main task. In this paper, we propose a new
auxiliary task selection mechanism, called auxiliary selec-

depth prediction. Kartal et al. focused on the terminal state of the environment and proposed an auxiliarytion, which adaptively selects auxiliary tasks according to
the main task. Speciﬁcally, it is designed as a DRL agent
that outputs weights for each auxiliary loss and adaptively selects auxiliary tasks for network updates on thebasis of the main task. The auxiliary selection model istrained simultaneously with other models and shares the
reward signal with the main task to ﬁnd the appropriate
auxiliary task according to the training stage of the maintask. We apply auxiliary selection to UNREAL, a methodthat introduces auxiliary tasks into video game strategy,
and analyze the selected auxiliary tasks and game scorestask called terminal prediction (TP) to predict how closethe current state is to the terminal state of the environment11). Hernandez-Leal et al. focused on the behavior of other agents in multi-agent problems and proposed
agent-modeling, an auxiliary task to predict the behaviorof other agents12). Experiments on cooperative and competitive tasks of multi-agent problems showed that theagent-modeling contributed to improving the stability oflearning, thereby improving the accuracy in each task.
In the DRL ﬁeld, there has been extensive research onvideo game strategy13), and the improvement of game
scores by using auxiliary tasks has been reported. Jaderberg et al. proposed unsupervised reinforcement learningand auxiliary learning (UNREAL), in which an unsupervised auxiliary task is learned simultaneously with themain task in deep reinforcement learning8). The main

to determine the selection of suitable auxiliary tasks forthe main task.
The contributions of our study are as follows.
• Our method eﬃciently improves the performance ofthe main task by utilizing deep reinforcement learning to select the optimal auxiliary task according to
the training stage and scene.

task of this method is a video game strategy from images (game screen frames) using asynchronous advantage
actor-critic (A3C)14). Thanks to solving the auxiliarytask by using part of the main task model and learningwith a weighted sum of the main and auxiliary losses, theauxiliary task improves the video game score. The authors used three diﬀerent auxiliary tasks in their study.
The ﬁrst is pixel control, which learns the agent’s behavior such that pixels in the input image are changed. Thistask is based on the idea that visual changes in the environment are associated with important events. In this
auxiliary task, the input image is divided into n×n grids,
and the agent learns an action value Q(c) to maximize
the pixel value change in each grid. Here, c denotes thenumber of grids into which the input image is divided.
The second auxiliary task is value function replay, which• Our method automatically suppresses unnecessary
auxiliary tasks and prevents the main task from losing accuracy, thus reducing the cost of designing auxiliary tasks.

2.

Related Works

2.1

Improving the performance of the maintask by introducing auxiliary tasks
In the ﬁeld of multitask learning, many investigatedhave studied how to improve the performance of the maintask by introducing auxiliary tasks. These studies aredescribed below, with a focus on supervised learning anddeep reinforcement learning.
In the ﬁeld of supervised learning, Liebel et al. introduced semantic segmentation and depth estimationas main tasks in automated driving and auxiliary tasks
for time and weather estimation7). They reported that

shuﬄes past experiences to learn the state value function
V (s). This task learns state values using experience replay15), an eﬀective mechanism for improving both thethe auxiliary tasks of time and weather estimation contributed to improving the accuracy of the main task.
Zhang et al. achieved more robust face landmark detec-

data eﬃciency and stability of the DRL algorithm. In
other words, it is equivalent to performing state valuefunction regression with an auxiliary task in addition totion by simultaneously optimizing the main task of facelandmark detection and the auxiliary tasks of face poseand attribute estimation9).
In the ﬁeld of deep reinforcement learning, Mirowski
et al. focused on navigation tasks in 3D environments
and proposed an auxiliary task, depth prediction, which
predicts depth information from RGB images10). A com-state value function regression with A3C, which promotes
optimal state value estimation. The third auxiliary task
is reward prediction, which predicts the reward to be obtained in an unknown state. In DRL, learning an agent’s
behavior requires accurate recognition of the states thatgive high rewards. However, reward acquisition is very

parison under various conditions, such as the use of depth

24

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）
Table 1 Comparison to proposed and conventional
methods
MethodTarget
How to selectObtain common policy
16)
Tel et al.
Policythrough distillation
Riedmiller et al.17) Policy
Adopt hierarchical policies
Du et al.18)
Adopt gradient similarity
Task
between tasks
Lin et al.19)
Ours

Task

However, these methods are diﬃcult to apply to tasks
that cannot be subdivided into smaller policies, and thetraining cost is high because the construction of auxiliary
policies requires individual training on a diﬀerent target.
The auxiliary task approach solves a task with an auxiliary purpose at the same time as the main task andselects which of these auxiliary tasks to use during learning. The methods by Du et al. and Lin et al. are auxiliary task approaches. These two methods focus on thegradient of the auxiliary loss, which is the loss of the auxiliary task. Du et al. utilize the cosine similarity between
the gradients18), and Lin et al. use the Taylor approximation including the gradient of the auxiliary loss19)to
calculate the similarity between tasks. This similarity isthen utilized to select which auxiliary losses to be used fortraining the main task. Since these methods can utilize
auxiliary tasks suitable for the main task during learning,they can reduce the training cost compared to the auxiliary policy and contribute to improving the performanceof the main task. However, the similarity of the auxiliary
losses used to select the auxiliary tasks is very sensitive
to the appropriate measure (e.g., how similar is it when
to use auxiliay tasks, etc.), as it depends on the task andthe training situation.
Our method, like those of Du et al. and Lin et al., is an
auxiliary task approach. We construct a DRL agent thatdynamically controls the coeﬃcient for the auxiliary loss,
thereby achieving the auxiliary task selection according
to the training status of the main task. Therefore, ourmethod optimizes the aforementioned coeﬃcients for auxiliary loss by DRL, which thus enables dynamic selectionof auxiliary tasks.

Select auxiliary tasks by DRL

sparse in many environments, and it takes a long timeto train a model that represents reward acquisition well.
Therefore, by solving the task of predicting the rewardthat can be acquired in the unknown state that follows
from sequential states, the task eﬃciently learns to recognize the states that contribute to reward acquisition. The
learning of this auxiliary task is achieved by multi-classcross-entropy classiﬁcation loss over three classes (zero,
positive, negative) of rewards for the next state. By introducing these auxiliary tasks, UNREAL achieved highscores on the main task, which was the maze capture at
DeepMind Lab.
2.2

Selection of auxiliary tasks

Auxiliary learning improves the performance of themain task, but if auxiliary tasks that are not suitable
for learning the main task are used, they will interferewith the learning of the main task, resulting in a loss of
accuracy. Therefore, it is necessary to introduce auxiliary tasks that are suitable for the main task. Several
methods have been proposed to solve this problem.
Conventional methods can be divided into those thattake an auxiliary policy approach and those that take an
auxiliary task approach. A comparison of the proposedand conventional methods is provided in Table 1. The
auxiliary policy approach constructs a number of policies

3.

with auxiliary purposes to the main task and selects the
optimal policy from these auxiliary policies. The methods by Tel et al. and Riedmiller et al. are auxiliary policy approaches. Tel et al. use knowledge distillation20)toobtain a common policy among several auxiliary policies
with diﬀerent roles16), while Riedmiller et al. achieve aux-

Proposed Method

The eﬀectiveness of the auxiliary task depends on themain task, and an inappropriate auxiliary task may prevent the learning of the main task. Therefore, we propose
auxiliary selection, which adaptively selects the auxiliarytask according to the main task. In this method, we apply our method to UNREAL, which has achieved a highiliary policy selection by adopting a hierarchical structurewith several low-level auxiliary policies and a main policythat selects the optimal auxiliary policy17). The main policy focuses on a speciﬁc task, whereas the auxiliary policies represent actions under other goals and conditions.
Therefore, it is easy to obtain feedback from the selection of auxiliary policies, as the agent’s behavior changes
signiﬁcantly depending on the selected auxiliary policies.

performance score by introducing three auxiliary tasks invideo game strategy, and extend it to a method that canlearn eﬃciently on various video games. In other words,
this method eﬃciently strategies video games by dynamically selecting auxiliary tasks according to the video game
to be solved.

25

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）     




­





     

  


 
  

 

  





0



  +     







  

  



               

Fig. 1 Overview of proposed method. Here, pixel control, value function replay,
and reward prediction are unsupervised auxiliary tasks proposed by Jaderberg et al.

3.1

3.2 Adaptive selection of auxiliary tasks
Figure 1 shows the network structure of our method.
In this Section, we introduce auxiliary selection (AS),which selects the most appropriate auxiliary task for thevideo game task to be solved. AS takes an image stored
in the replay buﬀer as input and outputs the state value
VAS (s) and the policy πAS . The AS policy πAS determines

Background8)

Similar to Jaderberg et al. , we assume a standard
reinforcement learning setting in which the agent interacts with the environment over several discrete time
steps. At time t, the agent receives observation otwith reward rt and generates action at . The agent’s
state st is a function of experience until time t, wherest = f (o1 , r1 , a1 , · · · , ot , rt ). The n-step return Rt:t+n

the binary weights for each auxiliary task, and indicates
whether or not each auxiliary task is utilized for training
the main task. The binary weights for each auxiliary task
are deﬁned as CPC = {0, 1}, CVR = {0, 1}, CRP = {0, 1}.
The AS policy πAS is calculated as

at time t is deﬁned as the discounted sum of rewards
n
Rt:t+n = i=1 γ i rt+i . The value function is the expected
return from the state s, Vπ (s) = E[Rt:∞ |st = s, π].
In our study, the main task is a video game strategy,
and the three UNREAL auxiliary tasks described in Section 2.1 are utilized as auxiliary tasks. The UNREAL
auxiliary tasks are unsupervised auxiliary tasks that are
not dependent on the main task. This makes it more difﬁcult to manually select suitable auxiliary tasks for themain task in advance compared to supervised auxiliary

πAS = (CPC , CVR , CRP ).

(1)

AS consists of two convolutional layers and a fullyconnected layer. Unlike other auxiliary tasks, the AS
network is constructed without sharing weights with themain task network, and AS is trained independently oftasks. In our method, we achieve the dynamic selection of
auxiliary tasks by building a DRL agent that selects themost appropriate auxiliary task according to the video
game to be solved. The main task is learned using Asynchronous Advantage Actor-Critic (A3C)14)as well as UNREAL. The A3C algorithm constructs an approximationof the policy π(a|s, θ) and the state value function V (s, θ)using the model parameter θ. The policy and statevalue are adjusted toward the n-step lookahead value
Rt:t+n + γV (st+n+1 , θ) using an entropy normalization
penalty LA3C ≈ LVR + Lπ − Es∼π [αH(π(·|s, θ))], where
LVR = Es∼π (Rt:t+n + γ n V (st+n+1 , θ− ) − V (st , θ))2 .

the main task. AS’s policy is learned on the basis of thesame reward as the main task. In other words, AS controls the binary weights of the auxiliary tasks to improve
the score of the video games, which is the main task.
3.3 Loss function
We formulate the loss function for our method as
 (c)L = LA3C + CPC
LQ + CVR LVR + CRP LRP . (2)cwhere LA3C is the loss of the main task (i.e., A3C) and (c)c LQ , LVR , and LRP are the losses of each auxiliarytask described in Section 2.1. Pixel control learns individual policies to maximally change the pixels of each grid(c)by dividing the input image into n × n grids. Thus, LQ

In this equation, θ− represents the model parameters before the model update.

26

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

The AS loss function is deﬁned as the sum of the losses
in Eqs. (3), (4), as

Algorithm 1 Proposed method
1: //Assume global shared parameters θ and θAS and global

shared counter T = 0


2: //Assume worker-speciﬁc parameters θ  and θAS
3: Initialize local step counter t ← 1
4: repeat
5:
Reset gradients: dθ ← 0 and dθAS ← 0
6:Synchronize worker-speciﬁc parameters θ = θ and

LAS = LASv + LASp .

Eqs. (3), (4), (5) show that AS is a DRL agent that selects auxiliary tasks. In other words, AS is constructed asa model with diﬀerent weights than the main task model
for solving video games, and is optimized by the same reward as the main task to achieve the selection of auxiliarytasks to improve scores in video games.


θAS
= θAS
tstart = t
Get state st
repeatPerform action at according to policy π(at |st , θ )Receive reward rt and new state st+1
Store experience (st+1 , rt , at ) in replay buﬀer
t←t+1
T ←T +1
until terminal st or t − tstart == tmax
Execute each auxiliary task with the experiences stored
in replay buﬀer
17:
Execute auxiliary selection with the experiences stored
in replay buﬀer
18:
Accumulate gradients dθ w.r.t. θ19:
Accumulate gradients dθAS w.r.t. θAS
20:
Perform asynchronous update of θ using dθ and of θASusing dθAS
21: until T > Tmax

7:
8:
9:
10:
11:
12:
13:
14:
15:
16:

3.4


denotes the auxiliary selection model
eters and θAS , θASparameters. Our method utilizes A3C as the training algorithm, which involves distributed learning by multiple
workers and asynchronous updating of model parameters.
Therefore, each worker has its own local model parame
and local step t. In addition, the global modelter θ , θASparameters θ, θAS and global step T are shared among
workers.
First, we synchronize the network parameters of eachworker θ and θASwith the shared parameters θ and θAS ,respectively. Then, agents of each worker repeatedly takeactions in an environment by following policy π(at |st , θ )until reaching a termination condition or tmax steps. The
experiences (st+1 , rt , at ) are stored in a replay buﬀer.

and CRP of AS are included in the loss function L of thismethod. Therefore, the learning of the AS network using
the loss function L is performed so that the loss is close
to zero, including the AS outputs CVR , CPC , and CRP .

Next, we execute the auxiliary selection and the threeauxiliary tasks in turn, and we compute the gradients dθ
and dθAS shown in Eqs. (2), (5). Using these gradients,
we update the global model parameters θ and θAS . This

That is, AS is learned so that CVR , CPC , and CRP are
zero. Thus, we deﬁne diﬀerent loss functions for learning
the AS network, and learn AS independently of the main
and auxiliary tasks.
The AS loss function is formulated using the loss
function of the state value VAS (s, θAS ) and the policy
πAS (a|s, θAS ), as

LASp = − log(πAS (a|s, θAS ))A(s, a)− αH(πAS (·|s, θAS )).

Algorithm

The processing ﬂow of our method is shown in Algorithm 1. Here, θ, θ denotes the main task model param-is the loss of PC, which is the loss of n-step Q-learning for
grid c. Our method achieves auxiliary task selection bymultiplying the loss of each auxiliary task used to update
the network parameters by the binary weights obtained in
AS. As shown in Eq. (2), the binary weights CPC , CVR ,−LASv = (r + γVAS (st+1 , θAS) − VAS (st , θAS ))2 .

(5)

update is performed asynchronously on each worker andis repeated until Tmax steps to adaptively select and learn
auxiliary tasks.

4.

Experiments(3)

We used the DeepMind Lab21)for the evaluation of our
method. DeepMind Lab mainly contains three games:i) nav maze static 01 (maze), ii) seekavoid arena 01(4)(seekavoid), and iii) lt horseshoe color (horseshoe).
The maze is a ﬁrst-person maze exploration game. The
agent receives +1 for each apple obtained along the way
and +10 for reaching the goal, and competes for the highest score within a given time. There are six actions thatcan be selected by the agent: move left, move right, move
forward, move backward, move parallel to the left, andmove parallel to the right. The seekavoid is a game in−are the network parameters before the AS netwhere θASwork update, and r is the reward of the main task (scoreof the video game). Also, entropy H(πAS (·|s, θAS )) is aterm to facilitate the search such that the network parameters do not converge to a local solution, and α is thescale parameter of entropy H(πAS (·|s, θAS )).

27

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

D QDYBQD]HBVWDWLFBE VHHNDYRLGBDUHQDBF OWBKRUVHVKRHBFRORU

Fig. 2 Game score transition during training in DeepMind Lab
Table 2 The number of times each auxiliary task was
selected in an episodewhich agents earn +1 for each apple and −1 for eachlemon they acquire, and compete to earn the highest scorewithin a time limit. There are six actions that can be selected by the agent: move left, move right, move forward,
move backward, move parallel to the left, and move parallel to the right. The horseshoe is a ﬁrst-person shooter
game. The agent attacks enemies appearing on the stage
with a laser and earns +1 by defeating them, and competes for a score within a time limit. The agent can choose
from seven actions: move left, move right, move forward,
move backward, move parallel to the left, move parallelto the right, and attack.
We compared our method with the following baselines:
UNREAL: Three auxiliary tasks are used for training.
PC, VR, and RP: Each uses its own auxiliary task fortraining.
The common hyperparameters during training were uniﬁed. The training steps were 5.0 × 107 steps for maze and

Env.
maze
seekvoid
horseshoe

PC was inadequate for seekavoid because the pixel values
changed signiﬁcantly even when negative rewards wereobtained. RP was also not eﬃcient because this environment oﬀers dense rewards. In contrast, UNREAL
and VR achieved higher scores. Surprisingly, VR outperformed UNREAL. In contrast to this result, our method
also achieved the same performance as VR and achieved
higher performance with fewer training steps than VR.
horseshoe. Figure 2(c) shows the scores for horseshoe.
Here, PC had the highest score, since the actions that
defeat enemies change pixel values signiﬁcantly. However, UNREAL outperformed the other methods, and our
method achieved the same performance as UNREAL.

seekavoid, and 1.0 × 108 steps for horseshoe.
4.1

Auxiliary task
PC
VRRP
435.4 487.8 369.0
0.3
300.0
0.0
8545.1 14.1 8998.2

Comparison according to game score

4.2

Figure 2 shows the score transition during the training of the comparison methods in DeepMind Lab. Here,Analysis of the selected auxiliary tasks

The number of actions selected by the auxiliary selection during one episode of each game is shown in
Figure 3. Here, the horizontal axis represents the actions {CPC , CVR , CRP } output from the auxiliary selection, and the vertical axis represents the number of times
each action was selected. From left to right: (a) mazewith 5.0 × 107 steps, (b) seekavoid with 5.0 × 107 steps,and (c) horseshoe with 1.0 × 108 steps. Table 2 lists thenumber of times each auxiliary task was selected in an
episode. We calculated the number of times the auxiliary task was selected as the average of 50 episodes. The
number of action steps in an episode was 900 for maze,
300 for seekavoid, and 9,000 for horseshoe.
The results for maze show that all auxiliary tasks wereequivalently selected. Because the appropriate auxiliary

the horizontal axis indicates the number of global steps to
update the network parameters, and the vertical axis indicates the score for each task. From left to right: maze,seekavoid, and horseshoe.
maze. Figure 2(a) shows the scores for maze. UNREAL and PC achieved higher performances, while thescores of VR and RP were almost zero. This means that
VR and RP did not improve the main task. The PC
encouraged the agent’s action of exploring the maze bychanging the pixel values. This enabled an agent to movein every corner of the maze environment. Our method
also achieved the same score as UNREAL and PC.
seekavoid. Figure 2(b) shows the scores for seekavoid.

28

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）{CPC , CVR , CRP }(a) maze{CPC , CVR , CRP }(b) seekavoid{CPC , CVR , CRP }(c) horseshoe

Fig. 3 The number of selections per action by auxiliary selection during one episodeselect auxiliary tasks that contribute to training the main
task.
4.3

Dynamic selection of auxiliary tasks according to training stages

In this Section, we analyze the percentage of auxiliarytasks selected for each training stage in order to conﬁrm
whether the optimal selection of auxiliary tasks based on
the training stage is achieved. The percentage of auxiliarytasks selected during one episode for each training phasein seekavoid is shown in Figure 5. Here, we studied
0.2×107 , 0.5×107 , 1.0×107 , and 5.0×107 training steps.

Fig. 4 Game scores in horseshoe with additional auxiliary task combinations

tasks for the maze task were UNREAL or PC, our method
equally selected all auxiliary tasks. In seekavoid, ourmethod stably selected the value function replay. Since
these results correspond to the results in Figure 2(b), ourmethod only selects auxiliary tasks that contribute to thetraining of the main task. In horseshoe, pixel control andreward prediction were often selected. Although the bestscore was achieved by UNREAL, auxiliary selection forhorseshoe did not select value function replay. To ana-

The selection rate of the auxiliary task is the average of
50 episodes of the number of selections per episode using
the model with each step trained.
From Figure 5, we can see that all auxiliary tasks wereselected to the same degree in 0.2×107 steps and 0.5×107
steps. Here, UNREAL scored about 25 and 30 points
higher for the 0.2 × 107 and 0.5 × 107 steps. From theseresults, we can assume that, in the early learning phase
up to 0.5 × 107 steps, our method selects each auxiliary

lyze the reason of this selection, we conducted additional
experiments. In addition to the results of the baselines
shown in Figure 2(c), we added the following baselines:
A3C-LSTM (without auxiliary tasks), and PC+RP (usespixel control and reward prediction). Figure 4 shows thescores of each baseline and our method. We can see herethat the score of VR was lower than that of A3C, andthat PC+RP achieved the same score as UNREAL andour method. Therefore, our method successfully removes

task just as well as UNREAL, which uses all auxiliarytasks to the same extent. On the other hand, we canconﬁrm that our method preferentially selects VR in the
1.0 × 107 step. UNREAL and VR had the highest scoreof about 30 points in the 1.0 × 107 step, so we can assume
that our method is able to select VR during the training
stage where VR is eﬀective. Here, only VR was selected
for the 5.0 × 107 step. In the 1.0 × 107 step of the scoregraph, there is a large diﬀerence of about 7 points between
the scores of UNREAL and VR, with VR obtaining thehighest score. Therefore, we consider that the number ofthe value function replay from the training of horseshoe.
The above ﬁndings demonstrate that our approach can

29

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）     
  

  

  

  

Fig. 5 Percentage of auxiliary task selection during one episode according to training stages (seekavoid)

ture work, we plan to apply our method to other auxiliary
learning methods and to experiment with various tasks.

times VR is selected by the auxiliary selection increased
between 1.0 × 107 steps and 5.0 × 107 steps. These results indicate that auxiliary selection is able to select the
combination of auxiliary tasks with the highest score in
the main task in accordance with the training stage ofthe main task.

5.

Acknowledgements
This work was supported by JST SPRING, Grant
Number JPMJSP2158.

References

Conclusion

1) I. Kokkinos: “Ubernet: Training a Universal Convolutional
Neural Network for Low-, Mid-, and High-level Vision using Diverse Datasets and Limited Memory”, Proc. of theIEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), pp.6129-6138 (2017).

In this paper, we proposed auxiliary selection, which
selects eﬀective auxiliary tasks to be used during learning of the main task. Auxiliary selection is designed asa deep reinforcement learning agent that controls the bi-2) M. Teichmann, M. Weber, M. Zoellner, R. Cipolla, and R.
Urtasun: “MultiNet: Real-time Joint Semantic Reasoning forAutonomous Driving”, Proc. of the IEEE Intelligent Vehicles
Symposium (IV), pp.1013-1020 (2018).

nary weights of the auxiliary task in order to improve theaccuracy of the main task. We applied our method to
UNREAL, which introduces unsupervised auxiliary tasks
in video game strategy, to allow dynamic selection of auxiliary tasks. This eliminates the need to manually design
auxiliary tasks for each main task and enables more eﬃcient learning of the main task. From experiments using
DeepMind Lab, we conﬁrmed that our method achievesa score that is equivalent to the optimal combination of3) A. Kendall, Y. Gal, and R. Cipolla: “Multi-task Learning using
Uncertainty to Weigh Losses for Scene Geometry and Semantics”, Proc. of the IEEE/CVF Computer Vision and Pattern
Recognition Conference (CVPR), pp.7482-7491 (2018).
4) Y. Qian, J.M. Dolan, and M. Yang: “DLT-Net: Joint Detection of Drivable Areas, Lane Lines, and Traﬃc Objects”, IEEE
Trans. on Intelligent Transportation Systems, vol.21, no.11,
pp.4670-4679 (2020).
5) R. Collobert and J. Weston: “A Uniﬁed Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning”, Proc. of the International Conference on Machine Learning (ICML), pp.160-167 (2008).

auxiliary tasks for each environment. The analysis of theauxiliary tasks selected by auxiliary selection showed that
our method can improve the performance of the main task
by selecting the appropriate auxiliary task for the video
game to be solved. In addition, by analyzing the auxiliary tasks selected at each training phase, we found that
our method selects the most appropriate auxiliary task
according to the training phase of the main task. As fu-

6) X. Liu, P. He, W. Chen, and J. Gao: “Multi-Task Deep Neural
Networks for Natural Language Understanding”, Proc. of theAnnual Meeting of the Association for Computational Linguistics (ACL) (2019).
7) Lukas Liebel and Marco Körner: “Auxiliary Tasks in Multitask Learning”, arXiv preprint, arXiv:1805.06334 (2018).
8) M. Jaderberg, V. Mnih, W.M. Czarnecki, T. Schaul, J.Z. Leibo,

30

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Hidenori ITAYA (Student Member)

D. Silver, and K. Kavukcuoglu: “Reinforcement Learning with
Unsupervised Auxiliary Tasks”, Proc. of the International Conference on Learning Representations (ICLR) (2017).

He received his B.E. in Computer Sciencefrom Chubu University, Japan in 2019. Hereceived his M.S. degree from the same institution in 2021. He has worked at DENSOWAVE incorporated since 2021, and is currently pursuing his Ph.D. from Chubu University. He is a member of the IIEEJ.

9) Z. Zhang, P. Luo, C.C. Loy, and X. Tang: “Facial Landmark
Detection by Deep Multi-task Learning”, Proc. of the European Conference on Computer Vision (ECCV), pp.94—108(2014).
10) P.W. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard,
A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu,
D. Kumaran, and R. Hadsell: “Learning to Navigate in Complex Environments”, Proc. of the International Conference on
Learning Representations (ICLR) (2017).

Tsubasa HIRAKAWA
He received his Ph.D. in Computer Sciencefrom Hiroshima University, Japan in 2017.
From 2017 to 2019, he was a researcher fellow
at the Chubu University, and he has been a
specially appointed associate professor at the
Chubu Institute for Advanced Studies, Chubu
University, Japan since 2019. He has also
been a lecturer in the Department of Computer Science, Chubu University, Japan since2021. He was a Fellowship of the Japan Society for the Promotion of Science from 2014
to 2017, and he was a visiting researcher at
ESIEE Paris, France, in 2014 and 2015.

11) B. Kartal, P. Hernandez-Leal, and M.E. Taylor: “Terminal
Prediction as an Auxiliary Task for Deep Reinforcement Learning”, Proc. of the AAAI conference on Artiﬁcial Intelligenceand Interactive Digital Entertainment (AIIDE) (2019).
12) P. Hernandez-Leal, B. Kartal, and M.E. Taylor: “Agent Modeling as Auxiliary Task for Deep Reinforcement Learning”, Proc.
of the AAAI conference on Artiﬁcial Intelligence and Interactive Digital Entertainment (AIIDE) (2019).
13) N. Justesen, P. Bontrager, J. Togelius, and S. Risi: “DeepLearning for Video Game Playing”, IEEE Trans. on Games,
vol.12, pp.1-20 (2017).
14) V. Mnih, A.P. Badia, M. Mirza, A. Graves, T. Lillicrap,
T. Harley, D. Silver, and K. Kavukcuoglu: “Asynchronous
Methods for Deep Reinforcement Learning”, Proc. of International Conference on Machine Learning (ICML), pp.1928-1937(2016).

Takayoshi YAMASHITAHe received his Ph.D, in Computer Sciencefrom Chubu University, Japan in 2011. Heworked at OMRON Corporation from 2002
to 2014 and was a lecturer in the Department of Computer Science, Chubu University, Japan from 2014 to 2017, where he wasalso an associate professor from 2017 to 2021.
He has been a professor at the same institute since 2021. His current research interests include object detection, object tracking,human activity understanding, pattern recognition, and machine learning. He is a memberof the IEEE, the IEICE, and the IPSJ.

15) V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness,
M.G. Bellemare, A. Graves, M.A. Riedmiller, A.K. Fidjeland,
G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou,
H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis,:“Human-level Control through Deep Reinforcement Learning”,
Nature, vol.518, no.7540, pp.529-533 (2015).
16) Y. Teh, V. Bapst, W.M. Czarnecki, J. Quan, J. Kirkpatrick,
R. Hadsell, N. Heess, and R. Pascanu: “Distral: Robust Multitask Reinforcement Learning”, Proc. of the Neural Information
Processing Systems (NeurIPS), pp.4496-4506 (2017).
17) M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave,
T. Wiele, V. Mnih, N. Heess, and J.T. Springenberg: “Learning
by Playing – Solving Sparse Reward Tasks from Scratch”, Proc.
of the International Conference on Machine Learning (ICML)(2018).

Hironobu FUJIYOSHIHe received his Ph.D. in Electrical Engineering from Chubu University, Japan, in 1997.
From 1997 to 2000, he was a post-doctoral fellow in the Robotics Institute of Carnegie Mellon University, Pittsburgh, PA, USA, working on the DARPA Video Surveillance andMonitoring (VSAM) eﬀort and the humanoid
vision project for the HONDA Humanoid
Robot. He is now a professor of the Department of Robotics, Chubu University, Japan.
From 2005 to 2006, he was a visiting researcher at the Robotics Institute, Carnegie
Mellon University. His research interests include computer vision, video understanding
and pattern recognition. He is a memberof the IEEE, the IEICE, the IPSJ, and theIEEE.

18) Y. Du, W.M. Czarnecki, S.M. Jayakumar, R. Pascanu, and B.
Lakshminarayanan: “Adapting Auxiliary Losses using Gradient Similarity”, arXiv preprint, arXiv:1812.02224 (2018).
19) X. Lin, H.S. Baweja, G.A. Kantor, and D. Held: “Adaptive Auxiliary Task Weighting for Reinforcement Learning”,
Proc. of the Neural Information Processing Systems (NeurIPS),
pp.4772-4783 (2019).
20) G. Hinton, O. Vinyals, and J. Dean: “Distilling the Knowledge in a Neural Network”, Proc. of the Neural Information
Processing Systems (NeurIPS) deep learning workshop (2015).
21) C. Beattie, J.Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright,
H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J.
Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A.
Bolton, S. Gaﬀney, H. King, D. Hassabis, S. Legg, and S. Petersen: “DeepMind Lab”, arXiv preprint, arXiv:1612.03801(2016).

(Received November 9, 2023)(Revised April 26, 2024)

31

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）
IIEEJ

Paper

-- Special Issue on Journal Track Papers in IEVC2024 --Real-Time Intuitive Interaction and Realistic Illumination for CT Volume Rendering
Kousuke KATAYAMA† , Toru HIGAKI† , Kazufumi KANEDA† (Member), Bisser RAYTCHEV† ,Wataru FUKUMOTO†† , Hidenori MITANI††† Graduate School of Advanced Science and Engineering, Hiroshima University ,†† Graduate School of Biomedical and Health Sciences, Hiroshima University<Summary> We developed a real-time, intuitive interaction and photo-realistic illumination method
for CT volume rendering. Our approach involves auto-stereoscopic display and hand-sensor-based gesturecontrol, as well as lightweight and eﬀective illumination, and a fast-sampling algorithm that enables them tobe rendered in real-time. Consequently, our rendering method achieved render volume data obtained fromgeneral CT examinations at real-time 4K stereo view, allowing intuitive comprehension of the 3D structure,and providing the realism required not only for diagnostics but also for educational materials and forensic
evidence.
Keywords: volume rendering, 3D user interaction, photo-realistic rendering

1.

ever, Monte Carlo method rendering has a lag until theimage is cleaned up and is not suitable for combination
with interactions that are in constant motion.
We propose to combine realistic appearance and intuitive comprehensibility with realistic lighting and new
real-time interactions, and to optimize a direct volume
rendering method to achieve this combination in a nonMonte Carlo method. The contribution of our proposed methodology can be summarised in three items:(1) we incorporated a real-time realistic rendering technique used in surface rendering to the volume rendering
method, (2) we improved the ray-marching acceleration
with modiﬁed mipmap creation, (3) we proposed the useof a spatial reality display and hand-tracking sensors forsuperior interaction.

Introduction

While CT volume renderings are useful for diagnostic
purposes, we focus on the use of CT volume renderings
for the visualization of forensic evidence and as a medical educational tool. CT volume rendering for forensic
evidence is used to visualize CT scans taken for screening
and preservation of evidence before autopsy. CT volume
rendering for medical educational purposes is also valuable because it is easier than working with specimens orreal objects and can be viewed from a greater degree inany angle than photographs. For these CT volume rendering applications, it is important to be able to representa realistic appearance and to be intuitively understandable by a non-specialist viewer.
Various methods have been proposed for CT volume
rendering in the past1)there are many methods for data
feature enhancement of CT volumes, and the method canbe selected according to the inside and outside of the data
space and the purpose. Representing a realistic appearance has also been achieved to some extent2). However,while real-time interaction is important for intuitive comprehensibility, it has not been possible to achieve both
realistic appearance representation and real-time interaction. This is because the computational cost of volume
rendering is still high. In some of the most realisticlooking, real-time examples to date, Monte Carlo techniques3)have been incorporated into volume rendering toensure real-time rendering and realistic lighting. How-

2.

Related Works

Omori et al. developed a method for displaying medical CT volume data in 3D using spatial reality display4).
Although real-time 3D and 4D displays are achieved withthis method, only simpliﬁed lighting is applied to volume
rendering, resulting in low photorealism.
Photorealistic rendering aims at a realistic representation of the object and the physical light transport results
are rendered to be the same as in the real world. Shading
techniques such as BRDF are used to make the reﬂective
properties of light photorealistic, as well as global lighting
and image-based lighting techniques. These techniques
are generally used for surface rendering. Cinematic ren-32

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig. 1 Real-time realistic illumination for contrast-enhanced neck CT volume rendering
using our real-time rendering method(b) Base Data
Scalar CT Data5123(d) Alpha Mipmap
for Adaptive Sampling Step Control

33MB

Alpha Value Mipmapping

268MB(example)(g) Final Image(e) Cross-Clipping & Shadowing

Magnitude

Normal and Magnitude
Scalar 5123 × 4(f) Right or Left Eye Image
Post Processing

4MB

Alpha
Mipmap Level 1 Level 2
2563 (1/8)
1283 (1/64)(c) 2D Transfer Function
for Boundary Emphasize

Linear Regression(a) Data
Pre Processing

Ray Casting
MethodNormal Buffer
Depth Buffer Rendered Image
Disk Sampling

Parallax
Processing

Stereo Image

Data Value

268MB
268MB
x4

2D Transfer Function
2048 x 512 RGBA

Re-use Alpha
Mipmap Level 2

4MB
ClippedAlpha Volume1283

Light Volume1283 RGBAO Buffer SSS Buffer

Fig. 2 Rendering pre-processing and post-processing system pipeline overview

dering5)and Exposure Render2)have been proposed as realistic volume rendering methods. These methods applyMonte Carlo rendering techniques3)to volume rendering,which is computationally expensive and unsuitable forreal-time rendering.
In order to improve the understandability of volume
data, non-photorealistic rendering methods may be usefulas well as realism. In volume illustration6), the normal direction of the volume and boundary surface intensity canbe used to enhance contours and boundaries and improvethe visibility of the geometry.
Studies on speeding up volume rendering have utilized
the hierarchical data structure of volumes7)and, more recently, image reconstruction using neural networks to reduce computational costs8).

3.

(b) Real-time colored shadowing technique(c) Screen space-based illumination technique thatavoids expensive calculations(d) Adaptive sampling step control for direct volume rendering based on mipmap volume(e) Intuitive interaction provided by a free-viewpoint autostereoscopic display and optical tracking sensors
Figure 1 shows a rendering with realistic lighting basedon our approach. Figure 2 shows the rendering systemusing a 5123 voxels volume as an example (Fig. 2-(b)). It
shows the data and its ﬂow used for pre-processing (Fig.
2-(a)) and direct volume rendering, and the image buﬀer
used for post-processing (Fig. 2-(f)).
3.1

Shading

In making volume rendering realistic, there is a weakness in the representation of volume rendering, which isthat the outlines and boundaries of objects represented byvolume rendering are unclear. This depends on the data,
but if the transparency gradient of the transfer function
according to the CT value is steep enough to make it
clear, artifacts will be created.
Therefore, to improve visibility, we have incorporated

Our Approach

Our volume rendering combines realistic illumination
and real-time interaction. Our approach includes the following:(a) Physically based shading with visibility control thatchanges transparency according to sight direction,
normal, and normal magnitude.

33

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

a volume illustration method6)of controlling transparency
by line of sight, normal, and its magnitude, withoutchanging the transparency gradient according to CT
value. By increasing the original transparency according to the magnitude, all areas except the boundary are
made transparent and the boundary is emphasized. A 2D
transfer function is used to consider magnitude, in which
the vertical axis of magnitude is added to the horizontal
axis of CT value in Fig. 2-(c). Then, an outline clariﬁcation process is performed during shading by opacifying
the outline by the angle between the line of sight and thenormal. The shading alpha and RGB values applied toeach sample in direct volume rendering are summarizedin Eq. (1) and (2). Where f is a 2-dimensional transfer
function of CT-value and its gradient magnitude, s is CTvalue, ⃗n is CT-gradient calculated by linear regression9)(Fig. 2-(b)), and d⃗ is ray direction, pm and ps are parameters for controlling metallicity and smoothness, respectively. PBS (Physically Based Shader) is a reused Unity
Standard Shader10)function. Equation (1) contains an alpha value increase to enhance the silhouette lines of theobject6). p1 and p2 are the silhouette enhancement parameter (Example: p1 = 8.0, p2 = 12).
))p2
(
(
⃗n
α = fα (s, |⃗n|) 1 + p1 1 − d⃗ ·|⃗n|(
RGB = PBS⃗
frgb (s, |⃗n|), d,⃗n, pm , ps|⃗n|(a) with SSAO(b) with SSAO and SSSSS
Fig. 3 Screen space post-processingseach wavelength component of the color, respectively. V
is the brightness function of light.
Lmid = Lin (1 − Cα (1 − Ci ))(1)

Lout = Lmid

)
(2)

3.3

V (Lmid )(1 − Cα )V (Lmid )

(3)

(4)

Screen space based technique

Post-processing performed in screen space is useful because it can be done in constant time regardless of theresolution of the volume. We process ambient occlusionand subsurface scattering in screen space (Fig. 2-(f)),
which could not be considered computationally expensive
in volume space to compute during direct volume rendering. Screen Space Ambient Occlusion (SSAO) is computed using common method12)with depth buﬀer and normal buﬀer. Screen Space Sub-Surface Scattering (SSSSS)
technique is commonly used in deferred rendering. However, since our method is a forward rendering, we used a
simpliﬁed version of the former SSSSS method13),14). Our
method does not use an incident light buﬀer, but treats
the forward rendering image as if it were incident light.
The Sub-Surface Scattering (SSS) light is computed witha diﬀuse proﬁle13),14)and disk sampling, and the ﬁnal image is maxed with the scattered light and the forward
rendered image (SSSSS = max(SSS, image)). In thismethod, only the areas brightened by SSS are pseudorepresented. Since SSS originally shows diﬀerences well at

3.2 Shadowing
As a shadowing method, a volumetric lighting model11)
is used that is capable of real-time processing. As in thelight volume light shown in Fig. 2-(e), the light volume
has 1/4 resolution on each side and progresses throughthe entire light one step per frame, so the performance impact is minimal (for a source volume of size 5123 , it can beprocessed in less than 1 ms per frame). The light volume
also illuminates the cross-section by using a cross-clippedalpha volume, which is described later in the interaction
section. We have included a process that considers thecolor of translucent voxels during shadow computation toincrease the cognitive eﬀectiveness of the shadow representation. To avoid unnatural results, the process applies
voxel color based on brightness values. Equation (3) and(4) represent the calculation of the color shadow. Where
Lin , Lmid and Lout are light of each length, before,
middle result, and after passing through the substance,
respectively. Cα and Ci are substance color, opacity, and34

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

mipmapsampling point

raykipping
blank s□ α = 0.0

modified ste
standarize
d step s■ 0.0 < α ≦ 0.125

p size■ 0.125 < α < 1.0■ α = 1.0

Fig. 4 Adaptive sampling step control

in the step control described below. These processes must
be repeated when the transfer function changes, but since
they take less than 10 ms for a base volume of size 5123 ,they do not aﬀect rendering unless the transfer function
is constantly changing.
Figure 4 shows the correspondence between the sampling width of the adaptive sampling step control and thearea-opacity value. There are three main levels of control
depending on the area-opacity value. Hereafter, level 1(1/2 of each side) of the alpha volume is represented as
A1 , level 2 (1/4 of each side) as A2 , and the alpha of theoriginal volume as A0 .

the edges of shadows, this method is also eﬀective for representing translucent materials such as soft tissues whencompared with and without SSS as shown in the ﬁgure.
Finally, parallax processing is added to the stereo rendering as shown in Fig. 2-(g). Figure 3 shows the eﬀects of
SSSSS and SSAO. Small shadows caused by ﬁne surface
irregularities are produced by SSAO, and the shadows arecolored and softened by SSSSS.
3.4

Adaptive sampling step control

In direct volume rendering, most of the computational
cost of a single ray lies in the ray-marching part. The
ray-marching parts have two main computational costs:
sampling the volume data and shading the samples. Since
shading is associated with the valid samples obtained insampling, we focused on reducing the number of samplings. We developed the adaptive sampling step control.
Adaptive sampling step control is a method of determining the sampling step interval using area-opacity values
expressed in mipmap alpha volume. By referring to themipmap values, a wide-area sample can be obtained todetermine the need for sampling and to omit sampling.
Compared to similar methods that use mipmaps7), thismethod is superior in that it can take larger steps due
to modiﬁcations to the generation of the mipmap, and it
can accelerate steps even in semi-transparent regions.
The data structure of the mipmap is the mipmap of thealpha volume when a 2D transfer function is assigned to
the CT volume, as depicted in the data ﬂow in Fig. 2(b) (c) (d). We have made sure that when mipmapping,
the lower level mipmap uses the maximum value rather
than the average of the upper level 8 voxels. This is
to avoid missing small objects. Also, after mipmapping,each voxel has applied a type of morphological processing
so that it retains the maximum value of the surrounding
26 voxels. This is to ensure that no over-skipping occurs

3.4.1

Blank skippingFirst, when α = 0 is sampled at A2 , The sampled areaof the A2 voxel and the surrounding area is completelytransparent due to maximum mipmapping and morphological processing. Thus, signiﬁcant blank skipping ispossible. The step size can be set 8 times wider thanthe usual 1 voxel wide step. This reduces the sampling oftransparent regions. This process does not aﬀect image
quality at all. We use A2 instead of A1 or A3 because
A2 was found experimentally to be the most eﬃcient.
In addition, the use of additional A1 or A3 was almost
ineﬀective because it would require additional memory
access.
3.4.2

Modiﬁed step size

Now, noting the fact that regions with high alpha values (α ∼ 1) can be rendered faster with standard step
size because the opacity accumulation ﬁnishes earlier, we
ﬁnd that a speedup is needed in translucent but neartransparent regions (α ∼ 0). To solve this problem, wechange the step size according to the alpha value of themipmap. Changing the step size requires a change inthe ray-marching integration formula for consistency inalpha value integration. We modify the alpha value α

35

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Spatial Reality Display

Ultraleap 3diFig. 5 Interaction devices overview

used in the integration formula according to the step sizer as shown in Eq. (5). Where α is sampled substancealpha, and α∗ is the substance alpha value used in the
integration formula.
α∗ = 1 − (1 − α)r(5)Fig. 6 Interaction of clipping cross-section plane

In changing the step size, sampling can be skipped forregions with low alpha values because they have less impact on the image and their transfer function settings
mean that they are not places to be focused on. However,
we need to make sure that they are low alpha areas sothat we do not skip sampling important high alpha areas.
Our mipmap can guarantee a maximum alpha value inthe neighborhood. We change the step size according to
the alpha value of our mipmap. We have experimentally
determined step size r so that speedup can be achieved
with minimal impact on image quality. Equation (6) is
the formula for determining the step size r. Where 2 isthe maximum step size when the alpha is almost 0, and0.125 is the maximum alpha value at which the step sizechange is made.
r =1+

2
max(0.125 − A2 , 0)0.125

MR, which require the user to mount a device. Therefore, we focused on touchless interactions. For our interactions, we use the Spatial Reality Display15)(SONY Corporation) and the Ultraleap3di16)(Ultraleap Inc.) (shownin Fig. 5). This glasses-free stereoscopic display uses abuilt-in camera and face recognition to estimate the position of the eyes, enabling free-viewpoint autostereoscopic
viewing as if it were a real 3D photograph. Since the position of the eyes is estimated, it is possible to actually
look into the projected image by looking into the eyes.
The Ultraleap3di is an optical hand sensor, which canrecognize the detailed position of both ﬁngers in the air
above the sensor. It allows for non-contact operation, so
if the sensor is placed in front of the display, the user canintuitively manipulate the 3D image by holding his or her
hands over it.
These two devices are incorporated with the renderingof our approach. Although there is no active interaction with the display, the free-viewpoint autostereoscopic
viewing allows intuitive perception of the stereoscopic effect, and the viewpoint is ﬁne-tuned by peering into thedisplay. Real-time, 4K stereo rendering is required, so
faster rendering is a must. Hand sensors can be used todetermine the rough angles of the volume display. It canalso be used to visualize a cross-section of the volume.
The cross-section of the volume is the plane that is identical to the palm of the hand (shown in Fig. 6). Thismethod of cross-sectional manipulation is more intuitiveand easier than using a mouse. This is because the palmof the hand can be determined all at once, whereas manipulating a plane requires six-dimensional manipulation(6)

3.4.3 Standard step size
Areas with high alpha values (α ∼ 1) do not require acceleration because the opacity accumulation is completed
early and can be rendered faster with standard step size.
The standard step size is the length of the shortest sideof the voxel, to account for the case where the voxel is
not a cube. If you want to further reduce noise or artists,
you can further reduce the step size, but it is not recommended because of the high computational cost.
3.5 Interaction
Considering its use as an educational or forensic resource, it is important that interactions be easy to perform. Therefore, complex operations such as using amouse and keyboard are inappropriate, as are VR and

36

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）
Table 1 Performance of 4K stereo rendering(a) opaque object, (b) translucent object, (c) translucent air

with Control

without Control

no shading
(a)
(b)
(c)

194 fps (5.2 ms)
100 fps (10.0 ms)
82 fps (12.2 ms)
34 fps (29.4 ms)

88 fps (11.4 ms)
62 fps (16.1 ms)
44 fps (22.7 ms)
14 fps (71.4 ms)

Table 2 Performance of realistic illumination technique withlower performance computer

Fig. 7 Various settings of transfer functionof position and rotation. To accurately reﬂect lighting
on the cross-section as well, a cross-sectional volume isprovided in the Fig. 2-(e) data ﬂow that sets the alphavalue to 0 for areas that become transparent due to crosssectioning before the light volume.

4.

Settings

Implements and Experiments

Settings

Frame Rateno shading
Phong shading
PBS shading
PBS and Shadowing
PBS, Shadowing and SSAO
PBS, Shadowing, SSAO and SSSSS

82 fps (12.2 ms)
67 fps (15.0 ms)
55 fps (17.9 ms)
50 fps (19.8 ms)
35 fps (28.7 ms)
26 fps (36.8 ms)

Table 3 Performance of adaptive sampling step control components with lower performance computer

We implemented our rendering system on Unity2020.
Direct volume rendering and post-processing are done bycustom shaders processed on the GPU. Normal and magnitude calculations, mipmapping, and pre-processing ofcross-sections and light volumes are also handled on the
GPU by compute shaders. Free viewpoint autostereoscopic viewing and optical hand sensor control are doneby an SDK provided by the manufacturer.
We measured the performance of the rendering systemwe created and the adaptive sampling step control in four
diﬀerent settings. The input was volume data of the Stanford Bunny17)(512 × 512 × 361) of ceramics taken by CT,
the computer used was CPU Core-i9-11900 CPU, GPU
RTX-3090, and we measured the performance with two
4K stereo renderings using Spatial Reality Display. Scene(a) assigns full transparency to air and opacity to ceramic,
with the table at a slightly reﬂective transparency. Scene(b) uses magnitude to make only the material boundary appear translucent. Scene (c) assigns translucency
to both air and ceramic, with air and ceramic changing
color. Scene (no shading) has the same transfer function
settings as Scene (a) and is an emission that displays thematerial colors as they are without shading. Figure 7
and Table 1 show the results of measurements with andwithout adaptive sampling step control for the four scenes
and adaptive sampling step control. In all conditions, theadaptive sampling step control succeeds in increasing the
speed, achieving real-time speeds of 30 fps or more.
We measured the computational load for each realistic
illumination technique in order to examine the choice ofeach illumination technique when rendered using a lower
performance computer. We used a computer: CPU Core-

Components

Scene (a)Scene (c)without Control
Blank Skipping
Modiﬁed Step Sizeboth Controls

16 fps (61.8 ms)
45 fps (22.3 ms)
26 fps (38.4 ms)
49 fps (20.4 ms)

15 fps (66.0 ms)
12 fps (83.4 ms)
28 fps (35.8 ms)
30 fps (33.2 ms)

i7-4770K and GPU GTX-970, rendering a single 4K image. Volume and transfer function settings are identicalto Scene (a) in Fig. 7. Table 2 shows the measurementswith 6 diﬀerent illumination settings. You can see howlong each eﬀect took by comparing the resulting frame
times with the previous and following settings. The lightest eﬀect was shadowing (1.9 ms) and the heaviest was
SSSSS (8.1 ms). Note that the processing costs for shading, shadowing depends on 2D transfer function, volume
resolution, and rendering resolution, and post-processing
mainly depends on rendering resolution.
We measured the performance of each of the adaptive
sampling step controls blank skipping and modiﬁed step
size. To measure the diﬀerences, we render single 4K
image using a computer with the same lower-performanceas in the experiment above. Volume and transfer function
settings are the same as in Fig. 7 (a) and (c). Table 3
shows the measured results. Blank skipping works well insettings with a lot of blanks, and modiﬁed step size works
well in settings with a lot of high translucent space. In
Scene (c), blank skipping rarely works, the result withblank skipping only is slower than without control due tothe cost of accessing the mipmap.
The rendering performance for human body CT shownin Fig. 8 and Table 4. A volume data of contrastenhanced body CT (512 × 512 × 1234 voxels) was used

37

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）(a) Skin(b) Trans.

(c) Clipping(d) Bones

Fig. 8 Rendering results of clinical CT(a) Conventional rendering

Table 4 Performance of 4K clinical CT rendering
Settings

Frame Rate(a) Skin Surface(b) Translucent Skin(c) Cross Clipping(d) Bones and Vessels

160 fps (6.3 ms)
124 fps (8.1 ms)
104 fps (9.6 ms)
200 fps (5.0 ms)

Table 5 Visual evaluation of proposed method

as the input, and single image rendered on the same computer as above. Figure 8 shows images rendered in 4K
resolution with diﬀerent transfer function settings andclipped to the target area (2160 pixels vertically). Realtime and realistic illuminated rendering was achieved
with expected clinical CT data and display settings.
To assess the visual usefulness of the proposed method,
we carried out a visual assessment experiment on the understandability of the three-dimensional structure of thevessels. The celiac artery in the early phase images of ten
abdominal contrast-enhanced CTs was evaluated. Tworadiologists (15 - and 14 years experience) specializing ininterventional radiology scored the images on the following three-point scale. Score 3: Vascular structures could
be identiﬁed at a glance without any rotation manipulation of the volume. Score 2: The vascular structures
could be identiﬁed by slightly rotating the volume. Score
1: The vascular structure could be identiﬁed by rotating the volume many times. We compared the scores ofthe proposed and conventional volume rendering methods. Here, the functions (a), (b), (c), and (e) presented
in Chapter 3 were disabled from the proposed method
and used as the conventional volume rendering method.
The representative images used in the visual evaluation
are shown in Fig. 9. The results of the visual evaluation
are presented in Table 5 and Table 6. Both readersscored higher with the proposed method (paired t-test, p＜ 0.01).

5.

(b) Proposed rendering

Fig. 9 Representative images used for visual evaluation

ReaderScore 3count

Score 2
count

Score 1
count

Average
score

1
2

7
6

3
3

0
1

2.8
2.5

Table 6 Visual evaluation of conventional method
ReaderScore 3count

Score 2
count

Score 1
count

Average
score

1
2

0
0

8
4

2
6

1.8
1.4

for direct volume rendering. With this method, even nonMonte Carlo direct volume rendering methods can eﬃciently handle sub-surface scattering, which is importantfor soft tissue representation. Our acceleration methods
can also be used with a selection of various illumination
techniques, allowing for real-time rendering even if you
do not have access to a high-performance computer. In
terms of interaction, the combination of a free viewpoint
stereoscopic view and an optical hand sensor has realized
a new intuitive operation in cross-sectional manipulation.
In this paper, we focused on volume data obtained by
CT scanning. However, the method will be useful for intuitive understanding of volume data acquired by diﬀerent methods, such as 3D observational data or simulated
data. However, the volume data that can be handled bythis method are only grid data, and special implementation or resampling will be required when handling volumedata with non-grid structures.
In this study, hand gesture-based interactive manipulation methods were not evaluated and should be carried out in future work. Operation using hand gestures is
not necessarily superior to operation using a conventional
mouse, and the non-contact feature is more valuable thanthe operability. It will be necessary to design an evaluation experiment that considers this.

Discussion and Conclusion

We developed a CT volume rendering method thatcombines realistic illumination with interactions that require fast rendering by developing an accelerated method

38

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Kousuke KATAYAMA
He received his B.E. and M.E. in Informatics
and Data Science from Hiroshima University
in 2022 and 2024. His research interests include medical volume rendering. Currently,
he is an employee of CAPCOM Corporation,
which is in the video game development industry.

References
1) C. Xu, G. Sun and R. Liang: “A survey of volume visualization
techniques for feature enhancement”, Visual Informatics, 5, 3,
pp. 70–81 (2021).
2) T. Kroes, F. H. Post and C. P. Botha: “Exposure render:
An interactive photo-realistic volume rendering framework”,
PLOS ONE, 7, 7, pp. 1–10 (2012).

Toru HIGAKIHe received his Ph.D. in Engineering from
Hiroshima University, Japan, in 2011. Currently, he is an associate professor in Graduate School of Advanced Science and Engineering, Hiroshima University, Japan. His current
research interests include medical imaging, visualization, and medical physics.

3) D. Lin, C. Wyman and C. Yuksel: “Fast volume rendering withspatiotemporal reservoir resampling”, ACM Trans. Graph., 40,
6 (2021).
4) A. Omori, R. Inuzuka and Y. Hirata: “Visualization of thecomplex double-outlet right ventricle anatomy using real-time
three-dimensional computer graphics”, JTCVS Tech., 18, pp.
97–99 (2023).
5) E. Dappa, K. Higashigaito, J. Fornaro, S. Leschka, S. Wildermuth and H. Alkadhi: “Cinematic rendering - an alternative
to volume rendering for 3D computed tomography imaging”,
Insights Imaging, 7, 6, pp. 849–856 (2016).

Kazufumi KANEDA (Member)He received his D.E. in system engineeringfrom Hiroshima University, Japan, in 1991.
Currently, he is a professor at Graduate
School of Advanced Science and Engineering,
Hiroshima University, Japan. His research interests include computer graphics, data visualization, and medical graphics.

6) P. Rheingans and D. Ebert: “Volume illustration: Nonphotorealistic rendering of volume models”, IEEE Trans. on Visualization and Computer Graphics, 7, 3, pp. 253–264 (2001).
7) M. Levoy: “Eﬃcient ray tracing of volume data”, ACM Trans.
Graph., 9, 3, pp. 245–261 (1990).
8) D. Bauer, Q. Wu and K. Ma: “Fovolnet: Fast volume rendering using foveated deep neural networks”, IEEE Trans. on
Visualization amp; Computer Graphics, 29, 01, pp. 515–525(2023).

Bisser RAYTCHEVHe received his Ph.D. in Informatics from
Tsukuba University, Japan, in 2000. Currently, he is an associate professor in Graduate School of Advanced Science and Engineering, Hiroshima University, Japan. His current research interests include machine learning and natural language processing.

9) L. Neumann, B. Csébfalvi, A. König and E. Gröller: “Gradient estimation in volume data using 4d linear regression”,
Computer Graphics Forum, 19, 3, pp. 351–358 (2000).
10) Unity: “Standard shader”, https://docs.unity3d.com/2020.3/
Documentation/Manual/shader-StandardShader.html (2020).
11) T. Ropinski, C. Doring and C. Rezk-Salama: “Interactive volumetric lighting simulating scattering and shadowing”, 2010
IEEE Paciﬁc Visualization Symposium (PaciﬁcVis 2010), Los
Alamitos, CA, USA, IEEE Computer Society, pp. 169–176(2010).

Wataru FUKUMOTO
He received his Ph.D. in Medicine from Hiroshima University, Japan, in 2018. Currently, he is an assistant professor in Graduate
School of Biomedical and Health Sciences, Hiroshima University, Japan. His specialties include diagnostic radiology, interventional radiology, and postmortem imaging.

12) M. McGuire, B. Osman, M. Bukowski and P. Hennessy: “Thealchemy screen-space ambient obscurance algorithm”, Proc.
of the ACM SIGGRAPH Symposium on High Performance
Graphics, HPG ’11, New York, NY, USA, Association for Computing Machinery, pp. 25–32 (2011).
13) P. H. Christensen: “An approximate reﬂectance proﬁle for efﬁcient subsurface scattering”, ACM SIGGRAPH 2015 Talks,
SIGGRAPH ’15, New York, NY, USA, Association for Computing Machinery (2015).

Hidenori MITANIHe received his Ph.D. in Medicine from Hiroshima University, Japan, in 2022. Currently, he is an assistant professor in Graduate School of Biomedical and Health Sciences,
Hiroshima University, Japan. His specialties
include diagnostic radiology interventional radiology, and emergency radiology.

14) E. Golubev:“Eﬃcient screen-space subsurface scattering using burley’s normalized diﬀusion in real-time”,
https://advances.realtimerendering.com/s2018/ (2018).
15) SONY: “Spatial reality display white paper: Technical background”, https://www.sony.net/Products/Developer-SpatialReality-display/jp/develop/WhitePaper.html (2023).
16) Ultraleap:“Ultraleap 3di”,
product/ultraleap-3di/ (2022).

https://www.ultraleap.com/

17) M. Levoy:“The stanford 3d scanning repository”,
https://graphics.stanford.edu/data/3Dscanrep/ (2000).

(Received November 10, 2023)(Revised April 7, 2024)

39

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）
IIEEJ

Paper -- Special Issue on Journal Track Papers in IEVC2024 -An Examination of Motion Analysis During Weight Illusion
by Impression Change of the Own Body
Joichiro MURAOKA† , Kosei TOMIOKA† (Student Member ), Yusei MURAISHI† ,Naoki HASHIMOTO†† , Mie SATO†† Utsunomiya University , †† The University of Electro Communications<Summary> The development of virtual reality technology has made it possible to easily change theappearance of a person, and it has been suggested that the impression change of the appearance using virtual
reality avatars aﬀects not only self-perception but also weight perception. We have studied the physicaleﬀects of the impression change of the own body during the weight illusion based on electromyogram. As
a result, a signiﬁcant correlation was obtained between the electromyogram and the degree of the weightillusion among the subjects, although the correlation was not signiﬁcant for each subject. In this study, toinvestigate the change in motion during the weight illusion, we analyze the motion under the impression
change of the own body through a comparison experiment of dumbbells’ weights. The results showed that
subjects who obtained a correlation between the degree of impression of the strength of the avatar andthe degree of weight illusion had a positive correlation between the degree of impression of the strength of
the avatar and the velocity of their motion. The results suggest that the velocity of motion, the degree of
impression of the strength of the avatar, and the degree of the weight illusion are related with each other.
Keywords: virtual reality avatar, self-perception, weight illusion, Proteus eﬀect

1.

output changes during the weight illusion by impression
change. As a result, a positive correlation between the
degree of the weight illusion and the change in EMG wasobserved for all subjects, but any signiﬁcant correlation
within each subject was not observed3).
Therefore, the purpose of this study is to clarify theeﬀects on the body during the weight illusion by the impression change using full-body avatars. Using avatars
with diﬀerent impressions, we investigate if there are relations between the degree of strength impression given
to the avatar (hereafter, avatar impression quantity), thedegree of weight illusion (hereafter, weight illusion quantity), and the change in movement speed when comparing
dumbbell weights.

Introduction

In recent years, research that gives users the illusionof impression by changing visual information has been
attracting attention. Among these, the study of theProteus eﬀect, in which the appearance of an avatar invirtual reality (VR) inﬂuences the user’s behavior andself-expression, is particularly well-known. Sumida etal.1)conducted experiments on weight perception using
avatars with diﬀerent muscle masses. The results suggested that changes in cognition in the VR space, represented by the Proteus eﬀect, even aﬀect the weight perception. Okubo et al. conducted an experiment in which
dumbbells were lifted (dumbbell lift) using a variety ofavatars2). The results suggested a negative correlation
between the weight perception and the strength impression in a variety of whole-body avatars. That is, thestronger the impression of avatar, the lighter the dumbbell tended to be felt. However, there is insuﬃcient research on the Proteus eﬀect and physical performance,
and there is room to examine if there is a possibility for
immersion in VR avatars to bring out users’ potential
power and physical capabilities. We used electromyogram(EMG) to investigate whether the user’s physical force

2.

Related research

Stereotypes and assumptions about visual informationaﬀect human cognition. This has led to the development of research on the relation between stereotypes
and assumptions and cognition, and the relation between
the appearance of avatars used by users and their selfperception has been actively investigated.
Yee showed that the appearance of a user’s avatar
may inﬂuence the user’s speech, behavior, and sociabil-

40

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

ity when communicating through an avatar in a virtual
space4). They call the phenomenon of behavior change
due to the appearance of avatar the Proteus eﬀect. Banakou et al. showed that the use of Albert Einstein’s
avatar, which is considered highly intelligent, contributed
to improved performance on cognitive tasks and reduced
negative prejudice toward the elderly5). These reports
conﬁrm that stereotypes and assumptions about visual
information aﬀect cognition in VR environments as wellas in real environments.
Recent studies have suggested that changes in cognition in VR spaces, as typiﬁed by the Proteus eﬀect, evenaﬀect weight perception1). Based on these studies, weinvestigated the eﬀect of the impression of strength on
the body replaced by a 3D model on weight perception2). The results showed that the more forceful one feels toward the 3D model of one’s own body, the lighter one
tends to perceive the lifted object to be2).
There have also been a few studies on the inﬂuence of
the Proteus eﬀect on the body, and Marcin et al. investigated the eﬀect of the Proteus eﬀect on the degree of
fatigue6). The results showed that the use of muscularavatars tended to increase the number of biceps curls.
Thus, the relation between cognition and weight perception and between cognition and fatigue have beenstudied, but there are no studies on diﬀerences in force
output and movement.

Fig. 1 Experiment scene (subject (left), subject’s appearance as seen by the subject through the HMD(right))Fig. 2 Example of object images

ject’s movements in a VR space. The subject could see
the avatar’s appearance synchronized with his/her own
movements using a virtual mirror installed in the VR
space. Figure 1 shows the subject during the experiment and an example of the avatar’s appearance in the
VR space presented on the HMD.
3.2

3.

Experiment

The purpose of this task is to evaluate the basic impression quantity of each avatar.
In the Avatar evaluation task, the subject ﬁrst selects“the heaviest object that the subject can hold by his/her
own strength” from a set of 32 object images as shown in
Fig. 2. Using this as the reference object, the weight ratio of each of the remaining 31 object images is evaluatedin comparison with the reference object. In the experiment, in order to collect the intuitive quantities relative
to the reference stimulus as accurately as possible, thesubject is asked to answer the weight ratio using a slider
that shows continuous values.
Next, this group of the object images are presented tothe subject to determine his/her impression of the avatar.
The subject sees himself/herself as an avatar in the VR
space through the HMD and evaluates his/her impressionof the avatar. For each of the 24 avatars, as shown in
Fig. 3, the subject selects the “heaviest object this avatar
can hold” from the set of 32 images. The basic impression
quantity of each avatar is deﬁned as Eq. (1).

The avatar impression quantity and the weight illusion
quantity are ﬁrst obtained through “Avatar evaluationtask” followed by “Weight comparison task.” During theweight comparison task, we measure the position coordinates of the subject’s right forearm during the weightcomparison of dumbbells with avatars. The velocity is obtained from the measured position coordinates. The correlations between the avatar impression quantity, weightillusion quantity, and velocity in motion are then examined.
3.1

Avatar evaluation task

Experiment environment

In this experiment, a head-mounted display (HMD)(HTC, VIVE Pro) and a motion capture system (OptiTrack, PrimeX13) were used to construct a system toevaluate the avatar impression quantity and weight illusion quantity. During the experiment, the subject’s movements were captured by the motion capture system andthe avatar’s movements were synchronized with the sub-41

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig. 3 Example of whole-body avatars used in the experiment

Basic impression quantity=

W eight ratio of the heaviest object selected
W eight ratio of the ref erence object (= 1.0)

(1)Fig. 4 States of the forearms (the start of the dumbbell
lift (left), the end of dumbbell lift (right))

3.3 Weight comparison task
The purpose of this task is to obtain the avatar impression quantity and the weight illusion quantity.
In the weight comparison task, taking into an accountthe burden on the subject, the experimenter selects six
avatars from the 24 avatars used in the avatar evaluation
task. How to select the six avatars is to calculate theavatar impression quantity deﬁned in Eq. (2) and to selectsix avatars whose avatar impression quantities are 100 orless in any combination of the selected two avatars.

know the weight of the dumbbell is 2 kg. In step (iv), a
slider that shows continuous values between 0.0 and 2.0 isused to rate the perceived weight of Dumbbell lift 2 compared to the weight of Dumbbell lift 1. The evaluatedweight ratio here is the weight illusion quantity of thistrial. The weight illusion quantity is deﬁned as Eq. (3).
W eight illusion quantity=

Basic avatar impression quantity of avatar B
Basic avatar impression quantity of avatar A(3)

While the subject performs the steps (ii) and (iii), theposition coordinates of his/her right forearm is obtained.
As shown in Fig. 4, markers are added to the right forearm of the motion capture suit. In addition, EMG sensors
are attached to the brachial bicep and forearm to measure
EMG.
The total number of trials is 36, including 30 avatar
combinations and six dummy comparisons.

Avatar impression quantity=

W eight of Dumbbell lif t 2
W eight of Dumbbell lif t 1(2)

On each trial, two avatars are chosen from the six selected avatars. The subject becomes the avatars’ appearances (appearance A and B) and compares the weights ofthe dumbbells. The following procedure is used for theweight comparisons.
(i) Posing while looking at the avatar’s appearance ofhimself/herself in the mirror.
(ii) Dumbbell lift with avatar A (Dumbbell Lift 1).
(iii) Dumbbell lift with avatar B (Dumbbell Lift 2).
(iv) Evaluate the weight ratio of Dumbbell Lift 2 when
the weight of Dumbbell Lift 1 is 1.0.
In the step (i), the subject is asked to look at theavatar in a mirror and performs some poses that he/she
can imagine from that avatar in order to increase his/her
immersion in the avatar. This step contributes to improvement of his/her sense of ownership to the avatar. In
steps (ii) and (iii), two of the six avatars are presented
at random, and the subject performs a 2 kg dumbbell
lift while looking at the avatar synchronized with his/her
movements in the mirror. Note that the subject does not

3.4 Post evaluation
At the end of the experiment, the subject is asked toanswer questions about his/her immersion and sense ofunity for each of the six avatars used in the weight comparison task. To answer, a seven-point Likert scale (−3for “did not feel at all,” 0 for “neutral” and +3 for “feltvery strongly”) is used. In this evaluation, a sense ofagency and a sense of ownership are used as indices tomeasure the immersion and the sense of unity. The following questions are asked in the post-evaluation.
Q1. Did you feel that the avatar’s movements matchedyour movements?
Q2. Did you feel that your avatar was reﬂected in the
mirror?
Q3. Did you feel as if you were the avatar?

42

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

weight_illusion_quantity

101

Fig. 5 EMG measurement (sensor position (left), example of resulting graph (right))

100

10−1

4.

10−2

Resultssubject

4.2

(4)

4.3

There are 150 data points from the ﬁve subjects. Of
these, 142 velocity data points were successfully obtained
and analyzed.
EMG data were measured with an EMG sensor(TRUNK SOLUTION CORPOTATION, TS-MYO). The
position of the EMG sensor is shown in Fig. 5. First,
EMG data were full-wave rectiﬁcation. The EMG data
point for each dumbbell lift were averaged over the fullwave rectiﬁed data within the analysis section. The EMG
data point for each trial is obtained using Eq. (5).

No.1

No.4

No.6

No.7

No.8

Correlation between the avatar impression quantity and the weight illusion
quantity

Correlation between the avatar impression quantity and the velocity data point

Spearman’s rank correlation analysis was performedon the avatar impression quantity and the velocity data
point, taking into an account the inﬂuence of outliers.
A signiﬁcant positive correlation was obtained (ρ = .36,
p < .001). There is a relation between the strength impression of the avatar and the lifting speed of the dumbbell. Figure 7 shows a scatter plot of the avatar impression quantity and velocity data points for the ﬁve
subjects.
The positive correlation between the avatar impression
quantity and velocity data points was conﬁrmed. Therefore, the velocity data was decomposed into X, Y and Z
components, and relations between each component andthe avatar impression quantity were analyzed. As a result, no signiﬁcant correlation was found between the X
and Z components and the avatar impression quantity

EM G data point
EM G data f or Dumbbell lif t 2
EM G data f or Dumbbell lif t 1

102

The purpose of this experiment is to clarify the eﬀects
of the illusion on the body. The ﬁve subjects showed anegative correlation between the avatar impression quantity and the weight illusion quantity. Spearman’s rankcorrelation analysis was conducted on the avatar impression quantity and the weight illusion quantity. A signiﬁcant negative correlation was obtained (ρ = −.44,
p < .001). Figure 6 shows a scatter plot of the avatar impression quantity and weight illusion quantity data points
for the subjects with the negative correlation.

V elocity data point=

101

Fig. 6 Scatter plot of the avatar impression quantity andweight illusion quantity data points

4.1 Analysis of the velocity and EMG
The analysis target section is from the start of thedumbbell lift to the end of the dumbbell lift. Figure 4
shows the states of the forearm at the start and end of
the dumbbell lift.
The velocity is calculated from the coordinates of themarker of the right forearm obtained by the motion capture system. The velocity data point for each dumbbell
lift is the average of the frame-by-frame velocities within
the analysis section. The velocity data point for each trial
is obtained using Eq. (4).

V elocity data f or Dumbbell lif t 2
V elocity data f or Dumbbell lif t 1

100

avatar_impression_quantity

Five male subjects aged 20-24 years, whose weight perception was judged to be normal by comparing the weightratio of two dumbbells, participated in the experiment.

=

10−1

(5)

There are 150 data points obtained from the ﬁve subjects.

43

2.0

2.0

1.5

1.5

velocity

velocityIIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

1.0

0.5

1.0

0.5

0.0

0.0
10−2

10−1

100

101

102

0.0

avatar_impression_quantity
subject

No.1

No.4

No.6

No.7

0.5

1.0

1.5

2.0

weight_illusion_quantity
subject

No.8

Fig. 7 Scatter plot of the avatar impression quantity andvelocity data points

No.1

No.4

No.6

No.7

No.8

Fig. 9 Scatter plot of the weight illusion quantity andvelocity data points

2.0
2.0

1.0

velocity_y

velocity_y

1.5

0.0

1.0

0.5
−1.0
0.0
10−2

10−1

100

101

102

0.0

subject

No.1

No.4

No.6

No.7

0.5

1.0

1.5

2.0

weight_illusion_quantity

avatar_impression_quantity
subject

No.8

No.1

No.4

No.6

No.7

No.8

Fig. 10 Scatter plot of the weight illusion quantity and
Y component velocity data points

Fig. 8 Scatter plot of the avatar impression quantity and
Y component velocity data points

4.4(X: ρ = .06, p = .50, Z: ρ = .12, p = .14). A signiﬁcant
positive correlation was found between the Y component
and the avatar impression quantity (ρ = .37, p < .001).
The correlation between the avatar impression quantity
and the Y component of the velocity data shows the same
trend as that between the avatar impression quantity andthe velocity data point. Therefore, there is a possibility that there is a relation between the degree of avatar
strength impression and the velocity of motion in the Y
direction. Figure 8 shows a scatter plot of the avatar impression quantity and Y component velocity data points
for the ﬁve subjects.

Correlation between the weight illusion
quantity and the velocity data point

Spearman’s rank correlation analysis was performed on
the weight illusion quantity and the velocity data point,
taking into an account the inﬂuence of outliers. No correlation was found (ρ = −.15, p < .01). However, the ﬁve
subjects showed negative correlations including a significant negative correlation (ρ = −.32, p < .01) for theweight illusion quantity and velocity data points. Therefore, it is possible that a relation between the weight illusion quantity and the velocity data point can be obtained.
Figure 9 shows a scatter plot of the weight illusion quantity and velocity data points for the ﬁve subjects.
The analysis of the avatar impression quantity and the

44

2.5

2.5

2.0

2.0

forearm_EMG

forearm_EMG

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

1.5

1.0

0.5

1.5

1.0

0.5

0.0

0.0
10−2

10−1

100

101

102

0.0

0.5

avatar_impression_quantity
subject

No.1

No.4

No.6

No.7

subject

No.8

Fig. 11 Scatter plot of the avatar impression quantityand forearm EMG data points

1.5

2.0

No.1

No.4

No.6

No.7

No.8

Fig. 13 Scatter plot of the weight illusion quantity andforearm EMG data points

2.5

2.5

2.0

2.0

biceps_EMG

biceps_EMG

1.0weight_illusion_quantity

1.5

1.0

0.5

1.5

1.0

0.5

0.0

0.0
10

−2

10

−1

10

0

1

10

10

2

0.0

0.5

subject

No.1

No.4

No.6

No.7

1.0

1.5

2.0

weight_illusion_quantity

avatar_impression_quantity
subject

No.8

No.1

No.4

No.6

No.7

No.8

Fig. 14 Scatter plot of the weight illusion quantity andbrachial bicep EMG data points

Fig. 12 Scatter plot of the avatar impression quantityand brachial bicep EMG data points

for the ﬁve subjects.

velocity data point conﬁrmed a similar trend both in theavatar impression quantity and velocity data points andin the avatar impression quantity and Y component velocity data points. Therefore, it was thought that thetrend could be also appeared in analyzing the weight illusion quantity and velocity data points by decomposing
the velocity data into X, Y and Z components. As aresult, no signiﬁcant correlation was found between the
X and Z components and the weight illusion quantity(X: ρ = −.03, p = .74, Z: ρ = −.09, p = .28). A
signiﬁcant negative correlation was found between the Y
component and the weight illusion quantity (ρ = −.23,
p < .001). Figure 10 shows a scatter plot of the weightillusion quantity and Y component velocity data points

4.5

Correlation between the avatar impression quantity and the EMG data point

Spearman’s rank correlation analysis was performed on
the avatar impression quantity and the forearm EMG
data point, taking into an account the inﬂuence of outliers. A signiﬁcant positive correlation was obtained(ρ = .36, p < .001). This is a diﬀerent trend from aprevious study3). Figure 11 shows a scatter plot of theavatar impression quantity and forearm EMG data points
for the ﬁve subjects.
Spearman’s rank correlation analysis was also performed on the avatar impression quantity and thebrachial bicep EMG data point, taking outliers into ac-

45

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

sible that subjects who have a relation between the avatar
impression quantity and the weight illusion quantity also
have a relation between the avatar impression quantity
and the change in motion. In addition, because the ﬁve
subjects showed negative correlations for the weight illusion quantity and velocity data points, it is possiblethat a relation between the weight illusion quantity andthe change in motion can be more clearly obtained byincreasing the number of subjects.

count. No correlation was found (ρ = .06, p = .48).
This result is similar to a previous study3). Figure 12
shows a scatter plot of the avatar impression quantity andbrachial bicep EMG data points for the ﬁve subjects.
4.6

Correlation between the weight illusion
quantity and the EMG data point
Spearman’s rank correlation analysis was performed on
the weight illusion quantity and the forearm EMG data
point, taking into an account the inﬂuence of outliers.
A signiﬁcant negative correlation was obtained (ρ = −.23, p < .01). This is a diﬀerent trend from a previous
study3). Figure 13 shows a scatter plot of the weightillusion quantity and forearm EMG data points for theﬁve subjects.
Spearman’s rank correlation analysis was also performed on the weight illusion quantity and the brachialbicep EMG data point, taking outliers into account. No
correlation was found (ρ = −.17, p < .05). This result issimilar to a previous study3). Figure 14 shows a scatter plot of the weight illusion quantity and brachial bicep
EMG data points for the ﬁve subjects.

5.

Acknowledgment
This work was supported by JSPS KAKENHI Grant
Number JP23K11195.

References
1) K. SUMIDA, N. OGAWA, T. NARUMI, M. HIROSE: “Proteus
Eﬀect of a Muscular Avatar on Weight Perception in Virtual
Reality”, VRSJ The 25th Annual Conference, 2A2–2 (2020).
2) Y. OKUBO, J. MURAOKA, M. SATO, N. HASHIMOTO: “Inﬂuence on Weight Perception by Change of Self-Perception
Based on Avatar’s Strength”, Journal of ITE, Vol. 77, No. 3,
pp.394–400 (2023).
3) J. MURAOKA, Y. MURAISHI, K. TOMIOKA, M. SATO, N.
HASHIMOTO: “Relation Between Weight Illusion and EMG
Signals by Impression Change of the Own Body”, ITE Annual
Convention 2023, 21C–4 (2023).

Discussion

This experiment showed a negative correlation between
the avatar impression quantity and the weight illusion
quantity, the positive correlation between the avatar impression quantity and the velocity data point, and thenegative correlation between the weight illusion quantity and the Y component of the velocity data point.
These suggest that the stronger the impression of avatar
strength, the lighter the object may be perceived and thefaster the object is lifted. The positive correlation was
also obtained between the avatar impression quantity andthe forearm EMG data point. Mero et al. showed thatparameters such as EMG also increased with a runningvelocity7). The positive correlations between the avatar
impression quantity and the velocity data point and between the avatar impression quantity and the forearm
EMG data point also support this suggestion.

6.

4) N. YEE, J. BAILENSON: “The Proteus Eﬀect: The Eﬀect ofTransformed Self‐Representation on Behavior”, Human Communication Research, Vol. 33, pp.271–290 (2007).
5) D. BANAKOU, S. KISHORE. M. SLATER: “Virtually Being
Einstein Results in an Improvement in Cognitive Task Performance and a Decrease in Age Bias”, Frontiers in Psychology,
Vol. 9 (2018).
6) M. CZUB, P. JANETA: “Exercise in Virtual Reality with a
Muscular Avatar Inﬂuences Performance on a Weightlifting Exercise”, Journal of Psychosocial Research on Cyberspace, Vol.
15, No. 3, Article 10 (2022).
7) A. MERO, P. V. KOMI: “Force-, EMG-, and Elasticity-Velocity
Relationships at Submaximal, Maximal and Supramaximal
Running Speeds in Sprinters”, European Journal of Applied
Physiology and Occupational Physiology, Vol. 55, pp.553–561(1986).

(Received November 12, 2023)(Revised May 28, 2024)

Conclusion

In this study, with the aim of clarifying the eﬀects on
the body during the weight illusion using a whole-bodyavatar with diﬀerent impressions of the avatars, we investigated the relations between the avatar impression
quantity, the weight illusion quantity, and the change inmotion by measuring the velocity and the EMG during
the weight comparison of dumbbells. As a result, it is pos-46

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Joichiro MURAOKA
He graduated from the Graduate School ofRegional Development and Creativity, Utsunomiya University in 2024. He received
a Master of Optical Science and Engineeringfrom Utsunomiya University in 2024.

Kosei TOMIOKA (Student Member)
He is a graduate student at the Graduate
School of Regional Development and Creativity, Utsunomiya University.

Yusei MURAISHIHe is a graduate student at the Graduate
School of Regional Development and Creativity, Utsunomiya University.

Naoki HASHIMOTO
He is a professor at the Graduate School ofInformatics and Engineering, the University
of Electro-Communications. He received aDoctor of Engineering from Tokyo Instituteof Technology in 2001. He is a member of
ACM SIGGRAPH, IEICE, ITE and VRSJ.

Mie

SATO

She is a professor at the School of Data Science and Management, Utsunomiya University. She received a Doctor of Engineeringfrom Tokyo Institute of Technology in 2001.
She is a member of ACM SIGGRAPH and
ITE.

47

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）-- Special Issue on Journal Track Papers in IEVC2024 --Smartphone-Based Continuous Authentication
Based on Flick Input Features Using Japanese Free Text
Shuto KINOSHITA†, Yasushi YAMAZAKI†† (Member)† Graduate School of Environmental Engineering, The University of Kitakyushu†† Faculty of Environmental Engineering, The University of Kitakyushu<Summary> With the rapid spread of smartphones, user authentication on smartphones has become essential. However,conventional user authentication methods using PINs, passwords, pattern locks, etc. have a problem in that users are notauthenticated continuously after the first successful authentication; therefore, there is a risk that an authenticated smartphonemight be used improperly by unauthorized individuals. To address this problem, continuous authentication that verifies theuser’s identity without burdening the user by continuously acquiring biometric information from his/her daily smartphone
usage has been proposed. Specifically, as smartphones are utilized for various purposes, ensuring the authenticity of the user
during text input actions is crucial. Therefore, in this paper, we focus on achieving continuous authentication on smartphones
on the basis of flick input behavior, which is a typical text input action by many smartphone users. We aimed to extractuser-specific features from Japanese free text input through flick operations in daily usage and evaluated the effectiveness ofcontinuous authentication on the basis of these extracted features. The simulation results indicated that a certain level ofauthentication accuracy can potentially be maintained by appropriately selecting suitable features.
Keywords: biometrics, smartphones, continuous authentication, behavioral biometrics, flick input, free text

owners in the U.S. is a victim of theft2). As a solution to thefirst problem, a biometric authentication method was
proposed with a function for recognizing the usage
environment on the basis of “context awareness,” which isthe concept of being able to respond to changes in thesituation of people, objects, and environments3). It was
shown that this method improves the reliability of user
authentication by allowing the system to adaptively select anauthentication method suitable for the user’s usage
environment at the time of authentication. As a solution to
the second problem, the implementation of a continuous
authentication function4) was proposed to prevent
unauthorized smartphone use after the first authentication
success by continuously acquiring biometric information
without burdening the user. In one of our previous studies5),
a usage environment-aware continuous authentication
system that combines these two functions was proposed, andits application can improve the convenience and security ofsmartphones. The system runs in the background tocontinuously authenticate the user.
Continuous authentication has been extensively
researched by using various features that can be extracted
from smartphone sensors and the devices’ internal
information, such as walking patterns, SMS activity, Wi-Fi

1. Introduction
With the rapid spread of smartphones, there has been an
increase in opportunities to handle users’ personal
information on smartphones, which makes user
authentication technology essential for security and privacy
protection. Conventional user authentication methods for
smartphones include PINs, passwords, and pattern locks;
however, these methods have a problem in that users cannot
unlock their smartphones when they forget the passwords
and patterns they have set. There is an additional problem inthat a malicious third party may impersonate a legitimate
user by shoulder hacking or password/pattern guessing froma residual fingerprint on the touch screen. Therefore,
biometric authentication technology that utilizes biometric
information obtained from sensors installed in smartphones
has been attracting attention1).
However, since smartphones are highly portable andauthentication is performed in various environments,changes in the usage environment may affect the reliabilityof biometric authentication on smartphones. Moreover, in
any of the above user authentication methods, thesmartphone may be stolen and misused by others after the
first authentication success. In fact, one in ten smartphone

48

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig.1 Usage environment-aware continuous authentication system 9)previous works5), we focus on a continuous authentication
system with a usage-environment recognition function basedon “context awareness” 9), as shown in Fig. 1.
This system utilizes a user authentication function that iscommonly installed in smartphones to verify the user’s
identity, and when it is confirmed, the system unlocks thedevice and starts continuous authentication. In thecontinuous authentication, both the usage environment
information and biometric information are acquired atregular intervals because the user’s behavior and thelocation in which the device is used may change from time
to time. As in our previous study9), the acquired usage
environment information includes acceleration, time, andscreen brightness, and the biometric information includes
face image and swipe touch information (contact position,
contact area, and contact time). After acquiring the usage
environment and biometric information, the system
recognizes the usage environment by utilizing the usage
environment information and selects an environment label
corresponding to the recognition result from a template
database. Then, the features in the template databaseincluded in the environment label are matched with thefeatures extracted from the acquired biometric information.
If the authentication succeeds, the system repeats theusage, application usage history, finger movements during
device operation, and changes in the touch area on thescreen6)–9). Moreover, as smartphones are utilized for various
purposes ranging from SNS and email to password or PIN
input, ensuring the authenticity of the user during text input
actions is crucial. Therefore, continuous authentication10)–12)
has been attracting attention, especially for text input actions
including flick input, which is used by approximately half ofthe smartphone users in Japan13). In previous studies,
specific text designated by the examiners has been used asverification data. However, requiring users to input specifictext for each authentication interrupts their ongoing tasks
and is not ideal for user convenience.
Therefore, in this paper, we propose a new continuous
authentication method based on flick input behavior. We
assume that users input Japanese free text during flick
operations in their daily use and that user-specific features,
representations of biometric traits unique to each user, areextracted from that input. We utilize these extracted features
for continuous authentication and validate the effectivenessof the proposed method.

2. Smartphone-based Continuous Authentication
In this paper, which is a successive study to one of our

49

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

acquisition of the usage environment and biometric
information. If the authentication fails, the system locks thedevice and goes back to the initial (conventional) user
authentication process.
As discussed earlier, ensuring the authenticity of
individuals during text input actions on smartphones is an
important challenge. Smartphones feature multiple text input
methods such as toggle input, free key input, and flick input,
and in Japan, flick input is the most widely used. Flick input
utilizes a keyboard layout based on the Japanese Gojuon
table14) and allows entering the intended characters bytapping each key followed by flicking up, down, left, orright. Since it involves different operations from normal
swipe touch input, the individual authentication using swipe
touch information introduced in our previous study9) is notsuitable for flick input. Therefore, acquiring flick input
features in addition to conventional biometric informationneeds to be considered for our system.

tennis)” five times each. The acquired data was randomly
divided into training data and evaluation data at a 4:1 ratio
for each text string, and machine learning was applied toassess the accuracy.
As described above, previous studies have evaluatedauthentication accuracy focusing on flick input behavior
using various features and training methods for continuous
authentication on smartphones. However, Motoyama et al.12)
used the same character texts for both training and validation
data, while Izumi et al.10) and Ito et al.11) used different
character texts for training and validation data, althoughthese texts were still specified by the examiners. In thecontext of continuous authentication on smartphones,
user-specific features should preferably be extracted fromnormal device interactions without imposing specificauthentication actions on users. Therefore, in continuous
authentication using flick input features, requiring the user
to input specific text when authenticating leads to aninterruption of daily-use operation, which is not desirablefrom the viewpoint of usability.
Considering the above, the main problem with theprevious studies on continuous authentication using flick
input features on smartphones is the interruption of daily-use
operation by requiring users to input specific text each time
the authentication is performed.

3. Related Work
Several studies have examined continuous authentication
using smartphone flick input features. For example, Izumi etal.10) conducted feature extraction from flick input inJapanese documents and verified the accuracy of individual
identification and user authentication using weighted
Euclidean distance and the Array Disorder method. In theirexperiments, they selected approximately 300 characters per
document as keystroke data from a Japanese translation of
the novel “Alice’s Adventures in Wonderland” and obtained
data from five documents from participants. Subsequently,
using the leave-one-out method, they evaluated the accuracy
by taking one document from all the profile documents,
treating it as an unknown document. Ito et al.11) categorizedeach character in the text input data by vowels andperformed authentication using five classifiers based on the
One Class Support Vector Machine (OCSVM)15). In theirexperiments, they acquired input data from approximately
200 characters in three different texts. Two of the texts wereused as training data, while the remaining text was used as
test data for authentication. The accuracy was evaluatedbased on this setup. Motoyama et al.12) proposed anauthentication method intended for short-duration
authentication, limiting the number of flick input charactersto ten or less. They also compared the identification
accuracy using various machine learning algorithms. In theirexperiments, they obtained the data by inputting two
different Japanese text strings of “さとうきびばたけ(sugar cane field)” and “ めー る と てに す (mail and

4. Proposed Method
We address the problem described in the previous
section by using a method for extracting user-specificfeatures from users’ flick operations during daily use. In thismethod, instead of extracting user-specific features from
specific text as in the previous studies, we extract them fromJapanese free text that users input with flick operations
during their daily use and then utilize these features for user
authentication. As shown in Fig. 2, in the previous methods,
the user’s daily-use operation is interrupted by inputting
specific text at each authentication timing in continuous
authentication. In contrast, the proposed method
circumvents the interruption of daily-use operation by
performing authentication using user-specific features
extracted from the user’s flick operations during daily use
such as SNS messages and Web searches.
In this paper, we focus on how to achieve continuous
authentication during text input on the basis of flick input
behavior in the system shown in Fig. 1. To this end, we
added flick input information to the “Biometric information”,
and flick input features to the “Feature extraction” in thesystem. Utilizing these changes, we propose an

50

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Fig.2 Comparison of continuous authentication between previous and proposed methods

authentication method based on user-specific features
extracted from Japanese free text input through daily flick
operations.

day. For the training data, we utilized the fixed text to ensurethat enough features were obtained. For the evaluation data,
we assumed free text and asked each participant to input fivepieces of arbitrary text in hiragana characters for each of thefollowing categories I–III.

5. Experiments
We conducted experiments to evaluate the effectivenessof the proposed continuous authentication. All of theexperiments were conducted under a protocol approved by
the University of Kitakyushu with informed consent fromthe participants.

I.

II.

5.1 Flick input data collection
III.
We used iPhone 11 and iPhone 12 devices to collectflick input data, targeting a group of 13 participants
comprising university students aged 18 to 24 (eight men andfive women). As a daily input method, 12 were using flick
input and the remaining one was using toggle input. As adaily-use device, 11 were using iOS devices and two wereusing Android devices for more than 5 years. During text
input using flick input, a keyboard (shown in Fig. 3(a)) isdisplayed. By tapping or tapping and then swiping each key(as shown in Fig. 3(b)), specific characters can be entered.
First, each participant, while seated, performed flick input toinput approximately 300 hiragana characters14) taken fromthe beginning of the novel “Run, Melos16)” as a fixed text fortraining data, which took an average of 4 minutes. Second,
they input free text for evaluation data. In this experiment, aself-made application was used to collect the data and boththe training and evaluation data were collected on the same

SNS Messages: Recent messages sent on social
networking platforms such as LINE, Instagram
DM, etc.
Web Search: Words or phrases you want to searchfor.
To-Do List: Plans and tasks scheduled for today
and beyond.

5.2 Feature extraction
We extracted features from the collected flick input data.
Referring to previous studies10)–12), we used the featureslisted in Table 1.
5.3 Outlier removal
The feature vectors created by arranging the extracted
features may contain outliers, which could potentially affectthe classification accuracy. We therefore removed outliers
using the local outlier factor (LOF)17) on the basis of past
research11). LOF is an outlier detection algorithm that
identifies anomalies within a dataset by comparing thedensity of data points to their surroundings. It calculates the
local anomaly factor for each data point and is used to
identify abnormal points in the dataset. Note that outliers

51

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Table 1 Feature types

No.
0
1
2
3
4
5

6

7
(a) Initial state(b) During flick input

8
9

Fig.3 Overview of flick input keyboard

were removed for each type of feature in this experiment. In
this case, the parameter k, which specifies the number ofdata points to be considered as neighbors for each data point,
was set to 7, and the LOF score threshold was set to 0.08.
These parameters were determined on the basis of the resultsof preliminary experiments.

10

11
12

5.4 Feature selection
13
Among the features listed in Table 1, 11 types (indicatedby numbers 1, 2, 3, 4, 11, 12, 13, 14, 15, 16, 18) were used
for tap actions (corresponding to vowel “a”), and all 18
types were used for flick actions (corresponding to vowel“i,” “u,” “e,” and “o”). We examined feature vectors
comprising all combinations of these 11 or 18 types andsearched for which combinations of feature vectors wereeffective for continuous authentication. The combinations offeature vectors resulted in 2,047 variations for tap actions
and 262,143 variations for flick actions.

14

15

16

17

5.5 Training
18
In the continuous authentication envisioned in this paper,
training data from persons other than the user themselves isdifficult to obtain. Additionally, creating 46 different
classifiers for each hiragana character would incur
significant training costs. Therefore, with reference to theliterature11), we created five classifiers using OCSVM for

FeatureVowel: Vowel in the input character (usedfor classifier selection)
Current Character: Current input character
Previous Character: Character input at theprevious time
Current Consonant: Consonant of thecurrent input character
Previous Consonant: Consonant of thecharacter input at the previous time
Flick X Displacement: Displacement of the
X-coordinate from the start to the end of the
flick
Flick Y Displacement: Displacement of the
Y-coordinate from the start to the end of the
flick
Flick Distance: Length of the trajectoryfrom the start to the end of the flick
Flick Curvature: Curvature of the flick
Flick Speed: Average speed from the start tothe end of the flick
Flick Angle: Angle between the line
connecting the start and end points of the
flick and the X (Y) axis
Input Time: Time from the start to the endof the flick
Time Gap Before: Time from the end of theprevious input to the start of the current
inputTime Gap After: Time from the end of thecurrent input to the start of next input
3-Axis Composite Acceleration: Averagevalue of composite acceleration in threeaxes during input
3-Axis Composite Angular Velocity:
Average value of composite angularvelocity in three axes during input
Touch Radius: Radius of the circle whentouching the screen at the beginning of the
flick
Average Touch Radius: Average radius of
the touched area from the start to the end of
the flick
Input Interval: Time duration from pressing
one key to releasing the next key

each vowel in the input characters (see Fig. 4). As a result,
we reduced the training cost and conducted user
authentication solely on the basis of the user’s own data.

52

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

5.6 Evaluation method
As described earlier, we used fixed text comprising
approximately 300 characters as training data and free text
as evaluation data. In the experiment, we used the training
data obtained from each participant to create classifiers foreach vowel in the input characters. Subsequently, we applied
the created classifiers to the user’s evaluation data andevaluation data from individuals other than the user forauthentication. Here, the authentication was performed on acharacter-by-character
basis.
We
evaluatedtheauthentication accuracy by(1)

TP (True Positives) represents the number of times the input
actions of the user were correctly identified as belonging to
the user. TN (True Negatives) represents the number oftimes the input actions of others were correctly identified as
belonging to others. FP (False Positives) is the number oftimes the input actions of others were incorrectly classified
as belonging to the user. FN (False Negatives) is the numberof times the input actions of the user were incorrectlyclassified as belonging to others. Note that the values of TP,
TN, FP, and FN were calculated for all participants’ data by
swapping the training data.

Fig.4 Classifier determination for input charactersand evaluation data, nor the influence of user’s proficiency
in flick input on authentication accuracy.
5.8 Consideration
From Table 2, among the combinations of feature
vectors that exhibited higher authentication accuracy, no. 11
was the most frequently included, followed by nos. 4, 15,
and 16. Nos. 11 (Input Time) and 16 (Touch Radius) werealso frequently included, which suggests that individualfeatures on the touchscreen during flick input operations arereflected in the input time for each character and the radiusof the finger touching the screen at the beginning of the flick.
Moreover, the abundance of no. 4 (Previous Consonant)indicates a substantial influence of the character enteredprior to the current one, meaning the position of the finger
before input. Furthermore, the prevalence of no. 15 (3-Axis
Composite Angular Velocity) suggests that the features
related to flick input operations manifest not only on thetouchscreen but also in the device’s motion. Thisobservation is further supported by the relatively highoccurrence of no. 14 (3-Axis Composite Acceleration).
According to Table 3, the authentication accuracy isapproximately 5% lower in this experiment than in theprevious studies. This difference can be attributed to thevariations in the data used. In the previous studies, bothtraining and validation data were based on the same dataset
or specific texts chosen by the examiners. In contrast, ourexperiment utilized free-text input. Moreover, ourexperiment assumed the text that users typically input and5.7 Experimental results
Authentication accuracy was evaluated on the basis of
the method outlined in 5.6. First, we calculated theauthentication accuracy for each feature vector. In this case,the parameters for OCSVM were set with the nu value of
0.9 to control the proportion of outliers and the kernel
function of rbf (radial basis function)15). These parameters
were determined on the basis of the results of preliminary
experiments. Table 2 presents the top five combinations offeature vectors that showed the highest authentication
accuracy for each vowel under these settings. The numbersin the “Feature vector combinations” column in Table 2
correspond to the feature numbers listed in Table 1.
Additionally, the results of comparing the authentication
accuracy obtained in this experiment with those fromprevious studies10)–12) are presented in Table 3. The average
False Reject Rate (FRR) and False Accept Rate (FAR) for
the feature vector combinations shown in Tables 2 and 3
were approximately 28% and 7%, respectively. In thisexperiment, we did not consider feature variation over time
when there is a time gap between the collection of training

53

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Table 2 Combinations of feature vectors with high authentication
accuracy for each vowel and their authentication accuracy

Vowel
“a”

“i”

“u”

“e”

“o”

Feature vector combinations
2, 4, 11, 14, 15, 16
2, 4, 11, 15, 16, 18
2, 4, 11, 14, 15, 16, 18
2, 4, 11, 15, 16
2, 11, 14, 15, 16
1, 2, 11, 13, 15, 17
1, 2, 3, 11, 13, 15, 17
1, 2, 11, 13, 15, 16, 17
1, 2, 3, 4, 11, 13, 15, 17
1, 2, 3, 8, 11, 13, 15, 17
16, 17
1, 4, 7, 10, 11, 16, 17
1, 4, 7, 10, 11, 15, 16, 17
1, 4, 7, 10, 11, 15, 16, 17, 18
1, 4, 7, 10, 11, 17
3, 6, 10, 11, 13, 15, 16, 17
3, 4, 6, 10, 12, 13, 15, 16, 17
3, 6, 8, 10, 11, 13, 15, 16, 17
3, 6, 9, 10, 11, 13, 15, 16, 17
3, 6, 10, 11, 13, 14, 15, 16, 17
4, 6, 10, 11, 12
4, 6, 8, 10, 11, 12
4, 6, 9, 10, 11, 12
4, 6, 10, 11, 12, 14
4, 6, 10, 11, 12, 15

Table 3 Comparison of authentication accuracy

Accuracy (%)

Accuracy (%)
91.46
91.46
91.46
91.42
91.42
91.30
91.30
91.30
91.30
91.30
91.89
90.94
90.89
90.89
90.83
92.54
92.54
92.54
92.54
92.54
92.74
92.74
92.74
92.74
92.74

Proposed

Vowel “a”
Vowel “i”
Vowel “u”
Vowel “e”
Vowel “o”

91.46
91.30
91.89
92.54
92.74

Izumi et al.10)
Ito and Shiraishi11)

96
Vowel “a”
Vowel “i”
Vowel “u”
Vowel “e”
Vowel “o”
Multi-classclassification
2-class classification

98.69
95.04
92.25
92.09
94.72
94.1

Motoyama et al.12)

97.8

effectiveness of continuous authentication on the basis ofthese extracted features.
Future work will include an evaluation of the entire
performance of the proposed system that includes the flick
input features described in this paper. Our principal areas ofstudy will include the selection of text to be used as training
data, experimental evaluations with an increased number ofparticipants and variation in features over time, andevaluations considering the balance between resource
consumption and authentication accuracy through theappropriate selection of authentication timing and methods
to overcome the issue of resource consumption duringsystem operation in practical implementation18).

utilized different text for each participant as validation data,
resulting in relatively shorter input texts. These factors arelikely responsible for the observed decrease inauthentication accuracy when comparing this study withprevious ones.
On the basis of the above results, in the context ofcontinuous authentication on smartphones using flick input
features, authentication accuracy tends to be lower when
free text is used than when fixed text is used. However, byappropriately selecting suitable features, a certain level ofauthentication accuracy can potentially be maintained.

Acknowledgment
This work was supported by JSPS KAKENHI Grant
Number JP20K11814.

References
1) P. A. Tresadern, C. McCool, N. Poh, P. Matejka, A. Hadid, C. Levy, T. F.
Cootes, S. Marcel: “Mobile Biometrics: Combined Face and Voice
Verification for A Mobile Platform”, IEEE Pervasive Computing,
Vol.12, No.1, pp. 79–87 (2013).
2) Lookout, “Phone Theft in America: What Really Happens When Your

6. Conclusion

Phone Gets Grabbed”,
https://www.lookout.com/blog/phone-theft-in-america (2014).

In this paper, we focused on continuous authentication
on smartphones utilizing flick input features. We aimed to
extract user-specific features from Japanese free text input
through flick operations in daily use and evaluated the3) Y. Yamazaki, T. Ohki: “Toward More Secure and Convenient User
Authentication in Smart Device Era”, IEICE Trans. on Information &Systems, Vol.E100-D, No.10, pp. 2391–2398 (2017).

54

IIEEJ Transactions on Image Electronics and Visual Computing Vol.12 No.1 （2024）

Distribution”, Neural Computation, Vol.13, No.7, pp. 1443–1471

4) V. M. Patel, R. Chellappa, D. Chandra, B. Barbello: “Continuous User

(2001).

Authentication on Mobile Devices – Recent Progress and Remaining16) Aozorabunko: “Hashire Merosu [Run, Melos]”,

Challenges”, IEEE Signal Processing Magazine, Vol.33, No.4, pp.

https://www.aozora.gr.jp/cards/000035/files/1567_14913.html (2000).

49–61 (2016).
5) T. Agawa, T. Higashi, Y. Yamazaki, T. Ohki: “A Study on Continuous17) M. M. Breunig, H. P. Kriegel, R. T. Ng, J. Sander: “LOF: Identifying

User Authentication for Smart Devices Considering Resource

Density-Based Local Outliers”, ACM SIGMOD Record, Vol.29, No.2,

Consumption (in Japanese)”, IEICE technical report, Vol.117, No.236,

pp.93–104 (2000).
18) Y. Watanabe, Y. Yamazaki: “A Study on Continuous Authentication

Biox2017-25, pp. 1–6 (2017).
6)

M. Ehatisham-ul-Haq, M.A. Azam, U. Naeem, Y. Amin, J. Loo:

Considering Implementation on Smart Devices (in Japanese)”, The“Continuous Authentication of Smartphone Users Based on Activity

Journal of the Institute of Image Electronics Engineers of Japan, Vol.52,

Pattern Recognition Using Passive Mobile Sensing”, Journal of

No.1, pp. 205–213 (2023).

Network and Computer Applications, Vol.109, pp.24–35 (2018).
7)(Received November 12, 2023)(Revised March 27, 2024)

F. Yao, S. Y. Yerima, B. Kang, S. Sezer: “Continuous Implicit
Authentication for Mobile Devices Based on Adaptive Neuro-Fuzzy
Inference System”, Proc. of 2017 International Conference on Cyber
Security and Protection of Digital Services (Cyber Security), pp.1–7

8)(2017).

Shuto KINOSHITA

T. Karanikiotis, M. D. Papamichail, K. C. Chatzidimitriou, N. -C. I.

He graduated from the Department ofInformation and Media Engineering at the
University of Kitakyushu in 2022. He completed
the master's program in Computer Science
Course at the Graduate School of Environmental
Engineering at the University of Kitakyushu in
2024. He engaged in research on biometricauthentication on smartphones during theacademic years.

Oikonomou, A. L. Symeonidis, S. K. Saripalle: “Continuous Implicit
Authentication through Touch Traces Modelling”, Proc. of 2020 IEEE
20th International Conference on Software Quality, Reliability andSecurity (QRS), pp.111–120 (2020).
9)

S. Kinoshita, Y. Watanabe, Y. Yamazaki: “Continuous Authenticationfor Smartphones Using Face Images and Touch-Screen Operation”,
Proc. of 2022 Asia-Pacific Signal and Information Processing

Yasushi YAMAZAKI(Member)
He received the B.E. and M.E. degrees inelectronics and communication engineeringfrom Waseda University, Tokyo, Japan, in 1993
and 1995, respectively, and the Ph.D. degree inelectronics, information and communicationengineering from Waseda University in 1998.
He is currently a professor in the Department ofInformation Systems Engineering at the
University of Kitakyushu, Fukuoka, Japan. He
received the IIEEJ paper award in 2004 and the
IIEEJ best paper award in 2012. His researchinterests include biometrics and information
security. He is a member of IIEEJ, IEICE, IPSJ,IEEE and ACM.

Association Annual Summit and Conference (APSIPA ASC), pp.
1556–1560 (2022).
10) M. Izumi, T. Samura, H. Nishimura: “Keystroke Dynamics with Flick
Input on Smartphone in Japanese Free Text Typing (in Japanese)”, Proc.
of the Information Processing Society of Japan Kansai Branch
Conference 2013, E-15 (2013).
11) S. Ito, Y. Shiraishi: “Proposal of Continuous Authentication Method
Focusing on The Features of Smartphone Flick Input Method (inJapanese)”, Proc. of the 25th Multimedia Communication andDistributed Processing System Workshop, pp.1–8 (2017).
12) W. Motoyama, S. Fukumoto, M. Kashima, K. Sato, M. Watanabe:“Studies on Biometrics Authentication by Behavioral Features at Flick
Input (in Japanese)”, IEICE technical report, Vol.119, No.445,
BioX2019-68, pp. 35–40 (2020).
13) JustSystems Corporation: “2 out of 3 Teenagers Use Flick Input toInput Text on Smartphones”,
https://prtimes.jp/main/html/rd/p/000000098.000007597.html (2015).
14) F. Bond, T. Baldwin: “Introduction to Japanese Computational
Linguistics”, in Readings in Japanese Natural Language Processing,edited by F. Bond, T. Baldwin, K. Inui, S. Ishizaki, H. Nakagawa, A.
Shimazu, pp. 1–28, CSLI Publication (2016).
15) B. Schoelkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, R. C.
Williamson: “Estimating The Support of A High-Dimensional

55

Call for Papers
Special Issue on
Image Electronics Technologies Related to VR/AR/MR/XR
IIEEJ Editorial Committee
In recent years, Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR),
Extended/Cross Reality (XR) have been applied to systems of various fields, and have brought
revolutionary innovation in them. The applied area includes Entertainment, Medical & HealthCare, Education, Practice & Disciplining, Manufacturing, Remote Work & Meeting, and so on,
and has been extended even for the purposes of Art and Environment Preservation. Especially, VR
technology will provide fully virtual environment and invite the users to another world in order toexperience unfeasible experiments. Also, AR technology will improve the efficiency of daily task
by shortening the access time to the desired information, and will empower the process tounderstand the individual utilized context at the same time.
On the other hand, the long-time effect of such technologies on human senses and mental stateshas been gathering wide interest, and how to coexist with such technologies will be the future big
theme for human beings.
In this special issue, we invite various categories (Ordinary paper, Short paper, Systemdevelopment paper, Practice Oriented Paper) of papers on designing and implementing
technologies that support VR/AR/MR/XR, and on the investigation and evaluation of suchtechnologies and systems being developed. We look forward to receiving your contributions.
1. Topics covered include but not limited to・Technologies and Systems on VR/AR/MR/XR, Meta-verse
・User Interface，Human Computer Interaction, Mutual Interaction
・Computer Graphics, Image Processing, Image Communication
・Computer Vision, Image Understanding, Pattern Recognition, Machine Learning, AI
・Collaboration, Education, Training, Medical Application, Remote Work/Meeting/Entertainment
2. Treatment of papers
Submission paper style format and double-blind peer review process are the same as theregular paper. If the number of accepted papers is less than the minimum number for the
special issue, the acceptance paper will be published as the regular contributed paper. We ask
for your understanding and cooperation.
3. Publication of Special Issue:
IIEEJ Transactions on Image Electronics and Visual Computing Vol.13, No.1 (June 2025)
4. Submission Deadline:
Tuesday, October 31, 2024
5. Contact details for Inquiries:
IIEEJ Office E-mail: hensyu@iieej.org
6. Online Submission URL:

http://www.editorialmanager.com/iieej/

56

Revised: January 6, 2017
Revised: July 6, 2018

Guidance for Paper Submission
1.

Submission of Papers(1) Preparation before submission
・ The authors should download “Guidance for Paper Submission” and “Style Format” from the“Academic Journals”, “English Journals” section of the Society website and prepare the paperfor submission.
・ Two versions of “Style Format” are available, TeX and MS Word. To reduce publishing costs
and effort, use of TeX version is recommended.
・ There are four categories of manuscripts as follows: Ordinary paper: It should be a scholarly thesis on a unique study, development orinvestigation concerning image electronics engineering. This is an ordinary paper topropose new ideas and will be evaluated for novelty, utility, reliability andcomprehensibility. As a general rule, the authors are requested to summarize a paper
within eight pages.
 Short paper: It is not yet a completed full paper, but instead a quick report of the partial
result obtained at the preliminary stage as well as the knowledge obtained from the said
result. As a general rule, the authors are requested to summarize a paper within four
pages.
 System development paper: It is a paper that is a combination of existing technology or it
has its own novelty in addition to the novelty and utility of an ordinary paper, and thedevelopment results are superior to conventional methods or can be applied to othersystems and demonstrates new knowledge. As a general rule, the authors are requested
to summarize a paper within eight pages.
 Data Paper: A summary of data obtained in the process of a survey, product development,
test, application, and so on, which are the beneficial information for readers even though
its novelty is not high. As a general rule, the authors are requested to summarize a paper
within eight pages.
・ To submit the manuscript for ordinsry paper, short paper, system development paper, or data
paper, at least one of the authors must be a member or a student member of the society.
・ We prohibit the duplicate submission of a paper. If a full paper, short paper, systemdevelopment paper, or data paper with the same content has been published or submitted toother open publishing forums by the same author, or at least one of the co-authors, it shallnot be accepted as a rule. Open publishing forum implies internal or external books,
magazines, bulletins and newsletters from government offices, schools, company
organizations, etc. This regulation does not apply to a preliminary draft to be used at anannual meeting, seminar, symposium, conference, and lecture meeting of our society or othersocieties (including overseas societies). A paper that was once approved as a short paper andbeing submitted again as the full paper after completion is not regarded as a duplicate
submission.
(2) Submission stage of a paper・ Delete all author information at the time of submission. However, deletion of reference
information is the author’s discretion.
・ At first, please register your name on the paper submission page of the following URL, andthen log in again and fill in the necessary information. Use the “Style Format” to upload yourmanuscript. An applicant should use PDF format (converted from dvi of TeX or MS Word57

format) for the manuscript. As a rule, charts (figures and tables) shall be inserted into themanuscript to use the “Style Format”. (a different type of data file, such as audio and video,
can be uploaded at the same time for reference.)
http://www.editorialmanager.com/iieej/
・ If you have any questions regarding the submission, please consult the editor at our office.
Contact:
Person in charge of editing
The Institute of Image Electronics Engineers of Japan
3-35-4-101, Arakawa, Arakawa-Ku, Tokyo 116-0002, Japan
E-mail: hensyu@iieej.org
Tel: +81-3-5615-2893, Fax: +81-3-5615-2894

2.

Review of Papers and Procedures(1) Review of a paper・ A manuscript is reviewed by professional reviewers of the relevant field. The reviewer willdeem the paper “acceptance”, “conditionally acceptance” or “returned”. The applicant is
notified of the result of the review by E-mail.
・ Evaluation method
Ordinary papers are usually evaluated on the following criteria:
 Novelty: The contents of the paper are novel.
 Utility: The contents are useful for academic and industrial development.
 Reliability: The contents are considered trustworthy by the reviewer.
 Comprehensibility: The contents of the paper are clearly described and understood by
the reviewer without misunderstanding.
Apart from the novelty and utility of an ordinary paper, a short paper can be evaluated byhaving a quickness on the research content and evaluated to have new knowledge withresults even if that is partial or for specific use.
System development papers are evaluated based on the following criteria, apart from thenovelty and utility of an ordinary paper.
 Novelty of system development: Even when integrated with existing technologies, thenovelty of the combination, novelty of the system, novelty of knowledge obtained fromthe developed system, etc. are recognized as the novelty of the system.
 Utility of system development: It is comprehensively or partially superior compared tosimilar systems. Demonstrates a pioneering new application concept as a system. The
combination has appropriate optimality for practical use. Demonstrates performance
limitations and examples of performance of the system when put to practical use.
Apart from the novelty and utility of an ordinary paper, a data paper is considered novel if
new deliverables of test, application and manufacturing, the introduction of new technology
and proposals in the worksite have any priority, even though they are not necessarily
original. Also, if the new deliverables are superior compared to the existing technology andare useful for academic and industrial development, they should be evaluated.
(2) Procedure after a review
・ In case of acceptance, the author prepares a final manuscript (as mentioned in 3.).
・ In the case of acceptance with comments by the reviewer, the author may revise the paper inconsideration of the reviewer’s opinion and proceed to prepare the final manuscript (as58mentioned in 3.).
・ In case of conditional acceptance, the author shall modify a paper based on the reviewer’s
requirements by a specified date (within 60 days), and submit the modified paper for
approval. The corrected parts must be colored or underlined. A reply letter must be attachedthat carefully explains the corrections, assertions and future issues, etc., for all of theacceptance conditions.
・ In case a paper is returned, the author cannot proceed to the next step. Please look at thereasons the reviewer lists for the return. We expect an applicant to try again after reviewing
the content of the paper.
(3) Review request for a revised manuscript
・ If you want to submit your paper after conditional acceptance, please submit the reply letterto the comments of the reviewers, and the revised manuscript with revision history to thesubmission site. Please note the designated date for submission. Revised manuscripts
delayed more than the designated date be treated as new applications.
・ In principle, a revised manuscript willl be reviewed by the same reviewer. It is judged either
accceptance or returned.
・ After the judgment, please follow the same procedure as (2).

3.

Submission of final manuscript for publication(1) Submission of a final manuscript
・ An author, who has received the notice of “Acceptance”, will receive an email regarding thecreation of the final manuscript. The author shall prepare a complete set of the final
manuscript (electronic data) following the instructions given and send it to the office by thedesignated date.
・ The final manuscript shall contain a source file (TeX edition or MS Word version) and a PDF
file, eps files for all drawings (including bmp, jpg, png), an eps file for author’s photograph(eps or jpg file of more than 300 dpi with length and breadth ratio 3:2, upper part of the body)for authors’ introduction. Please submit these in a compressed format, such as a zip file.
・ In the final manuscript, write the name of the authors, name of an organizations,introduction of authors, and if necessary, an appreciation acknowledgment. (cancel macros in
the Style file)・ An author whose paper is accepted shall pay a page charge before publishing. It is theauthor’s decision to purchase offprints. (ref. page charge and offprint price information)(2) Galley print proof
・ The author is requested to check the galley (hard copy) a couple of weeks before the paper ispublished in the journal. Please check the galley by the designated date (within one week).
After making any corrections, scan the data and prepare a PDF file, and send it to our officeby email. At that time, fill in the Offprint Purchase Slip and Copyright Form and return thescanned data to our office in PDF file form.
・ In principle, the copyrights of all articles published in our journal, including electronic form,
belong to our society.
・ You can download the Offprint Purchase Slip and the Copyright Form from the journal on our
homepage. (ref. Attachment 2: Offprint Purchase Slip, Attachment 3: Copyright Form)(3) Publication
・ After final proofreading, a paper is published in the Academic journal or English transaction(both in electronic format) and will also be posted on our homepage.

59

IIEEJ Trans. on Image Electronics and Visual Computing
Vol. 12 No. 1 2024

Editor in Chief: Osamu Uchida
The Institute of Image Electronics Engineers of Japan
3-35-4-101，Arakawa，Arakawa-ku，Tokyo 116-0002，Japan©2024 IIEEJ