IWIN202
International Workshop on Informatics

3URFHHGLQJVRI,QWHUQDWLRQDO:RUNVKRSRQ,QIRUPDWLFV

6SRQVRUHGE\,QIRUPDWLFV6RFLHW\IWIN20
International Workshop on Informatics

3URFHHGLQJVRI,QWHUQDWLRQDO:RUNVKRSRQ,QIRUPDWLFV

6SRQVRUHGE\,QIRUPDWLFV6RFLHW\

3XEOLFDWLRQRIILFH
,QIRUPDWLFV/DERUDWRU\7VXMLPDFKL.LWDNX1DJR\D-DSDQ
3XEOLVKHU7DGDQRUL0L]XQR3UHVLGHQWRI,QIRUPDWLFV Society,6%1
3ULQWHG LQ-DSDQ7DEOHRI&RQWHQWV
6HVVLRQ$UWLILFLDO,QWHOOLJHQFH··································· 1
&KDLU.HL+LURL  6HS (1)

Improve Measuring Suspiciousness of Bugs in Spectrum-Based Fault
Localization With Deep Learning ···················································· 3
Hitoshi Kiryu, Shinpei Ogata, and Kozo Okano(2)

Automated Scoring Gateball Realtime with Camera, YOLO, and
RTSP ····················································································· 9
Trong Pham Dinh, Mikiko Sode Tanaka,
Ami Uchikata, and Kouki Ishioka(3)

Automatic Derivation of a Transition Model from a JapaneseRequirement Specification under a Restricted Grammar ························ 13
Koki Shimokawa, Hiroya Ii, Maiko Onishi, Shinpei Ogata, and Kozo
Okano

L



6HVVLRQ6\VWHPVDQG6HUYLFHV ·································· 21
&KDLU7DNX\D<RVKLKLUR   6HS
(4)

Design of Air Flow Channel for the Droplet Spray Type Olfactory
Display ················································································· 23
Yohei Seta, Mitsunori Makino, Yuichi Bannai,
and Motofumi Hattori(5)

3URSRVDORI3RWWHG)ORZHU$UHD6HJPHQWDWLRQ0HWKRG
8VLQJ+RXJK7UDQVIRUP·····āāāāāāāāā················································3
Shun Fujii, Yoshitaka Nakamura, Hiroshi Inamura,and Shigemi Ishida(6)

A Feature Data Distribution Scheme for Person Tracking Systems
with Multiple Cameras································································ 3

Satoru Matsumoto and Tomoki Yoshihisa

6HVVLRQ0XOWLPHGLD ······················································ 45
&KDLU7RPRNL<RVKLKLVD   6HS
(7)

Stationary Human Recognition by Multiple Vertically Arranged
2D-LiDARs ············································································· 47
Takuya Watanabe, Yuya Sawano, Yoshiaki Terashima,
and Ryozo Kiyohara(8)

Development and Evaluation of a Broadcaster Support Method using
MR Stamps in 360-degree Internet Live Broadcasting ··························· 53
Yoshia Saito and Kei Sato

(9)

Object Recognition Method Using Locus of Gestures Detectedby YOLOv5 ············································································ 61
Tsukasa Kudo
LL

6HVVLRQ'LVDVWHU0LWLJDWLRQ ····································· 69
&KDLU$NLKLUR+D\DVKL   6HS
(10)

An Analysis of Tweets in Disasters for Extracting Rescue Requests··························································································· 71
Yuki Koizumi, Junji Takemasa, Toru Hasegawa,and Yoshinobu Kawabe(11)

How Can SNS Data Contribute to Disaster Damage Assessment? ············ 79
Kei Hiroi, Akihito Kohiga, and Yoichi Shinoda(12)

Actual Condition of DisasterPrevention Radio Broadcasting in Atsugi City and
Its Artificial Intelligence-Based Discrimination of Difficulty in Listening··· 
Takumi Hashimoto, Takahiro Miura, Hideo Kasuga,
Tetsuo Tanaka, Yoshimichi Ogawa, and Mari Ueda

6HVVLRQ6RFLDO1HWZRUNDQG'DWD$QDO\VLV······9
&KDLU.R]R 2NDQR   6HS
(13)

Verification of the Best Time to See Cherry Blossoms Estimation Method
using Time Series Prediction Method··············································9Junya Sato, Yusuke Takamori, Masahiro Fujimoto,Tomonari Horikawa, Masaki Endo, Shigeyoshi Ohno,
Masaharu Hirota, and Hiroshi Ishikawa(14)

A Study of Different Values between Individuals and Companies
Regarding Personal Information ··················································· 103
Mayu Satake and Hideki Goromaru

LLL

6HVVLRQ&RPPXQLFDWLRQ ··········································· 113
&KDLU.DWVXKLNR.DML   6HS(15)

Organization Training Formulation Method in Project Management - Building
a Skills Matrix with Rubric Assessment ········································· 115
Akihiro Hayashi, Michi Komura, and Midori Ishihara(16)

Proposal for Continuous Improvement of Empowerment Arts Therapy- A Method of Self-Assessment by Applying Rubric - ························ 121
Michi Komura, Midori Ishihara and Akihiro Hayashi(17)

A Study of Progress Management Method using TA-Bot for Software
Development Class ································································· 127
Koichi Ono, Ryozo Kiyohara, and Yoshiaki Terashima
Satoru Matsumoto and Tomoki Yoshihisa(18)

An Experiment of an Office Worker Passing by the Surrogate Robotof a Remote Worker ································································ 133
Kosuke Sasaki, Zijie Yuan, and Tomoo Inoue.H\QRWH6SHHFK ································································· 139
&KDLU7RPR\XNL<DVKLUR   6HS(I)

Toward Software Defined Social Infrastructure based on Cyber Physical
Systems Technology ······························································· 141
Mr. Takeshi Saito, Information & Communication Platform
Laboratories, Corporate Research & Development Center,Toshiba Corp.

LY

6HVVLRQ$SSOLFDWLRQ ·····················································163
&KDLU7RPR\D.LWDQL   6HS
(19)

Development of an Avatar Remote Communication System for ALS Patients
Using Video Conferencing Tools ·················································· 165
Nagisa Kokubu, Riku Sugimoto, Shouma Hamada, Takayo Namba,
and Keiichi Abe(20)

Pedestrian Cooperative Autonomous Mobility - Path Planning Adapted to
Pedestrian Face Direction - ························································ 171
Yuto Yada, Shunsuke Michita, Seiji Komiya, and Toshihiro Wakita(21)

Implementation of Spatio-temporal Fencing for Crowd Sensing in a
Smartphone Applications·························································· 177

Shota Suzaki, Nobuhito Miyagawa, and Katsuhiko Kaji
(22)

Sleep Effects on Noise. - Comparison of Sleep Effects Between Students andWorking People····································································· 15
Tatsuki Takahashi, Masaaki Hiroe, and Mari Ueda

6HVVLRQ,R7DQG1HWZRUN········································189
&KDLU0DVDVKL6DLWR  6HS(23) A Study on Application of Zero Trust Architecture to IoT Actuators············ 191

Nobuhiro Kobayashi(24)

Implementation on the Evaluation Platform for Large-Scale BACS ········ 201

Kohei Miyazawa, Tetsuya Yokotani, and Hiroaki Mukai(25) Firmware Distribution with Erasure Coding for IoT Devices·····················207

Takenori Sumi, Yukimasa Nagai, and Hiroshi Mineno

Y0HVVDJHIURPWKH*HQHUDO&KDLUVIt is our great pleasure to welcome all of you to Nachi-Katsuura, Wakayama, Japan for
the 16th International Workshop on Informatics (IWIN 2022). This workshop has been
held annually by the Informatics Society. Since 2007, the workshops were held in Naples
in Italy, Wien in Austria, Hawaii in the USA, Edinburgh in Scotland, Venice in Italy,
Chamonix in France, Stockholm in Sweden, Prague in Czech Republic, Amsterdam in
Netherlands, Riga in Latvia, Zagreb in Croatia, Salzburg in Austria, Hamburg in Germany,
Wakayama in Japan (virtually), and Fukui in Japan (virtually), respectively.
In IWIN 2022, 25 papers were accepted after peer reviewing by the program committee.
Based on the papers, eight technical sessions were organized in a single-track format,
which highlighted the latest research results in the areas such as Artificial Intelligence(AI), Systems and Services, Multimedia, Disaster Mitigation, Social Network and Data
Analysis, Communication, Application, and Internet of Things (IoT) and Network.
IWIN2022 will also welcome one keynote speaker: Mr. Takeshi Saito, general manager
of Information and Communication Platform Laboratories of Corporate R&D Center,Toshiba Corp. We really appreciate his participation in the workshop.
We would like to thank all the participants and contributors who made the workshop
possible. It is indeed an honor to work with a large group of professionals around theworld to make the workshop a great success. We are looking forward to seeing you all in
the workshop. We hope you enjoy IWIN 2022.
August 2022
Tomoyuki Yashiro
Hideki Goromaru

viiOrganizing Committee
General Co-Chairs
Tomoyuki Yashiro (Chiba Institute of Technology, Japan)
Hideki Goromaru (Chiba Institute of Technology, Japan)

Steering Committee
Hitoshi Aida (The University of Tokyo, Japan)
Toru Hasegawa (Osaka University, Japan)
Teruo Higashino (Kyoto Tachibana University, Japan)
Tadanori Mizuno (Aichi Institute of Technology, Japan)
Jun Munemori (The Open University of Japan, Japan)
Yuko Murayama (Tsuda University, Japan)
Ken-ichi Okada (Keio University, Japan)
Norio Shiratori (Chuo University / Tohoku University, Japan)
Osamu Takahashi (Future University Hakodate, Japan)

Program Chair
Kei Hiroi (Kyoto University, Japan)

Financial Chair
Tomoya Kitani (Shizuoka University, Japan)

Publicity Chair
Yoshitaka Nakamura (Kyoto Tachibana University, Japan)

Program Committee
Akihiro Hayashi

Hironobu Abe (Tokyo Denki University, Japan)(Shizuoka Institute of Science and Technology, Japan)

Hiroshi Inamura (Future University Hakodate, Japan)

Akihito Hiromori (Osaka University, Japan)

Hiroshi Mineno (Shizuoka University, Japan)

Fumiaki Sato (Toho University, Japan)

Hiroshi Sugimura

Hideyuki Takahashi (Tohoku Gakuin University, Japan)(Kanagawa Institute of Technology, Japan)viii

Tetsuya Yokotani

Hiroshi Yoshiura (Kyoto Tachibana University, Japan)(Kanazawa Institute of Technology, Japan)

Kanae Matsui (Tokyo Denki University, Japan)
Katsuhiko Kaji (Aichi Institute of Technology, Japan)

Tomoki Yoshihisa (Osaka University, Japan)

Kazuyuki Iso (NTT, Japan)

Tomoo Inoue (University of Tsukuba, Japan)

Kei Utsugi (Hitachi, Ltd, Japan)

Tsukasa Kudo
(Shizuoka Institute of Science and Technology, Japan)

Keiichi Abe (Kanagawa Institute of Technology, Japan)
Ken Ohta (NTT DOCOMO, Japan)

Yoshia Saito (Iwate Prefectural University, Japan)

Kozo Okano (Shinshu University, Japan)

Yoshiaki Terashima (Soka University, Japan)

Makoto Imamura (Tokai University, Japan)

Yoshinobu Kawabe(Aichi Institute of Technology, Japan)

Mari Ueda (Kanagawa Institute of Technology, Japan)
Masaji Katagiri (iU, Japan)

Yu Enokibori (Nagoya University, Japan)

Masakatsu Nishigaki (Shizuoka University, Japan)

Yuichi Bannai(Kanagawa Institute of Technology, Japan)

Masaki Endo (Polytechnic University, Japan)

Yuichi Tokunaga

Masaki Nagata (Shizuoka University, Japan)(Kanazawa Institute of Technology, Japan)

Michiko Oba (Future University Hakodate, Japan)

Yuki Koizumi (Osaka University, Japan)

Mikiko Sode

Yukimasa Nagai(Kanazawa Institute of Technology, Japan)
Minoru Kobayashi (Meiji University, Japan)(Mitsubishi Electric Corporation, Japan)

Naoya Chujo (Aichi Institute of Technology, Japan)

Yusuke Gotoh (Okayama University, Japan)

Orie Abe

Yusuke Ichikawa(NTT Service Evolution Laboratories, Japan)(Information Communication Media Laboratory, Japan)
Ryozo Kiyohara(Kanagawa Institute of Technology, Japan)
Satoru Matsumoto (Osaka University, Japan)
Shigemi Ishida (Future University Hakodate, Japan)
Shigeyoshi Ohno (Polytechnic University, Japan)
Shinichiro Mori (Chiba Institute of Technology, Japan)
Shinji Kitagami(Fukui University of Technology, Japan)
Takaya Yuizono(Japan Advanced Institute of Science and Technology)
Takayasu Yamaguchi(Akita Prefectural University, Japan)
Takuya Yoshihiro (Wakayama University, Japan)
Tetsuya Shigeyasu(Prefectural University of Hiroshima, Japan)ixInternational Workshop on Informatics ( IWIN 2022 )Improve Measuring Suspiciousness of Bugs in Spectrum-Based Fault Localization
With Deep Learning
Hitoshi Kiryu† , Shinpei Ogata‡ , and Kozo Okano‡†

Graduate School of Engineering, Shinshu University, Japan†
21w2025g@shinshu-u.ac.jp‡
Faculty of Engineering, Shinshu University, Japan‡{ogata, okano}@cs.shinshu-u.ac.jp

Abstract - Localizing Faults is integral for debugging in developing software. Spectrum-Based Fault Localization (SBFL)
is a technique to localize faults. SBFL calculates the suspiciousness score for each line in a program using code coverage oftests. Some studies apply a deep neural network to SBFL. In
these studies, the suspiciousness is calculated by giving a virtual
coverage to a trained network. This paper proposes a method to
calculate the suspiciousness score of lines in a program from theprogram spectrum with deep learning. The method also provides
a ranking based on the score. In calculating the suspiciousness
score, We focus on the diﬀerence between the coverage of afailed test and an alteration of the coverage. As a result of theevaluation of the proposed method, it is conﬁrmed that the effectiveness of the proposed method is comparable to Ochiai andmore eﬀective than the suspiciousness calculation method usedin other studies.
Keywords: SBFL, Fault Localization, Deep Learning
Figure 1: Example of SBFL

1 INTRODUCTION
When a problem caused by code is found in software development or maintenance, it is necessary to localize and ﬁx bugs.
Generally, such a debugging needs a lot of time and humanworks. Many techniques to localize faults and ﬁx bugs havebeen studied to support developers in debugging. One example
of a bug-ﬁxing technique is GenProg[1], which outputs code thatpasses all the test suites with a genetic algorithm. In techniqueto localizing bugs, Various studies [2]–[5] have been conducted.
These studies proposed methods to identify statements that cause
bugs by using bug reports, trace information, and visualization.
Another technique for localizing the fault is Spectrum-BasedFault Localization (SBFL). SBFL localizes faults in the sourcecode based on the code coverage and test results of each test.
It calculates the suspiciousness of each statement containing a
bug and provides a ranking based on the suspiciousness. The
basic idea of SBFL is that a line executed in a failed test is more
likely to contain a bug, while a line executed in a successful testis less likely to contain a bug. The metrics for calculating thesuspiciousness of a bug are based on four values described inTable 1.

Table 1: Four Values for Calculation of Suspiciousness
Number of failed tests that execute the program element.
Number of passed tests that execute the program element.
Number of failed tests that do not executethe program element.
np Number of passed tests that do not executethe program element.

ef
ep
nf

ef
Ochiai = (ef + nf )(ef + ep )(Ochiai)ef N
ep + nf(Dstar)Dstar(∗ = N ) =For example, Ochiai[6] and Dstar[7] proposed following formulas. Here, ‘N’ is the exponent variable of ef . Suspiciousness
scores for each line are calculated by these formulas, and represents the probability that each line causes a bug. The score
tends to be higher if a line is executed more frequently when thetest fails.

3International Workshop on Informatics ( IWIN 2022 )Figure 2: Architecture of the Network
Figure 1 shows an example of SBFL. The source code in theﬁgure is a function that returns the result of FizzBuzz for a given
number i as input. The ﬁfth line of the function contains abug. This is because the conditional statement is incorrect. The
suspiciousness of each line is calculated by Ochiai based on the
results and the coverage of the tests. The suspiciousness is thehighest on lines ﬁfth and sixth, which are executed only when
the test fails. This indicates that the suspiciousness of the buggylines is calculated properly. Thus, from the test results and thecoverage, it is possible to identify the location of bugs.

a ranking of lines that are likely to cause bugs based on themeasured impact. In measuring the impact of the network on abug prediction, we focus on the change in bug prediction between
an actual failed coverage and virtual coverages that altered theactual coverage. We consider the diﬀerence between them as
the suspiciousness score.
The proposed method is better than the conventional method
in measuring the suspiciousness from the network. It is conﬁrmed that the proposed method gives a larger value of suspiciousness to the buggy line and a smaller value to the non-buggy
line than the conventional measuring method. While comparing
a metric Ochiai, no signiﬁcant diﬀerence was conﬁrmed between
the proposed method and Ochiai.
In the following sections, Section 2 describe the proposed
method. Section 3 presents the results in the evaluation experiments and Section 4 discuss the results in the experiments.
Finally, we conclude in Section 5.

Deep learning is a technology that has been showing results ina wide variety of ﬁelds, including image recognition and natural
language processing. The ﬁeld of fault localization is no exception either. Ikeda et al.[8] proposed the method that localizes afault method with a neural network. They trained the network
to predict test results from execution traces and used ablated
traces to localize a fault method. In the ﬁeld of SBFL, the study[9] has been conducted to compute the suspiciousness using
deep learning. In this study, three architectures, RNN (Recurrent Neural Network), CNN (Convolutional Neural Network),
and MLP (MultiLayer Perceptron), are proposed to computethe suspiciousness of bugs, and CNN shows the best results.
In another study[10], which calculates the suspiciousness using
RBF networks (Radial Basis Function Network), concluded that
RBF networks are more eﬀective than existing methods such as
Ochiai in fault localization. In these studies, in order to calculate
the suspiciousness, a virtual coverage which only a certain line
is executed is input to the trained network. The output of the
network is treated as the suspiciousness of the line have bugs,
and a ranking based on the suspiciousness is provided.

2 PROPOSED METHODThis section describes the methodology of SBFL using a Convolutional Neural Network in detail.
The proposed method provides suspiciousness values thateach line contains a bug in a source code and a ranking of
the suspiciousness. CNN shows the best result in Zhang et al.
[9]. That is why we use Convolutional Neural Network as thearchitecture of the neural network in the proposed method.
In the proposed method, ﬁrst, a neural network learns about arelation between the coverage of the source code in tests and thetest result. The network takes the coverage as input, and output
respectively probabilities of a test result, pass and fail. In order
to measure the impact on decision making about a test result in
the network, a coverage of in failed test and a virtual coveragethat altered the failed test coverage are input. We determine theimpact based on a diﬀerence between outputs of the original
coverage that fails the test and the virtual coverage. The impact
on decision making is considered as the suspiciousness of bugs.
The trained network cannot be applied to other source code. This

This paper proposes a method to compute the suspiciousnessof lines in the source code from the program spectrum with deeplearning. The proposed method shall identify a single defect inthe source code. Using a network trained to predict test resultsfrom the source code coverage of each test case, suspiciousness
scores are measured for each line.We measure the impact ofeach line on bug prediction in the trained network, and provide

4International Workshop on Informatics ( IWIN 2022 )Figure 3: Example of Calculation of Suspiciousness of Third Line.
is because the network learns the relation between a executionof lines in a certain source code and test results.

lines and the corresponding suspiciousness scores. Lines thatare not executed in failed tests are excluded from the ranking,because they are not likely to contain bugs.

2.1 Network Architecture and Training

3 EVALUATION EXPERIMENT

Figure 2 shows the structure of the CNN used in the proposed
method. It has two Convolutional layers, a MaxPooling layeron the output of the Convolutional layers, two Fully-Connected
layers, and one output layer. The sigmoid function is used in the
output layer, and ReLU is used for the other activation functions.
The kernel size of the convolutional layer, the number of thechannels, and the nodes of the Fully-Connected layer depend on
the target program size. The network takes a test coverage as
input and learns a nonlinear relationship between the coverageand the test results. The output of the network is P, and F ,which represent the probability of the test passing and failing,respectively.

In order to evaluate the proposed method, evaluation experiments are conducted on two OSS to see if the proposed method
is eﬀective in localizing faults. The coverage used in the experiments was collected using the tests provided with the OSS.
Most of the tests included in the OSS are those whose results
are passed. In this study, since the coverage with Pass and Failtest results is needed, we manually seeded a part of the code tocreate a program including bugs. The numbers in the tables inthis section are rounded up at the fourth decimal point.

3.1 Target Program
We choose the following two programs as target programs forthe experiments:

2.2 Extract Suspiciousness of a Bug• scikit-learn is a library for machine learning. A bug
is seeded in the program “kernel_approximation”. The
LOC of the source code is 1011. Tests attached to thelibrary collect the coverage. The tests are given additionalrandom seed values to increase the number of tests. A
total of 288 coverage tests were generated, including 138
failed tests and 150 successful tests, and are used in our
experiments.

A virtual coverage is inputted to the network after training
to measure the impact on deciding test results in the trained
network based on the diﬀerence between the virtual coverage andthe original coverage. Figure 3 shows an example of measuring
the impact of the third line. In order to calculate the impactof the execution of the third line on the determination of test
results in the network by examining the change in the outputof the original failed coverage and coverage that the third line
is changed to zero in original the coverage. Some studies[9],[10] treat the output of the virtual coverage in which only theline is executed as the score of the suspiciousness of a bug. We
believe that the diﬀerence between the original coverage and theoutput of the coverage with the altered execution more strongly
indicates the impact on the test results than such methods. If
a line, which is changed to zero in the coverage contributes
to a failure, the elimination of the execution might increase theprobability that the result is passed. Since the large change in thelikelihood of the network’s prediction signiﬁes the bigger impactof the execution, we consider the change to be the suspiciousness
score.
The values of suspiciousness are ranked in descending orderand output as a result of fault localization. The ranking presents• Html5lib is a library for parsing HTML. A bug is seededin the program “serializer”. The LOC of the source code
is 409. Tests attached to the library collect the coverage.
A total of 235 coverage tests were generated, including 23
failed tests and 212 successful tests, and are used in our
experiments.
The proposed method is applied to the coverage collectedfrom the above two OSS. Compare three bug suspiciousness
measurement methods including the proposed method. The
measurements compared are the following:• proposed-measure : A diﬀerence between outputs of original coverage that fails the test and the virtual coverage.

5International Workshop on Informatics ( IWIN 2022 )• one-hot : An output of coverage which only a certain line
is executed.

score. This is because their appearances in the coverage are thesame. In contrast, from Tables 2 and 4, we can see that the
buggy line is ranked in 21th. There is actually not much of adiﬀerence between the suspiciousness score of the buggy line
and the ﬁrst ranked line. As the result in Table 5, the buggy line
195 and non-buggy line 158 are tied for the top.

• Ochiai : Suspiciousness based on Ochiai described in
Section 1.

3.2 Experiment 1

3.3 Experiment 2

The score of the suspiciousness and the rank for seeded buggy
line 195 are calculated by three measurement methods. Table 2
shows the result of the methods.

The score of the suspiciousness and the rank for seeded buggy
line 330 are calculated by three measurement methods. Table 6
shows the result of the methods.

Table 2: Suspiciousness and Rank of The Buggy Linemeasurement
suspiciousness rank
proposed-measure
0.615
1
one-hot
0.525
21
Ochiai
0.818
1Table 6: Suspiciousness and Rank of The Buggy Linemeasurement
suspiciousness rank
proposed-measure
0.715
1
one-hot
0.526
1
Ochiai1.0
1Table 3: Top 5 of proposed-measure in experiment 1
Rank Line suspiciousness
1
195
0.615
2
158
0.589
3
199
0.007
4
194
0.006
5
981
0.005

Table 7: Top 5 of proposed-measure in experiment 2
Rank Line suspiciousness
1
330
0.715
2
331
0.019
3
329
0.011
4
325
0.007
5
327
0.007

Table 4: Top 5 of one-hot in experiment 1
Rank Line suspiciousness
1
720
0.528
2
981
0.528
3
668
0.527
4
449
0.527
5
951
0.527

Table 8: Top 5 of one-hot in experiment 2
Rank Line suspiciousness
1
330
0.526
2
327
0.518
3
337
0.514
4
321
0.512
5
305
0.512

Table 5: Top 5 of Ochiai in experiment 1
Rank Line suspiciousness
1
195
0.818
2
158
0.818
3
186
0.716
4
24
0.716
5
14
0.716

Table 9: Top 5 of Ochiai in experiment 2
Rank Line suspiciousness
1
330
1.0
2
329
0.804
3
336
0.793
4
335
0.793
5
325
0.793

The proposed-measure and Ochiai ranked the line seeded thebug ﬁrst, while one-hot ranked 21st.
Tables 3, 4 and 5 display the top ﬁve in the ranking based on
suspiciousness score of proposed-measure, one-hot and Ochiai
respectively. The line 195 is the line seeded a bug.
Table 3 shows that the measurement classiﬁes properly the
buggy line and other lines. Line 158’s score closes to line 195’s

All of the measurements ranked the line seeded the bug ﬁrst.
Table 7 shows that the measurement distinguished the buggy
line and the others. In Tables 6 and 8, the buggy line is ranked in
the ﬁrst. However, similarly to the experiment for scikit-learn,
the suspiciousness score of 21st is 0.499. This score hardly

6International Workshop on Informatics ( IWIN 2022 )In both experiments, the method was applied to more than
200 test cases and dozens of failed tests. The experiments arebuilt on adequate failed tests and passed tests. In repositories
that are a collection of actual bugs, it is frequent that there are
only one or two failed tests. In order to ensure the eﬀectivenessand generality of the method, it is required to apply the method
to such actual bugs.

diﬀers from the score of the ﬁrst. As the result in Table 9, thebuggy line 330 got score 1.0. Because this line is executed in
all failed tests.

4 DISCUSSION
In both experiments, there is not much diﬀerence among the
top 20 of the score that one-hot calculated. On the other hand,In both cases, we can see that the proposed-measure properly
scored the buggy line. There is a large diﬀerence between the
Top score and other lines. It is examined using Gini coeﬃcientwhether the three measurements give properly higher scoresto buggy lines, and lower scores to non-buggy lines. Sincesome scores of proposed-measure contain negative values, thediﬀerence values from the minimum are used in the calculation.
Tables 10 and 11 show Gini coeﬃcients of the three methods
in the two experiments. Each of the coeﬃcients in the Table
indicates that one-hot gives one-hot gives a fair score to each line,
whereas the other two measurements give an unbalanced score
to each line. From these results, we believe that the measurement
proposed-measure is more eﬀective than one-hot in whether abuggy line is determined a high suspiciousness score.

5 CONCLUSION
This paper proposes a method that combines SBFL and DNN.
The proposed method trains a network that predicts test results
based on coverage of a source code, and measures the impactof each line on bugs from the trained network, and provides
a ranking based on the measurement. In measuring the suspiciousness of bugs, we focus on the diﬀerence between the outputof the original coverage and the output of the virtual coveragethat altered the original coverage, and treat the diﬀerence value
as the impact of the bug. In the experiment, we obtained better results than the conventional measurement method, which
treats the output of a virtual coverage in which only a certain
line is executed as suspiciousness, and the proposed method iscomparable to Ochiai.
In future work, we want to apply this method to SBFL in code
blocks instead of lines. The occurrence of lines in a block must
be the same. Hence, if a line is determined to be highly possibly
buggy, any line in the block containing that line may be the same
possibly buggy. In addition, the experiments in this paper applied the method against artiﬁcially seeded faults in the projects
that have adequate quantity of test cases to evaluate the eﬀectiveness. We would like to conduct experiments against actual
bugs such as Defects4J[12] to further evaluate the eﬀectivenessof the proposed method.

Table 10: Gini Coeﬃcients in Experiment 1
proposed-measure one-hot Ochiai
0.637
0.079
0.323

Table 11: Gini Coeﬃcients in Experiment 2
proposed-measure one-hot Ochiai
0.783
0.028
0.973

ACKNOWLEDGEMENT

From the results of the experiments, It is conﬁrmed that the
proposed method is eﬀective to localize faults. The eﬀectivenessof the proposed method can be concluded it is comparable to
Ochiai and is more eﬀective than the one-hot method used inprevious studies.
In the scikit-learn experiment, line 158 ranked in second place
had the same appearance in coverage as line 195 containing thetrue bug, that is, if one is executed, the other is also executed.
Therefore, it is reckoned from the coverage that both lines areequally likely to cause a bug. This is similar to SBFL without
DNN, which calculates the probability of bugs based on the
coverage, and thus cannot distinguish between lines that have theidentical occurrence in the coverage. This results in a problem
that a impact on bugs is determined equally. This problem
can be solved by satisfying branch coverage to augment uniquecoverage in some cases, but this is not the case for unbranchedprograms. Therefore, it is necessary to narrow down the linesthat have a close probability of each other by using information
extracted from execution traces, source codes, error messages,etc.

Part of this work is supported by fund from Mitsubishi ElectricCorp.
The research is also being partially conducted as Grant-in-Aid
for Scientiﬁc Research A (18H04094) and C (21K11826).

REFERENCES[1] Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest,
and Westley Weimer, “A Systematic Study of AutomatedProgram Repair: Fixing 55 out of 105 Bugs for $8 Each,”
ICSE, pp.3-13 (2012)[2] Jaechang Nam, Song Wang, Yuan Xi, and Lin Tan, “A bug
ﬁnder reﬁned by a large set of open-source projects,” Information and Software Technology, Vol.112, pp.164–175(2019)[3] Sunghun Kim, Thomas Zimmermann, Kai Pan, and E.
James Jr. Whitehead, “Automatic Identiﬁcation of BugIntroducing Changes,” 21st IEEE/ACM International Con-7

International Workshop on Informatics ( IWIN 2022 )ference on Automated Software Engineering (ASE’06),
pp.81-90 (2006)[4] Sokratis Tsakiltsidis, Andriy Miranskyy, and Elie Mazzawi, “Towards Automated Performance Bug Identiﬁcation in Python,” 2016 IEEE International Symposium on
Software Reliability Engineering Workshops (ISSREW),
pp.132-139 (2016)[5] Keigo Matsushita, Masaki Matsumoto, Kazuhiko Ohno,
Takahiro Sasaki, Toshio Kondo, and Hiroshi Nakashima,“A Debugging Method Based on Comparison of Execution
Trace,” Symposium on Advanced Computing Systems andInfrastructures (SACSIS), Vol.2011, pp.152-159 (2011) (inJapanese)[6] Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund,“On the accuracy of spectrum-based fault localization,”
Testing: Academic and Industrial Conference Practice andResearch Techniques, pp.89-98 (2007)[7] W. Eric Wong, Vidroha Debroy, Yihao Li, and Ruizhi
Gao, “The DStar method for eﬀective software fault localization,” IEEE Transactions on Reliability, Vol.63, No.1,
pp.290-308 (2014)[8] Takuma Ikeda, Kozo Okano, Shinpei Ogata, and Shin
Nakajima, “Localization of Fault Methods and Ablationof Execution Traces Using A Machine Learning Model toClassify Test Results,” IEICE Technical Report, Vol.121,
No.416, pp.13-18 (2022) (in Japanese)[9] Zhuo Zhang, Yan Lei, Xiaoguang Mao, Meng Yan, Ling
Xu, and Xiaohong Zhang, “A study of eﬀectiveness of deeplearning in locating real faults,” Information and Software
Technology, Vol.131, No.1, pp.1–16 (2021)[10] W. Eric Wong, Vidroha Debroy, Richard Golden, Xiaofeng Xu, and Bhavani Thuraisingham, “Eﬀective Software Fault Localization Using an RBF Neural Network,”
in IEEE Transactions on Reliability, Vol.61, No.1, pp.149169 (2012)[11] “scikit-learn: machine learning in Python ― scikit-learn
1.1.1 documentation,” https://scikit-learn.org/stable/ (referred April 27, 2022)[12] René Just, Darioush Jalali, and Michael D. Ernst, “Defects4J: A database of existing faults to enable controlled
testing studies for java programs,” in Proceedings of the2014 International Symposium on Software Testing andAnalysis, pp.437–440 (2014)8

International Workshop on Informatics ( IWIN 2022 )        OLO  
'%$#$!!%%$! #!)$%!(%! $-+$())*)%$%"%,&$$)'$)%$"%""%$%"%,&$.     / &"$)!$-+) &
(% $&)*$!$-+) &  3F743>>E BABG>3D;FK :3E 9D36G3>>K 675D73E76
AH7DF:7B3EF K73DE ):;ED7E73D5:3;?EFA4G;>63EKEF7?
F:3F 3>>AIE B>3K7DE 53@ 7@<AK B>3K;@9 I;F: 43E;5 DG>7E
I;F:AGF D787D77E EG5: 3E 4AI>;@9  753GE7 A8 ;F F:7 B>3K7D
53@ 5D73F7 @7I B>3KEFK>7E 43E76 A@ ;F  K D76G5;@9 F:7
6;88;5G>FK8AD479;@@7DEI753@;@5D73E7F:75AH7D397A8F:7
3F743>>  @ F:;E D7E73D5: I7 E7F F:7 53?7D3 A@ F:7 93F7E
F:3FI7D74G;>FI;F:'3EB47DDK&;3@6F:73?7D3?A6G>7FA
5A>>75F 63F3 I:7@ F:7 43>> B3EE7E F:DAG9: F:7 93F7 F:7@
FD3@E87D;FFAF:7&;@D73>F;?7I;F:'73>);?7(FD73?;@9&DAFA5A>  @ F:7 & ;F GE7E F:7 .%"% ?A67> FA 67F75F F:743>>3@6F:7@FD35=;F ):7EKEF7?IAD=76 I7>>;@3I;D7>7EE
@7FIAD=I;F:F:7>3F7@5KA8FD3@E87DD;@963F347FI77@F:7E7
67H;57E47;@9G@67D ?E ):;EEKEF7?I3E34>7FA5:75=
I:7F:7DF:743>>B3EEF:DAG9:F:793F7AD@AF3@653>5G>3F76
F:7E5AD78AD735:43>>B3EE;@93GFA?3F;53>>K AI7H7DI;F:
F:7EB776A8F:743>>;@5D73E;@9F:7355GD35K;EE;9@;8;53@F>K
D76G576  3F743>>'73>F;?7%4<75F7F75F;A@')(&A)

I;F: KAG@9 B7AB>7 4753GE7 A8 F:7 :73F: BDA4>7? 3@6
FD3@EBADF BDA4>7?E  ):7D78AD7 ;F ;E ;?BADF3@F FA >AA= 8AD
?AD7KAG@9B7AB>7FA<A;@F:;EEBADF @366;F;A@FAF:7397BDA4>7?F:75A?B>7J;FKA8F:7DG>7E
3@6F:793?7B>3KA893F743>>;E3>EA3@A4EF35>78ADF:7@7I
479;@@7D EE:AI@;@;9GD7 3F743>>;E393?7B>3K76
4KFIAF73?E735:I;F:8;H7B>3K7DEF:3F>736;@9F73?GE7E
D7643>>EA66@G?47DE3@6F:78A>>AI;@9F73?GE7EI:;F7
43>>E 7H7@ @G?47DE  ):7 B>3K7D EFD;=7E F:7 43>> 8DA? F:7
EF3DF3D73FAI3D6EF:78;DEF93F7FAB3EEF:743>>F:DAG9:;F3E
E:AI@;@;9GD7 



 

   


3F743>>;E3EBADF3@6EFD3F79K93?7F:3FI3E;@H7@F764K
;<; (GLG=; A8 #7?GDA FAI@ A==3;6A ;@  3@6
9D36G3>>K4753?7BABG>3DF:DAG9:AGF 3B3@8DA?F:7  E
FAF:7 E3@6F:7@8A>>AI;@9F:3FBABG>3D;FK3F743>>4793@
EBD736;@9 8DA? 3B3@ FA F:7 IAD>6 EF3DF;@9 8DA? E;3
5AG@FD;7E FA F:7 *@;F76 (F3F7E 6GD;@9 F:7  E 0 1 
AI7H7D ;@ F:7 >3EF  K73DE F:7 @G?47D A8 3F743>>PE
B>3K7DE;ED3B;6>K675D73E;@98DA?  B7AB>7;@ FA
  B7AB>7 0 1  ):7 D73EA@ 8AD F:;E 675D73E7 ;E F:7
3H7D397397A8B>3K7DE3@6F:76;88;5G>F;7EA83F743>>DG>7E
01 
 A8 F:7    ?7?47DE A8 F:7 3B3@ 3F743>>767D3F;A@ 3D7 AH7D   K73DE A>6  ):7D78AD7 3F743>> ;E
I7>>=@AI@ 3E 3 EBADF 8AD 7>67D B7AB>7  ):7D7 3D7 ?3@K
BDA4>7?E I;F: F:;E :;9: 3H7D397 397  %@7 A8 F:7? ;E F:7
B:KE;53>5A@6;F;A@ @7H7DKEBADFF:7B>3K7D:3ED7CG;D76FA47;@F:747EFB:KE;53>5A@6;F;A@6GD;@9F:7?3F5:FA4734>7
FA B>3K I:A>7 F:7 ?3F5: 3@6 ?3=7 F:7 D;9:F <G69?7@F 3F
7H7DK ?A?7@F  ):;E ;E :3D6 8AD 7>67D B7AB>7 4753GE7 F:7:G?3@4A6K@3FGD3>>K;E;@5D73E;@9>KI73=7D3@6F:74D3;@PE
D735F;A@ EB776 ;E E>AI7D F:3@ I:7@ F:7K I7D7 KAG@9 ):7D78AD7F:7BAEE;4;>;FKF:3FF:7K53@@AFB3DF;5;B3F7;@F:7
?3F5:AD?GEFF3=734D73=;@F:7?;66>7;E7JFD7?7>K:;9: 8F:7D7;E3B7DEA@I:A53@PFB3DF;5;B3F7F:7K3D78AD576FA
EFAB F:7 ?3F5:  #AD7AH7D I;F: A>6 397 F:7 F;?7 F:7K
47>A@9A@F:7F73?I;>>678;@;F7>K47E:ADF7DI:7@5A?B3D76

     


;9GD7 ):7DG>7A83F743>>):7B>3K7D3;?EFAB3EE:;E :7D43>>F:DAG9:F:7E75A@693F7
3@6F:7F:;D693F7;@F:3FAD67D %@7BA;@F8AD7H7DKF;?7F:7
43>> B3EE7E F:DAG9: F:7 8;DEF E75A@6 3@6 F:;D6 93F7E I:;>7
FIABA;@FE3D73I3D676I:7@F:743>>:;FEF:79A3> ):7F73?
F:7D78AD7 E5AD7E  BA;@FE ;8 3>> 8;H7 F73? ?7?47DE 8;@;E: 753GE7 A8 F:;E F:7 B>3K7D :3E FA F:;@= EFD3F79;53>>K 34AGF
:AI FA 97F F:7;D F73?PE 43>> FA :;FE F:7 9A3> 3E ?G5: 3E
BAEE;4>7 3@6 =;5= F:7 ABBAE;@9 F73?PE 43>> AGF  @ AD67D FA
5A?7 GB I;F: F:7 ?AEF ABF;?3> EFD3F79K B>3K7DE ?GEF
G@67DEF3@6F:7DG>7EA8F:793?7F:7;DF73?PEEFD7@9F:E3@6
I73=@7EE7E3EI7>>3EBD76;5FF:7ABBA@7@FPEB>3KEFK>7 ):;E
I;>> F3=7 3 >AF A8 F;?7 ;@ AD67D FA 6A I:;5: F:7 479;@@7DE
6A@PF :3H7  ,7 @776 FA 8;@6 3 I3K FA =77B F:7? B>3K;@9
7JB7D;7@576 7@AG9: FA 47 34>7 FA 5A?7 GB I;F: EFD3F79;7E
43E76A@F:3F DA? F:7 34AH7 ;@ F:;E D7E73D5: I7 7JB7D;?7@F76 I;F:
3BB>K;@9 A) 67H;57E 3@6 A4<75F 67F75F;A@ F75:@A>A9K FA
3GFA?3F7E5AD;@9;@3F743>> ):7@43E76A@F:;EEKEF7?I7
EG997EF@7I3@6E;?B>793?7B>3KF:3F;E?AD78D;7@6>K8AD
479;@@7DEFA4734>7FAEF3DFB>3K;@9F:;EEBADF 

9International Workshop on Informatics ( IWIN 2022 )   

?A6G>7 I:;5: ;E D7EBA@E;4>7 8AD 5A>>75F;@9 63F3 6GD;@9 F:7
?3F5: 3@6 E7@6;@9 ;F FA F:7 >A53> &  ):7 E75A@6 B3DF ;E 3
63F3 BDA57EE;@9 ?35:;@7 F:3F D757;H7E 63F3 8DA? D7?AF7
53?7D3 ?A6G>7E F:7@ BDA57EE7E F:7? GE;@9 A4<75F
D75A9@;F;A@ 3>9AD;F:?E ;@ AD67D FA 53>5G>3F7 F:7 E5AD7 GDD7@F>K;@AD67DFAB>3K3F743>>I7@776F:D77D787D77E
I:A3D73D787D773@3EE;EF3@FD787D773@63D75AD67D ,;F:
F:;EEKEF7?I753@3GFA?3F73@653>5G>3F7F:7E5AD7A8735:
B>3K7DI;F:AGFD787D77EI;F:>7EE@G?47DA8D7CG;D76B7AB>7
;@ F:7 ?3F5: 3@6 93;@ ?AD7  3H3;>34>7 B>3K7DE F:7D74K
?;@;?;L;@9F:7BDA4>7?A8>35=A8B>3K7DE 

 ''#"#"! & #!""
!&" $'@ F:7 8;7>6 A8 3@;?3> 47:3H;AD 3@6 75A>A9K EFG6K
A4E7DH;@9 F:7 ?AH7?7@F A8 3@;?3>E ;E F:7 ?AEF ABF;?3>
D7E73D5: ?7F:A6  AI7H7D ;8 F:7 D7E73D5:7D 6A7E F:;E
:;E :7DE7>8F:7D7I;>>47?3@KBDA4>7?EEG5:3E;FD7CG;D7E3>AF A8 F;?7 3@6 788ADF AD 63?397 FA F:7 3@;?3>PE @3FGD3>
:34;F3F 6GD;@9 A4E7DH3F;A@  )A EA>H7 F:;E BDA4>7? I7 53@GE7 >3E7D E7@EADE 3@6 3 :K4D;6 53?7D3 FA FD35= 3@;?3>E
3GFA?3F;53>>K 01  @ F:;E D7E73D5: :7 GE76 F:7 :K4D;6
53?7D3 FA 53F5: 3@K ?AH7?7@F A8 3@;?3>E 3@6 D76D3I F:7
?AH7?7@F A8 F:7 A4E7DH76 3@;?3>  #AD7AH7D ;@ AD67D FA
3HA;6 @A;E7 3@6 7DDAD ;@ ;?397 D75A9@;F;A@ :7 GE76 67BF:
;?397E 3@6 8AD79DAG@6 7JFD35F;A@ GE;@9 F:7 '$(
3>9AD;F:?  E 3 D7EG>F A8 F:;E 7JB7D;?7@F ;F I3E 34>7 FA
D75A9@;L7 F:7 ?AH7?7@F A8 F:7 F3D97F 3@;?3> 3@6 FD35= ;F5ADD75F>K     
@F:;EE75F;A@I7I;>>7JB>3;@F:753?7D3?A6G>7I:;5:;E
B3DFA85A>>75F;@963F36GD;@9F:7?3F5:3@6E7@6;@9;FFAF:7
>A53>?35:;@7;@AD67DFA53>5G>3F7F:7E5AD7A8735:B>3K7D ):;E53?7D3?A6G>7;E?367GBA8F:D77B3DFE  '3EB47DDK&;/7DA,  3?7D3
 3FF7DKKGE;@9'3EB47DDK&;/7DA,;F3>>AIEF:;E53?7D3?A6G>7
FA4734>7FAGE7F:7')(&BDAFA5A>I:;5:;EGE76FAFD3@E87D
H;67AAD;?39763F3;@D73>F;?7I;F:3>3F7@5KA8>7EEF:3@
 ?E  ,7 GE7 F:7 D7?AH34>7 53?7D3 ;@ F:;E ?A6G>7
4753GE7 F:7D7 I3E @AF 3 '3EB47DDK &; /7DA , F:3F :3E 3
4G;>F;@53?7D33@64KGE;@9F:7D7?AH34>753?7D3I7I;>>
:3H7 ?AD7 ABF;A@E FA F7EF I;F: ?3@K 53?7D3E F:7D74K
675;6;@9 I:;5: A@7E 3D7 F:7 ?AEF ABF;?3> 8AD F:7 EKEF7? GDD7@F>KI73D7GE;@93?:43FF7DK;@F:;E?A6G>7 
AI7H7D I7 D73>;L76 F:3F ;F ;E FAA ?G5: 8AD F:;E E?3>>
?A6G>7 3@6 3D7 B>3@@;@9 FA 5D73F7 3 @7I 67E;9@ A8 F:7
?A6G>7 I;F: 3 E?3>>  ?: 43FF7DK ;@ AD67D FA ?3=7 ;F
475A?773E;7DFAE7F;@F:793F7  ($$#%')&'!#%' '&" #&&"# &@AD67DFA;?BDAH7F:7788;5;7@5KA8F:7B>3K7DPEFD3;@;@9
F75:@A>A9;7E EG5: 3E A) @F7D@7F A8 ):;@9E 3@6 DF;8;5;3> @F7>>;97@57 3D7 ;@5D73E;@9>K 47;@9 3BB>;76
I;67>K;@F:7FD3;@;@9BDA57EEA8F:7B>3K7DE @7J3?B>7A8
F:7? ;E F:7 H;67A 8776435= EKEF7?01 I:;5: BDAH;67E
EGBBADF8AD5A@6G5F;@9EB75;3>;L76FD3;@;@93@6?3;@F3;@;@9
6;E5;B>;@7  @ F:;E D7E73D5: :7 E7F GB H;67A 53?7D3E ;@
E7H7D3>B>357E;@AD67DFAD75AD6F:7BDA57EEA8FD3;@;@9F:7
5A35:I;>>4734>7FA5:75=F:7?AH7?7@FA83F:>7F7E8DA?
H3D;AGE 3@9>7E  AI7H7D I:;>7 D75AD6;@9 F:7 H;67A F:7
53?7D3 I;>> 47 @AF 34>7 FA F3=7 B;5FGD7E  )A EA>H7 F:;E
BDA4>7? :7 675;676 FA 4G;>6 3 ?A4;>7 H;67A EKEF7? GE;@9& &F75:@A>A9K    
 




  










   ;9GD7 GFA?3F76(5AD;@9(KEF7?
):;E EKEF7? 53>5G>3F7E F:7 E5AD7 A8 735: B>3K7D 43E76 A@;?397 63F3 F:3F I7D7 5A>>75F76 8DA? F:7 53?7D3 ?A6G>7
34AH7 F:7 93F7 3@6 E7@6E F:7 D7EBA@E7 435= FA F:7 53?7D3
?A6G>738F7D8;@;E:;@9 53>5G>3F;@9 F:7E5AD7 ):;EI3E?367
GB A8  B3DFE 3E F:7 ;9GD7  ):7 8;DEF B3DF ;E 3 53?7D3

;9GD73?7D3#A6G>7

      ):;E ;E F:7 ?AEF ;?BADF3@F B3DF A8 F:7 3GFA?3F76 E5AD;@9
EKEF7? I7 GE7 7FEA@ -3H;7D $- 3E F:7 >A53> ?35:;@7

10

International Workshop on Informatics ( IWIN 2022 )I:;5: ;E GE76 FA D757;H7 F:7 ;?397 63F3 E7@F 8DA? D7?AF76
53?7D3?A6G>7EI;F:')(&BDAFA5A>3@6BDA57EEF:7?I;F:
F:7 .%"% A4<75F 67F75F;A@ 3>9AD;F:? ;@ AD67D FA 53>5G>3F7
F:7E5AD7A8F:7B>3K7D .%"%.AG%@>K"A5=%@57;EA@7
A8 F:7 ?AEF BABG>3D A4<75F 67F75F;A@ 3>9AD;F:?E D757@F>K ;FGE7E @7GD3> @7FIAD=E FA BDAH;67 D73>F;?7 A4<75F 67F75F;A@
I;F: 7JFD7?7>K 83EF 3@6 355GD3F7  @ ?& ?73EGD76 3F 
%* .%"%H ;E 7CG3> FA A53> "AEE 4GF 83EF7D 34AGF 
F;?7E  @ F:;E D7E73D5: I7 675;67 GE;@9 F:;E 3>9AD;F:? FA
67F75F 43>>E 8DA? F:7 5A>>75F76 63F3 4753GE7 ;F 3>>AIE D73>
F;?7 67F75F;A@ >;9:F :;9: BD75;E;A@ 3@6 73EK FA FD3;@ F:7
?A67>  @ F:;E B3DF I7 GE7 BKF:A@ I:;5: ;E 3 :;9:>7H7>
;@F7DBD7F76 3@6 97@7D3>BGDBAE7 BDA9D3??;@9 >3@9G397 FA
IAD=I;F:F:7.%"%H?A67> .%"%HI3E5D73F7643E76
A@   >3@9G397 I:;5: I3E F:7 >3@9G397 BKF:A@ I3E4G;>F A@  753GE7 A8 ;F F:7 EKEF7? 53@ IAD= E?AAF:>K
I;F:AGF 5A@8>;5FE EG5: 3E EK@F3J AD 5A?B;>7D 7DDADE  @ F:;E
>A53> ?35:;@7 F:7 ;?397 63F3 I;F:F:7 ')(& BDAFA5A> I3E5AH7DF FA 3 6;?7@E;A@ H75FAD 01010'1 ;@ F:7 D3@97  M
3@6F:7@GE;@9F:7.%"%H?A67>FA5:75=7H7DKB;J7>
FA 67F75F F:7 4AG@6;@9 4AJ A8 F:7 A4<75F ;@ F:7 ;?397  ,7GE7 F:7 67F75F76 4AG@6;@9 4AJ FA 53>5G>3F7 F:7 5AAD6;@3F7E
A8F:743>>FA5:75=I:7F:7DF:743>>B3EE7EF:DAG9:F:793F7
AD@AF .%"%?A67>@AFA@>KD7EBA@6EFAF:74AG@6;@94AJ
4GF3>EAD7EBA@6EFAF:75>3EEA8 67F75F76A4<75FE ,;F:F:7
D7EBA@E75>3EE;F;E34>7FA=@AII:;5:43>>B3EE76F:DAG9:
F:793F73@653>5G>3F7F:7E5AD7A8F:7B>3K7D )AEG??3D;L7
F:;E B3DF F:7 53>5G>3F76 E5AD7 ?35:;@7 IAD= 3E E:AI@ ;@;9GD7 





 






















;9GD7):7355GD35K3@6F:7EB776A8F:743>>
@ F:;E 7JB7D;?7@F I7 F7EF F:7 355GD35K I;F: AGD 5GEFA?
.%"% ?A67> I:;5: I3E FD3;@76 I;F:   B;5FGD7E  
B;5FGD7E 8AD 735: 5>3EE  ;DEF I7 F7EF I;F: D3@6A?  
F7EF;@9 B;5FGD7E I:;5: EB>;F 8DA? F:7 63F3E7F  ):7 D7EG>F A8
F:;E8;DEFF7EFE:AIEF:3FI;F:F:7;?397F:7355GD35K8AD3>>
5>3EE7E;E34AH7 3E;9GD7 ):7:;9:7EF355GD35K;EF:7
43>>  3@6 F:7 >AI7EF ;E 43>>  DA? F:;E 8;DEF 7JB7D;?7@F
D7EG>F I7 G@67DEF3@6 F:3F 8AD F:7 43>>E I;F: F:7 E3?7
5:3D35F7D;EF;5E F:7 355GD35K A8 67F75F;A@ ;E >AI7D F:3@ F:7
AF:7DE AD7J3?B>7F:743>> 3@6F:743>> F:3F:3E3B3DF
E:AI@G?47DN OEAI:7@F:7;?397A8F:743>>A@>KE:AIE
N OF:7?A67>53@PF67F75F;FI7>> @F:7E75A@67JB7D;?7@FI7F7EFF:7EKEF7?I;F:F:7D3I
H;67A F:3F I3E FD3@E87DD76 8DA? F:7 53?7D3 ?A6G>7 A@ F:7
93F7  ,7 F7EF I;F:  ?7F:A6E ;@ F:;E 7JB7D;?7@F  ):7 8;DEF
A@7;EI;F:F:7E>AIEB776A8F:743>>47;@947>AI ?E E ):7 E75A@6 A@7 ;E I;F: 3 :;9: EB776 F:3F ;E 34AH7  ?E E 
@;9GD7FE:AIEGEF:3FI;F:F:7:;9:EB776A8F:743>>
F:7355GD35KA867F75F;A@;E675D73E;@9 ):7D73EA@;EI:7@
F:7 43>> B3EE76 F:7 93F7 3F :;9: EB776 F:7 53?7D3 :36
6;88;5G>FK 53BFGD;@9 5>73D>K F:7 E:3B7 A8 F:7 43>> I:7@ F:7
6;887D7@5747FI77@F:7;@BGF;?3973@6F:7>73D@76;?397;E
>3D97;F;E6;88;5G>F8ADF:7EKEF7?FAD75A9@;L7;F5ADD75F>K 

!#

!    












 

  






  "   
"







 "   ;9GD7):7355GD35KA8735:43>>;9GD7(5AD73>5G>3F7&DA57EE    

  ):763F3E7FGE76;@F:;E7JB7D;?7@F;E?3674KAGD?7?47DE
I:;5: ;@5>G67E
 E3?B>7 B;5FGD7E F:3F I7D7 >347>76 ;@.%"%8AD?3F3EE:AI@;@F:7F34>7  
)34>7 ):7.%"%8AD?3FGE768AD63F3E7F
>3EE
-.,;6F:
7;9:F
M 
M 
M 
M


     
DA? F:7 7JB7D;?7@F3> D7EG>F F:7 EKEF7? E:AIE F:3F ;F ;E
34>7FAIAD=I;F:D73>93?7E;@3F743>>;@3E;?B>7?A67
>;=753>5G>3F;@9E5AD7I:7@7H7DF:743>>B3EE76F:DAG9:F:7

  

11

International Workshop on Informatics ( IWIN 2022 )93F7 ,7EG997EFFIA93?7B>3KE43E76A@F:;EEKEF7?I:;5:
;E73E;7D8AD479;@@7DEI:A6A@AF=@AI3>AF34AGFF:7DG>7E   ):7B>3K7DEI;>>EF3DFF:7;DFGD@355AD6;@9FAF:7@G?47D
A843>>E8DA?F:7EF3DF3D73 AD735:F;?7F:743>>B3EE7E
F:DAG9:F:793F7F:7B>3K7DI;>>93;@A@7BA;@F ,:A:3E
9AF BA;@FE8;DEFI;>>47F:7I;@@7D   ):7B>3K7DI;>>476;H;676;@FAFIAF73?E43E76A@F:7
@G?47D A8 F:7 43>>  7H7@A66  %@7 F73? I;>> 47 F:7
3FF35=7D F:7 AF:7D I;>> 47 F:7 6787@67D  AD A@7 FGD@
F:7B>3K7D53@EFD;=7F:743>>A@57 ,;F:7H7DK43>>F:3F
B3EE7EF:DAG9:F:793F7:;E :7DF73?I;>>97FA@7BA;@F ):7 6787@67D F73? I;>> F:;@= 34AGF :AI FA 4>A5= F:7
3FF35=7D B3EE F:7 43>> F:DAG9: F:7 93F7  8 F:7 3FF35=7D
F73?97FE BA;@FF:7KI;>>47F:7I;@@7D AI7H7D;8
F:7 3FF35=7D F73? :3E@PF 9AFF7@ F:7  BA;@FE 38F7D 
FGD@EF:76787@67DEPF73?I;>>47F:7I;@@7D :FFBE III <EF397 <EF 9A <B 3DF;5>7 <E;E7   2 
 23DF;5>7 5:3D <3  
@ F:;E D7E73D5: I7 4G;>6 3 EKEF7? @3?76 GFA?3F76
(5AD;@9EKEF7?GE;@9A)3@6;@AD67DFAEA>H7F:7:;9:
3H7D397 397 3@6 >35= A8 B>3K7D BDA4>7? ;@ 3F743>>  ):7
D7EG>F A8 F:7 7JB7D;?7@F E:AIE F:3F F:7 EKEF7? ;E 34>7 FA
IAD=A@E;?B>793?7B>3K 




 

):;ED7E73D5:;EEGBBADF764K%)AG@63F;A@ 



 



0 1 ,AD>6 3F743>> *@;A@ ;EFADK A8 3F743>>
:FFBE 93F743>> AD <B I9G B>3K B>3K2  :F?>
0 1 ):7 675>;@7 ;@ 3F743>>PE BABG>3F;A@ 3@6 D73EA@:FFBE III @7IE
BAEFE7H7@ 5A? 3D5:;H7E
  2  :F?>)"557EE 
01.3L3=;,3F3DG   (FG6KA@(BADFE5F;H;F;7EA8
F:7 >67D>K 75>;@;@9 3F743>> &ABG>3F;A@ @@G3>
#77F;@9A8F:7 3B3@7E7(A5;7FKA8&:KE;53>6G53F;A@  +A>G?7  D6    B   &G4>;E:76  
%@>;@7(($ 
:FFBE 6A; AD9
 <EB75A@8  2 :FFBE III <EF397 <EF 9A <B 3DF;5>7 <EB75A@8  
2  23DF;5>7 5:3D <3
01 )EG@7 $363 #5!;@@7EE 3?7E .3EGA $393; E3?;
)EG5:;63 !A<; #3EG63 7F75F;A@ A8 @;?3>EA?4;@;@9 ' 3@6 7BF: ?397E &DA5776;@9E A8
F:7GLLK(KEF7?(K?BAE;G?A8F:7 3B3@@F7>>;97@F
@8AD?3F;A@GLLK(A5;7FK
+A>   F:GLLK(KEF7?(K?BAE;G?(7EE;A@#  B '7>73E7
63F7
 :FFBE 6A; AD9 8EE  2
:FFBE III <EF397 <EF 9A <B 3DF;5>7 8EE 2 23DF;
5>7 5:3D <3
01 )A?A=3LG #;GD3 (GBBADF (KEF7? 8AD F:>7F7E 3@6
A35:7E *E;@9 +;67A3E76 AGD@3> A8 6G53F;A@3>
(KEF7?E3@6@8AD?3F;A@(KEF7?E
+A> $A 
 B     &G4>;E:76  %@>;@7 (($  &D;@F
(($  :FFBE 6A; AD9  <E;E7  12

International Workshop on Informatics ( IWIN 2022 )Automatic Derivation of a Transition Model from a Japanese Requirement
Speciﬁcation under a Restricted Grammar
Koki Shimokawa† , Hiroya Ii† , Maiko Onishi‡ , Shinpei Ogata*, and Kozo Okano*†‡

Graduate School of Science and Technology, Shinshu University, Japan
Graduate School of Humanities and Sciences, Ochanomizu University, Japan*Faculty of Engineering, Shinshu University, Japan* {ogata,okano}@cs.shinshu-u.ac.jp

Abstract - In this paper, we propose a method for extracting state transitions, which are necessary elements
for model checking, from requirement speciﬁcations written in a natural language. Requirement speciﬁcations areoften written in a natural language and may contain ambiguous expressions. Model checking is a method to checkfor such ambiguous expressions using mathematical techniques. The proposed method extracts state transitions
from requirement speciﬁcations based on restricted grammars. We also present the results of an experiment toevaluate the eﬀectiveness of the proposed method.

describes the experimental evaluation methodology. Section 7 describes the results. Section 8 discusses on the
results. Finally, Section 9 provides conclusion and future
outlook.

2
2.1

Morphological Analysis

Morphological analysis is the process of dividing input
sentences into words and recognizing their parts of speechand conjugations. In this paper, our research group use an
existing tool, kuromoji [3]. It is an open-source Japanesemorphological analyzer developed in Java, and is capableof word segmentation, part-of-speech tagging.

Keywords: Natural Language Processing, Requirement
Speciﬁcation, Model Checking.

1

PRELIMINARIESINTRODUCTION
2.2

Requirement Speciﬁcation documents used in softwaredevelopment are often written in natural languages [1].
Natural languages may contain ambiguous expressions in
the text. This leads to the discovery of bugs in the testingprocess, which results in signiﬁcant rework costs. Modelchecking approaches are regarded as a useful method toverify whether the speciﬁcation is good requirement or not.
Model checking, however, requires specialized knowledge
such as temporal logic. Thus, we believe that a method toautomatically derives models from requirement speciﬁcations in natural languages will help designers and developers to share models without contradiction.
In our previous study, we proposed a method for automatically extracting state variables and state transitions
using morphological and syntactic analysis for the purpose of automatically creating state transition diagrams[2]. The results show that the recalls of state variables andvariable values are 1.00 for 36 sentences, and the precisionsof state variables and variable values are 0.93 and 0.83.
However, the signiﬁcant increase in parsing rules resulted
in a major problem of extracting unnecessary elements.
In this article, we consider that restricting the input syntax to some extent can solve the above-mentioned. Section
2 explains the terminology. Then Section 3 describes previous studies. Section 4 discusses related research. Section 5 gives the proposed sentence structure. Section 6

XPath

XPath stands for XML Path Language [4]. It is a concise
syntax for extracting speciﬁc parts of a document in XML
format. Using this tool, you can follow the elements you
want to extract from a tree of sentences. This tool allows
you to follow the elements you want to extract from a treestructured document like a ﬁle path.

2.3

NuSMV

NuSMV is a software tool for the formal veriﬁcationof ﬁnite state systems. Other model checking tools besides NuSMV include SPIN and UPPAAL. It has beendeveloped jointly by FBK-IRST and by Carnegie Mellon
University [5]. NuSMV allows to check ﬁnite state systems against speciﬁcations in the temporal logics CTL
and LTL. The input language of NuSMV is designed toallow the description of ﬁnite state systems that range
from completely synchronous to completely asynchronous.
The NuSMV language provides for modular hierarchicaldescriptions and for the deﬁnition of reusable components.
The basic purpose of the NuSMV language is to describe
the transition relation of a ﬁnite Kripke structure [6]. This
provides a great deal of ﬂexibility, however, it can introduce danger of inconsistency for non expert users at thesame time.

13

International Workshop on Informatics ( IWIN 2022 )3

PREVIOUS RESEARCH

fact type(associated an object type) map attribute name.
As a result, class diagrams could be generated with highaccuracy.
There are also studies that convert from natural language processing to SBVR representation [14]. It has beenreported that when a meta-model is created and requirement speciﬁcations in natural languages are converted intothat meta-model, the formal logic of SBVR allows requirements to be easily processed by machines and element extraction to be performed with high accuracy. Similar study
is being conducted on Japanese speciﬁcations. Study is
being conducted to develop a support tool that automatically generates UML text from Japanese requirement speciﬁcations using natural language processing [15]. The tool
automatically generates object diagrams, class diagrams,
sequence diagrams, use case diagrams, and state machine
diagrams. As a result, they were able to successfully generate UML with high accuracy. However, it has the disadvantage of being cumbersome when the proposed method
does not match the method that the designer wants toadopt.
Another study proposed a method to automatically generate a temporal logic formula that can be mapped to an
inspection formula as an intermediate process of extracting
state-transition relations from conditional sentences thatappear in speciﬁcations [16].

In order to extract the elements of the transition model,
we decomposed sentences into words and tagged them withparts of speech using the morphological analyzer kuromoji.
Further, our research group created a syntactic tree using
the CYK method [7]. Then, the proposed method classiﬁed the syntax with XPath and extracted the speciﬁed
elements. As a result, the recalls of state variables andvariable values were 1.00, and the precisions were 0.93 and0.83, showing high accuracy for the 36 input sentences described in the description of the Electric Pot (7th edition)[8]. However, there is a wide variety of word orders in the
input sentences, such as inverted sentences and emphatic
sentences. Our research group considered that creating
rules for syntax tree creation and element extraction inconsideration of such cases would require an unrealistic
number of rules. In fact, the number of syntax tree creation rules for 36 input sentences was 163, and the numberof element extraction rules increased as well, resulting in
the extraction of extra elements and reducing the precision.

4

RELATED WORK

There are a number of studies that convert speciﬁcations written in natural language to UML (Uniﬁed Modeling Language) or Model Checking.
In order to extract the components of the model, there
is an attempt to use a dependency parsing instead of manu5 PROPOSED METHOD
ally creating phrase structure rules [9]. They used “ChaboCha”
The ultimate goal of this research is to automatically[10] for the dependency parsing. “ChaboCha” uses macreate state transition models from requirement speciﬁcachine learning called Support Vector Machine, so there istions written in natural languages and model checking. In
no need to set rules for parsing [11]. As a result, all stateorder to improve the aforementioned problem, we devised avariables and variable values could be extracted as in themethod to extract elements by deﬁning a certain restricted
extraction with clause structure rules, but extra elements
stylistic description in our previous study [17]. Then, awere extracted.
GUI element support tool was devised to present the exIn addition, there are also studies that have attempted totraction results to the speciﬁcation writer in real time at
generate class diagrams by extracting elements from Enthe stage of creation, and to encourage the writer to corglish sentence speciﬁcation based on the SBVR (Semanrect the sentence. In this study, we decided to devise antics of Business Vocabulary and Business Rules) [12] staninput sentence that can be easily converted to a state trandard established by OMG (Object Management Group)
sition model description automatically, referring to previ[13]. In order to map SBVR to the UML class model, it
ous studies. The following sections describe the elements
is a matter of extracting SBVR vocabularies from SBVR
to be extracted, the input sentence creation method, therules and translating SBVR vocabularies into the basic elproposed grammatical structure, and the outline of the elements of the UML class model (class names, relations,ement support tool.
etc.). The SBVR vocabulary consists of deﬁnitions of all
speciﬁc terms and concepts used in the course of business. They classify SBVR vocabularies as common nouns,
5.1 Extracted Elements
proper nouns, verb concepts, quantiﬁcation(“a”, “an” andpreﬁxed with “s”), partitive fact types (“is part of,” “inThe model used in this study is NuSMV. The descripcluded in,” or “belong-to”), etc. using SBVR rules andtion of NuSMV consists of sections deﬁning the set of state
natural language processing. And then, they use their own
variables and variable values, the initial state, and the desrules to map from the classiﬁed SBVR vocabulary to thetination state [17].And we believe that if we can extract allelements needed for the class diagram. For example, athe necessary elements for the destination state, we can excommon noun maps to a class name. Characters or unary
tract all of them. Because there is always a set of variable14

International Workshop on Informatics ( IWIN 2022 )values for the state variable in the description of the destination state.
Therefore, we decided to extract the ﬁve elements ofstate variable, pre-transition variable value, transition condition, user behavior, and post-transition variable value inorder to also enable automatic creation of state transition
diagrams from requirement speciﬁcations written in natural language. The description part of the destination state,
which represents the ﬁve elements of state variables (red),pre-transition variable values (blue), transition conditions(green), user behavior (purple), and post-transition variable values (yellow), is shown in Figure 1.
In addition, state variable, pre-transition variable value,user behavior, and post-transition variable value are essential to execute NuSMV.

Figure 2: Example of referenced speciﬁcation description(the contents of line 65). When performing model checking, the contents described in the 65th to 69th lines of Listing 1 were substituted into the “check” variable of the LTL
expression on the 73rd line. As a result, the model checker
returned true on all checks. Therefore, it was conﬁrmedthat no counterexamples for the ﬁve LTL properties(lines
65 to 69) were found.
In this way, it was found that the state transition model
corresponding to water boiling function of the Electric pot(7th edition) satisﬁed properties from 1 to 5(lines 65 to 69)
We believe that the model created can be used for safety
veriﬁcation. In addition, it is also possible to intentionally output a counterexample in model checking to check
whether a speciﬁc operation is possible. For example, “Ifthe ‘condition are met when the lid was open, is model
checked. As a result, as shown in Fig. 3, desired counterexample was output.
Thus, it was possible to conﬁrm that there was no problem with the model. Since it was conﬁrmed that there wasno problem in the model, We consider a statement that
includes the elements needed to create a model.

Figure 1: Example of model checking description

5.2

Sentence Preparation Procedure

We created speciﬁcation sentences with restricted grammars in order to make the sentences easy to be automatically extracted into the NuSMV model. The requirements
speciﬁcation document used as reference in this study is theElectric Pot (7th edition) [8]. In creating the restriction
grammar, we ﬁrst wrote the NuSMV model description of
the Electric Pot (7th edition). The reason for creating a
NuSMV description is to ﬁnd transition state statementsthat are omitted in the requirements speciﬁcation description so that NuSMV can be executed. For example, “Whenthe lid is open and the user closes it, the lid closes” is notincluded in the speciﬁcation, so it was complemented. Two
students with knowledge of NuSMV read the speciﬁcation
description (in Figure 2) and created a NuSMV model.
Next, they checked whether the created model satisfy
timed properties expressed in LTL. The letter “Ubetween variables “condition
and “checkin the LTLSPEC description stands for “until. The whole expression“condition until check stands for “condition” holds until“check” holds. An example of condition phrase in a natuvariable is “theral language corresponding “condition
user is pressing or releasing the hot water button while
the water is being supplied. Another example of those is“the water temperature is low when the user presses theboil button.
On the other hand, an example of condition phrase in anatural language corresponding “checkvariable is “thesafety lock does not turn on and the water does not boil

Listing 1: Example of NuSMV model
1
2
3
4

5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

15

MODULE main
VAR
Controller : {supply, supply_stop,boil, can_use, cann_use, idle};
Human : {kp_s, p_b, wait, i_o, u_o,c_r, o_r,r_s, on_slb, off_slb,
adj_wl};Water_level : {proper,not_proper};Rid : {open, close};
Temp : {low, high};
S_L:{on,off};
DEFINEotherwise :=TRUE;ASSIGNinit(Controller):= can_use;
init(Human) := wait;
init(Water_level) :=not_proper;
init(Rid):= open;
init(Temp):= low;init(S_L):= on;
next(Controller) := case
Human = kp_s & Temp =International Workshop on Informatics ( IWIN 2022 )high & Controller!= cann_use &Rid = close & S_L= off : supply;
Controller = supply &Human = r_s :supply_stop;
Human = p_b &Water_level =proper & Temp =low & Rid = close: boil;Human = i_o &Controller =cann_use: can_use;
Human = u_o &Controller =can_use : cann_use

22

23

24

25

28

29

34
35

36

37
38
39
40
41

47

61
62
63
64
65
66

next(Human) := case
Rid = open &Water_level=
not_proper :adj_wl;Human = kp_s : {kp_s,
r_s};Controller != cann_use& Controller !=supply: {kp_s,
wait,p_b, u_o, c_r, o_r, on_slb,
off_slb} ;Controller = cann_use: {i_o, o_r, c_r

67
68

69
70

71
72
73

};

DEFINE

check1 := S_L = on &Controller = supply;
check2 := Rid_sensor = close& Controller = supply;
check3 := Rid = open &Controller = boil;
check4 := Water_level =not_proper & Controller =supply;check5 := Temp = low &Controller = supply;
condition1 := (Controller =supply) -> ((Human = kp_s)|(Human = r_s));condition2 := (Human = p_b)->(Temp = low);
condition := condition1 &condition2;
LTLSPEC !(condition U check2)

otherwise : Human;next(Water_level) := case
Human = adj_wl & Rid =open &
Water_level =not_proper: proper

;

42
43
44
45
46

next(S_L) := case
Human = on_slb & S_L =off & Rid =close & Controller!= supply : on;
Human = off_slb & S_L= on & Rid =close & Controller!= supply : off;
Human = r_s & S_L =off
: on;otherwise : S_L;
esac;

60

;
esac;

esac;

next(Temp) := caseController = boil :high;Human = wait : low;otherwise : Temp;
esac;

59

Controller = can_use :idle;
Controller = boil :can_use;Human = c_r &Controller = idle
& Water_level =proper : boil;
otherwise : Controller

27

esac;

53
54
55
56
57
58

;

26

30
31
32
33

48
49
50
51
52

Controller !=supply : open;otherwise : Rid;esac;otherwise :Water_level;next(Rid) := case
Human = c_r & Rid =open: close;
Human = o_r & Rid =close & Controller!= boil &Figure 3: Counterexamples for boiling transitions

16

International Workshop on Informatics ( IWIN 2022 )5.3

Proposed Sentence Structure

5.3.4

As shown in Section 5.1, the proposed sentence structure
consists of ﬁve elements. Figure 4 to Figure 7 show thesentence structures and examples of sentences using the
Element Extraction Support Tool.
5.3.1

Proposed Sentence (5)As shown in Figure 7, the ﬁfth is a sentence structure
consisting of a pre-transition variable value and a posttransition variable. In the requirements document used asa reference [8], there are diﬀerent variable names with thesame meaning.
Therefore, we adopted this sentence structure in order torecognize that the variable names have the same meaning
by describing this sentence.

Proposed Sentence (1)The ﬁrst is a sentence type that describes all elementsof the state variable, pre-transition variable value, transition condition, user behavior, and post-transition variablevalue, as shown in Figure 4.

Figure 7: Proposed sentence structure (5)Figure 4: Proposed sentence structure (1)

5.3.2

5.4

With reference to the tools devised in related studies[17], we developed a tool. Figure 8 shows the overview of
the tool. By analyzing input sentences written in an Excelﬁle, the contents of the xml ﬁle describing the extracted
elements are output.
Figure 9 shows the window view of the tool. The left sideshows the input sentences of the excel ﬁle and the right sideshows the extraction results. The tool currently provides
four functions: saving of input text, deleting input text,
generating a new input text, and analyzing the results of
the input text.
We will discuss each of these functions in detail in thefollowing sections.

Proposed Sentence (2)The second is a sentence structure in which elements otherthan transition conditions are arranged as shown in Figure
5.

Figure 5: Proposed sentence structure (2)5.3.3

Element Extraction Support Tool

Proposed Sentences (3) and (4)The third and the fourth are sentence structures (1) and(2) without the state variable. The reason for adopting
this sentences is that the state variable is not implicitly
described in the speciﬁcation. Therefore, we will consider
providing the ability to supplement the state variable withthe proposed tool (in Fig. 6).
Figure 8: Tool Overview

5.4.1

Editing and Saving

The area for describing requirement speciﬁcations is a text
ﬁeld that can be freely edited on the tool. You can freely
edit the text on the tool.

Figure 6: Proposed sentence structures (3) and (4)17International Workshop on Informatics ( IWIN 2022 )Figure 9: Developped Element Support Tools
When we press the “save” button at the bottom of thewindow on the left side of Figure 9 , the text ﬁle that
executes the program saves the input sentences.
5.4.2

Figure 11: Generating Function

DeletionIf there is the unwanted sentence as shown in Figure
10, press the “Delete” button to the left of the component
name. Then, we can delete the sentence by pressing our“Update” or “Delete” button.

Figure 10: Delete Function

5.4.3

Generation

Pressing the “New” button at the bottom of the left window in Figure 9, the window shown in the red frame in
Figure 11 appears. We then select the name of the new
input statement component we are adding. If component
name is not in the selection, we can type it directly in the
text ﬁeld.
5.4.4

Analysing Results

When we press the “Update” button at the bottom of thewindow, we will extract the elements from the input sentences and display them as shown in Figure 12.

6

Figure 12: Analysis Result Display

EVALUATION EXPERIENCE

The descriptions in Chapters 2 and 3 of the Electric Pot(7th edition) [8] are translated into the NuSMV model.

18

International Workshop on Informatics ( IWIN 2022 )structure can signiﬁcantly reduce the number of parse tree
creation rules.

We then converted again into the proposed sentence structure in Japanese. As shown in Figure 13, 31 sentences areused as input sentences. The following research question isformulated to verify the validity of the proposed sentence
structure.
The RQ1 is veriﬁed by calculating the reproduction recall and the precision.
RQ1: Can the proposed sentence structure be used to extract elements accurately?
RQ2: Can the number of parse tree creation rules be reduced compared to the 163 rules in the previous study [2]?

Table 1: Recall Precision
Elements
State variable
Pre-transition variable value
Transition condition
User behavior
Post-transition variable value

8

Recall
1.00
1.00
1.00
1.00
1.00

Precision
1.00
1.00
1.00
1.00
1.00

CONSIDERATION

We found that the proposed sentence structure greatly
improves the extraction accuracy. The proposed method ofsentence structure construction can be applied to a variety
of sentence styles with a small number of syntax tree creation rules. In the evaluation experiment, element extraction was performed on sentences created using a sentence
structure that is easy to extract elements from the NuSMV
state transition model. Therefore, it is easy to imagine thatthe reproduction recall and precision are both 1.00 for all
the elements.
In this paper, we created input sentences based on thestate transition model after creating the state transition
model. However, we believe that other requirement speciﬁcation descriptions with the proposed sentence structure
can be used to automatically create descriptions of transition relationships sentences.
However, limiting the number of input sentences has created the problem of whether users can write verbose sentences or not.

9

The goal of this paper is to create state transition models directly from requirement speciﬁcations. Therefore, weclassiﬁed the elements necessary to compose a descriptionof the transition relationship into ﬁve categories. Theseﬁve elements are the elements that compose the description of the transition relationship between the variable values possessed by the state variables. We also proposed a
ﬁve sentences pattern to facilitate the extraction of theseelements. Moreover, the descriptions of the transition relations in the state transition model were converted into
Japanese sentences with the proposed sentence structure.
Then, we conducted an extraction experiment of the converted Japanese sentences using natural language processing. The result was that the recalls and precisions for all
elements were 1.00. In addition, we also developed an element extraction support tool.

Figure 13: Input Sentences

7

CONCLUSION

EXPERIMENTAL RESULTS

Table 1 summarizes the results of the expreriments. It is
judged that the elements are consistent even when particles
or verbs are attached to the elements to be extracted, since
the element speciﬁcation is done in phrases. RQ1 and RQ2
can be answered as follows.
RQ1: The recall and precision for all elements are 1.00, as
shown in Table 3. Therefore, it is possible to extract theelements accurately.
RQ2: The number of parse tree creation rules used in theevaluation experiment was 56, and the proposed sentence

19International Workshop on Informatics ( IWIN 2022 )There are two things we plan to do in the future. The
ﬁrst is to deﬁne the sentence structure of the statement,
including the description of the initial state and state rules.
Therefore, we redeﬁne the sentence structure of initial andtransition states so that they do not cover each other.
Then, we believe that all the elements necessary to create a state transition model for NuSMV can be extracted
from natural languages by modifying the syntax tree creation rules and element extraction rules.
The second is to implement a guidance function in thetool that would allow the proposed sentence structure to
be described. Current element support tools cannot extract elements without understanding the sentence structure proposed by the writer. Therefore, we believe that
we can improve the current tool by adding a function that
prompts the user to rewrite the sentence if the writer does
not write the sentence with the proposed sentence structures.

Acknowledgement
This research is being partially conducted as Grant-inAid for Scientiﬁc Research A (18H04094) and C (21K11826).

REFERENCES[1] Meyer Bertrand: “On formalism in speciﬁcations,”
IEEE Software, vol. 2, no. 1, pp. 6–26 (1985).
[2] Hiroya Ii, Kozo Okano, and Shinpei Ogata: “Improving Accuracy of Automatic Derivation of State Variables and Transitions from a Japanese Requirements
Speciﬁcation,” Joint Conference on Knowledge-Based
Software Engineering Springer, pp. 20-34 (2020).
[3] atilika:“Open source Japanese morphological analysis engine developed in Java,”
https://www.atilika.com/ja/kuromoji/(2022.6.7
access)(in Japanese).
[4] MDNWeb
Docs,“XPath,”https://developer.mozilla.org/ja/docs/Web/XPath/(2022.1.20 access).
[5] NuSMV,“An
overview
of
NuSMV,”
https://nusmv.fbk.eu/NuSMV/ (2022.6.7 access).
[6] Edmund M. Clarke, Orna Grumberg, Daniel Kroening, Doron Peled, and Helmut Veith: “Model Checking, second edition (Cyber Physical Systems Series),”
The MIT press (2018).
[7] Tadao Kasami: “An eﬃcient recognition and syntaxanalysis algorithm for context-free languages,” Coordinated Science Laboratory, pp. 1-46 (1965).
[8] SESSAME,“Instructional
Materials
for
Embedded
Systems
GOMA1015
Type
Requirement
Speciﬁcation,”
https://www.sessame.jp/workinggroup/WorkingGroup2/(2021.3.29 access) (in Japanese).

20[9] Masanosuke Ohto, Hiroya Ii, Kozo Okano, and Shinpei Ogata: “Proposal of Extracting State Variables and Values from Requirement Speciﬁcations inJapanese by using Dependency Analysis,” 25th International Conference on Knowledge Based and Intelligent information and Engineering Systems, pp. 16491657 (2021).
[10] CaboCha: “Yet Another Japanese Dependency Structure Analyzer,” https://taku910.github.io/cabocha/(2022.10.8 access)(in Japanese).
[11] Taku Kudo, Yuji Matsumoto: “Japanese Parsing
by Stepwise Application of Chunking,” IPSJ, Vol43,
No.6, pp. 1834-1842 (2002)(in Japanese).
[12] Standard Development Organization:“SBVR,”
https://www.omg.org/spec/SBVR/1.5/About-SBVR(2022.1.24 access).
[13] Hina Afree and Imran S. Bajwa: “Generating UML
class models from SBVR software requirements speciﬁcations,” 23rd Benelux Conference on Artiﬁcial Intelligence, pp. 23-32 (2011).
[14] Imran S. Bajwa, Mark G. Lee, and Behzad Bordbar:“SBVR business rules generation from natural language speciﬁcation,” AAAI 2011 Spring Symposium
Series (2011).
[15] Masakazu Takahashi, Satoru Takahashi and
Yoshikatsu Fujita : “A Proposal of Adequateand Eﬃcient Designing of UML Documents for
Beginners,” Knowledge-Based Intelligent Informationand Engineering Systems, KES 2007, vol. 4693, pp.
pp 1331–1338 (2007).
[16] Maiko Onishi, Hiroya Ii, Shinpei Ogata, and Kozo
Okano: “Extraction method for transition relations
from conditional statements in natural language requirements speciﬁcations,” Technical Report of IEICE, vol. 2021-SE-208, no. 6, pp. 1-6 (2021) (inJapanese).
[17] Hiroya Ii, Masanoseke Ohto, Hitoshi Kiryu, Shinpei Ogata, and Kozo Okano: “Proposal of a formof requirement speciﬁcations for automatic transition
model derivation and the derivation method,” Technical Report of IEICE, vol. 121, no. 94 , pp. 13-18(2021) (in Japanese).
[18] Alessandro Cimatti, Edmund Clarke, Fausto
Giunchiglia, and Roveri Marco: “NuSMV : a
new symbolic model checker,” International Journal
on Software Tools for Technology Transfer2, pp.
410-425 (2000).

International Workshop on Informatics ( IWIN 2022 )'HVLJQRI$LU)ORZ&KDQQHOIRUWKH'URSOHW6SUD\7\SH2OIDFWRU\'LVSOD\Yohei Seta*,***, Mitsunori Makino**, Yuichi Bannai***, and Motofumi Hattori****

Graduate School of Science and Engineering, Chuo University, Japan**
Faculty of Science and Engineering, Chuo University, Japan***
Department of Information Media, Kanagawa Institute of Technology, Japan{a15.s845, makino.fme7}@g.chuo-u.ac.jp{bannai, hattori}@ic.kanagawa-it.ac.jp$EVWUDFW -To date, olfactory displays based on various
principles have been developed to present controllable scents.
In this study, to design a novel piezoelectric olfactory
display, we conducted experiments to determine theappropriate channel height at which the bottom of the air flow
channel does not come in contact with the sprayed purified
water; in addition, we obtained the appropriate shape of theconnection between the liquid fragrance atomizing
mechanism and the air flow channel.
Accordingly, it was inferred that when the distance from thepiezoelectric element surface to the bottom of the air flow
channel was ≥ 70 mm, contact between the atomized purified
water and the bottom of the channel could not be confirmed.
It was also deduced that the retention of atomized droplets
can be reduced by connecting the airflow channel with arectangular hole that matches the size of the bottom of theliquid fragrance tank and reducing the distance from theupper surface of the airflow channel to the bottom of theliquid fragrance tank to ≤ 1 mm.

In principle, the atomized fragrance components may besubject to turbulent diffusion in the process of beingtransported by air or retention in airflow channel depressions.
Seta et al. conducted experiments to validate the performanceof Nakamura et al.'s piezoelectric olfactory display [3].
Accordingly, it was verified that the atomized liquidfragrance contacted the bottom of the airflow channel before
being fully vaporized, and then remained as droplets.
These problems may result in the continuous presentationof scents independent of the intention of the system, which
may severely affect the accuracy of the results obtained from
olfactory characterization studies via this system.
In this study, to design novel piezoelectric olfactory
displays, experiments were conducted to determine the air
flow channel height at which the bottom of the air flow
channel does not come into contact with the atomized liquidfragrance. The condition of the atomizing droplets wascaptured using a digital camera from the side, and thepresence or absence of droplet adhesion was verified from thevideo. We checked for atomization by altering the height of
the floor and confirmed the minimum distance at which the
droplets do not adhere to the floor.
A simulation also measures the amount of the atomized
substance exiting from the air flow channel with a delay fromthe system's intended scent presentation time due toentrapment of the atomized substance in the vortex generated
by the depression in the connection between the droplet
atomization mechanism and air flow channel.
Accordingly, it was deduced that when the distance fromthe surface of the piezoelectric element to the bottom of thechannel was ≥ 70 mm, contact between the atomized purified
water and the bottom of the air flow channel was not observed.
Instead of the conventional method of connecting the liquidfragrance tank and the air flow channel with a circular hole, a
rectangular connection was made to match the size of thebottom of the liquid fragrance tank, and the distance between
the upper surface of the air flow channel and the bottom of
the liquid fragrance tank was reduced to ≤ 1 mm, thereby
reducing the retention of the atomized substance due to vortex
in the depression that connects the liquid fragrance tank andthe air flow channel.
By designing the air flow channel of a piezoelectricolfactory display that considers these two results, thepresentation of scents is expected to be possible in a timecloser to the intention of the system than with conventional
devices.

.H\ZRUGV: olfactory display, pulse ejection, droplet
atomization

 ,1752'8&7,21
Various forms of olfactory display have been developed forsystem-controllable scent presentation devices. Bannai et al.
developed droplet atomization-type olfactory displays thatcan be controlled in 100-ms increments, to primarily
investigate human responses to olfactory stimuli [1].
A unique feature of droplet atomization olfactory displays
is the pulsed presentation of the scents. Pulsed presentationof scents is a method of presenting scents by intermittently
atomizing liquid fragrances for a short period (0.1–0.3 s),
thereby reducing the amount of atomization and the effects ofadaptation and lingering scents.
Nakamura et al. developed an olfactory display using a
piezoelectric element as a droplet atomization mechanism, toimprove maintainability and simplify the structure of droplet
atomization-type olfactory displays. [2]. Nakamura et al.'s
piezoelectric olfactory display vibrates its piezoelectricelement at 100 kHz when a voltage is applied, and the liquidin contact with the surface of the piezoelectric element can beatomized as tiny droplets from the opposite side through thecentral group of micropores. This piezoelectric element is
attached to the bottom of the liquid fragrance tank and cancontrol the atomization of the liquid fragrance in 1-ms
increments.

23

International Workshop on Informatics ( IWIN 2022 )detection threshold does not change during inhalation at the
beginning, in the middle, or at the end of inhalation, and thatthe human olfactory system can perceive scents from the
beginning to two thirds of the inhalation time within two
seconds, the average inhalation time in a person at rest [5].
Furthermore, a study was conducted to investigate theinteraction between olfaction and vection, a self-moving
sensation derived from vision, using an inkjet olfactory
display. The obtained results strongly suggested that vection
stimulation influences olfaction [6].
However, the problem with this olfactory display is that its
structure is complicated and it is not easy to clean. It has beenverified that almost all the atomized fragrance droplets
vaporize, but the fragrance adheres to the connection between
the ink cartridge and the airflow channel, as well as to theinjection port. Furthermore, the principle of droplet ejection
is based on the thermal method, in which bubbles are
generated in a liquid by heating, and then ejected by thepressure of the bubbles, which limits the use of heat-sensitivefragrances.

 0(7+2'2/2*< 2OIDFWRU\'LVSOD\Among the devices that present stimuli to the five humansenses, those that present visual stimuli are called displays,
while those that present olfactory stimuli are called olfactory
displays. Odors are sensed when odorants with a molecularweight of less than approximately 300 are captured from theair by the olfactory receptor cells in the olfactory epitheliumof the nasal cavity. Olfactory displays are artificial scentpresentation systems that provide a mechanism that delivers
fragrance components to the user's nose by volatilizing
liquids such as essential oils and synthetic fragrances, which
are sources of the scent.
In olfactory displays, the basic functional components
include a volatilization method of liquid fragrance, aconcentration control method of volatilized fragrance
combined with a mixing method of multiple fragrances, anda time control method to determine the presentation time andwhen the odor is delivered to the user; in addition, various
methods have been proposed [4].

 3LH]RHOHFWULF2OIDFWRU\'LVSOD\Nakamura et al. developed a piezoelectric olfactory displaythat adopts a piezoelectric element as a droplet atomization
mechanism to circumvent the disadvantages of inkjet
olfactory displays, such as thermal injection, difficulty indownsizing the atomization mechanism, and complicatedcleaning and maintenance [2].

 ,QNMHW7\SH2OIDFWRU\'LVSOD\Bannai et al. developed an inkjet olfactory display using an
inkjet printer cartridge as the droplet atomization device.
The inkjet olfactory display employs the nozzle of acommercially available inkjet printer to eject tiny droplets ofa solution containing a certain concentration of liquidfragrance, instead of ink. A fan installed at the rear of thedevice blows air to vaporize microdroplets in the air anddirects fragrance components to the user's nose, thereby
presenting a scent to the user. The microdroplet ejection canbe controlled in 0.1-s increments, and 255 ejection heads cansimultaneously eject droplets. The number of microdroplet
ejections can be modified in the application, and the ejection
volume can be altered in 256 steps from 0 to 255. Eachejection head can eject approximately 4.7 pL droplets atintervals of 1–150 ejections per 0.1 s. Hence, the amount andtiming of the liquid fragrance ejection can be fine-tuned, thus
enabling the pulse presentation of scents by repeatedly
presenting minute droplets of liquid fragrance forsignificantly short durations.
It has been reported that this method reduces the challenges
posed by olfactory adaptation, in which the user becomes lesssensitive to smells when continuously exposed to odor stimuli,
including the problem of lingering smells, in which smells
remain in space and mix when the olfactory display iscontinuously used, compared to olfactory displays that
continuously present scents [5]. By presenting scents tosubjects with varying intensities and high accuracy, it waspossible to determine the detection threshold, which is theminimum intensity at which a person perceives the presenceof a certain scent.
Experiments were conducted using this olfactory display tomeasure human olfactory characteristics, and revealed that it
is possible to perceive scents without adaptation for morethan one minute by presenting pulses of scents, that the

Figure 1: Cross-sectional image of piezoelectricolfactory display (side).
Figure 1 illustrates the operation of the piezoelectricolfactory display. The device uses a fan to blow the liquidfragrance atomized by a piezoelectric device built into aliquid fragrance tank, to present a scent to the user. The
piezoelectric device has a diameter of approximately 1 cm
and 100 micropores of 9-μm diameters drilled in its center.
Although the liquid fragrance in the liquid fragrance tankdoes not pass through the micropores owing to surface
tension when no voltage is applied, it is atomized through themicropores by vibration when voltage is applied. The
piezoelectric device oscillated once every 10 μs, and each
oscillation could control the atomization at approximately
167 pL. A fan was employed to blow air over the atomized
liquid fragrance, promote the volatilization of the liquidfragrance, and present the scent to the user.
A 3-mL capacity liquid fragrance tank was placed above thepiezoelectric device, and the fragrance in the tank wascontinuously ejected from the piezoelectric device. The tank,piezoelectric device, and piezoelectric device holder at thebottom of the tank were removable for easy cleaning andmaintenance.

24International Workshop on Informatics ( IWIN 2022 )An experiment to measure the mass of purified water
atomized 10 times for 10 s verified that the average injection
mass was 0.17 g (standard deviation: 0.0077) and remained
constant regardless of the number of atomization. Different
types of fragrances have different viscosities, surface tensions,and other characteristics; hence, the fragrance solutions to beatomized in olfactory displays must be adjusted for theircomposition.

However, it is known that the cavity shape facing the flow
triggers unstable flow owing to the contact between fluidswith significantly different velocities, thereby resulting invortex-like flow [7].
When such vortexes are generated, fragrance components
may be retained because of the vortexes. This may cause thefragrance components that remain in the cavities to graduallyleak out and reach the system user, even after the end of
atomization. This problem has not been considered inconventional piezoelectric olfactory displays.
To accurately determine a person’s scent detection
thresholds using a droplet atomizing olfactory display, it isnecessary to switch between odorless and attached states witha high degree of accuracy [8]. If the presentation state by thesystem is odorless but the subject is actually presented with ascent because of the above problem, the olfactorycharacteristics of the subject may not be measured correctly.

 /LPLWDWLRQVRIWKHSLH]RHOHFWULFGHYLFH
ROIDFWRU\GLVSOD\Nakamura et al.'s olfactory display air flow channel has across section 20-mm high and 60-mm wide, which follows
the structure of the inkjet olfactory display. To install theliquid fragrance tank, a 10-mm rise section was provided at
the top of the air flow channel, and a cylindrical hole with adiameter and height of 15 mm and 15 mm, respectively, wasmade in the rise. The surface of the liquid fragrance tank was
attached to the top of the raised section, and atomization was
directed vertically toward the bottom of the air flow channel.
To prevent the piezoelectric device from being directly
touched by hands, a 5-mm high dimple exists from the bottomof the liquid fragrance tank, as a frame to protect the surfaceof the piezoelectric device, and a space for fixing thepiezoelectric device exists above the frame. Therefore, thedistance from the atomization position of the piezoelectric
device to the bottom of the channel is 40 mm.
This is equivalent to the atomized height of an inkjet
olfactory display, and previous experiments using inkjet
olfactory displays reported that the atomized substance wassufficiently vaporized in the air flow channel [1]. Therefore,
it was assumed that the atomized substance, used in theexperiment, in the piezoelectric olfactory display vaporizes
immediately after ejection and does not remain as droplets on
the wall surface.
However, the atomization mechanisms of the inkjet andpiezoelectric device olfactory displays are different,
including the composition of the liquid fragrance solution;
hence, the vaporization of the atomized substance is also
expected to be different.
Seta et al. created an acrylic plate air flow channel with aheight of 20 mm and a width of 60 mm to confirm theconditions in the air flow channel immediately after theatomization of droplets in the piezoelectric device olfactory
display of Nakamura et al. They connected a liquid fragrance
tank of Nakamura et al.'s method to the acrylic plate air flow
channel and observed the inside of the air flow channel bycapturing video images of the inside of the air flow channel
at the time of atomization. The results showed that theatomized substance adhered to the bottom surface of theconventional structure as droplets at the time of atomizationof a banana liquid fragrance and purified water andevaporated later than after the system had finished atomizingthem.
A cylindrical cavity exists at the connection between theliquid fragrance tank and the air flow channel to maintainmore distance between the bottom of the air flow channel andpiezoelectric device.

 0(7+2'
In this study, we experimentally determined the height of
air flow channel where no droplet adhesion is observed on thebottom of the air flow channel when liquid fragrance isinjected in a piezoelectric olfactory display.
Simulations were also performed to confirm vortex
generation and retention of the atomized substance in the
depression at the connection between the fragrance atomizing
mechanism and air flow channel and to determine the shapeof the connection to reduce vortex generation and retention.

 ([SHULPHQW&RQILUPDWLRQRIWKH
UHTXLUHGDLUIORZFKDQQHOERWWRPGLVWDQFH
GXULQJSXULILHGZDWHUDWRPL]DWLRQ
To determine the minimum distance at which a droplet isatomized by the piezoelectric device without contacting thebottom of the airflow channel, an experimental airflow
channel was constructed using acrylic plates. Figure 2
presents the configuration and dimensions of theexperimental airflow channel.
In the experiment, the distance from the piezoelectric
device to the bottom of the airflow channel was varied, and avideo of the droplet atomization was captured. The video
images of the atomized droplets were observed and checked
for the presence or absence of droplet adhesion to the bottomof the air flow channel. The threshold value of the bottom
height for the presence/absence of deposits at the bottom of
the airflow channel is defined as the minimum distancewithout contact with the airflow channel bottom whenatomizing droplets.
Previous experiments have demonstrated that liquidfragrances containing ethanol or other solvents vaporize
faster than purified water, and by determining the thresholdof the bottom height relative to purified water, it is possible
to consider liquid fragrances that may be used in the future[3].  Therefore, only purified water was utilized in thisexperiment.
The experimental airflow channel employs a bottom surface
whose height can be modified as desired, which is referred toas a movable floor in this study. The movable floor was an

25

International Workshop on Informatics ( IWIN 2022 )reproducibility of the experimental results, it was necessary
to stabilize the airflow in the experimental airflow channel.
In this experimental air flow channel, a grid-like structureused for rectification in the experimental wind tunnel was
installed in the cross section of the air flow channel that flows
into the air flow channel from the fan mounting area. Previous
experiments have demonstrated that the grid shape iseffective in stabilizing the airflow in the air flow channels of
piezoelectric olfactory displays when the pitch width, plate
thickness, and open area ration are 5 mm, 1 mm, and 60%,
respectively [9]; in addition, the same parameters will be
adopted in this study. The fan was mounted in the directionof intake from the top of the experimental apparatus andperpendicular to the cross-section of the air flow channel, toreduce the influence of the low-speed portion at the center ofthe fan.
During the experiment, an anemometer was placed at thecenter of the cross section, which is at the end of theexperimental air flow channel, and the power of the fan was
adjusted to achieve the target wind velocity while checking
the wind velocity. The experiment was allowed only between
wind speeds of 1.8 ±0.03 m/s.

3UHOLPLQDU\H[SHULPHQW&RQILUPRIZLQGYHORFLW\GLVWULEXWLRQLQWKH
H[SHULPHQWDODLUIORZFKDQQHOTable 1: Average wind velocity of channel cross section
Rectification grid

Horizontal
VerticalHorizontal
Vertical(h = 100 mm) (h = 100 mm) (h = 200 mm) (h = 200 mm)
Average wind velocity (m/s)
Standard deviation

Figure 2: Structure of experimental channel (unit: mm).
acrylic plate with the same dimensions as the bottom of the
airflow channel, wrapped with vinyl tape at both ends, andfixed in position by friction with the acrylic plate on thechannel side. Therefore, the movable floor can be easily
removed when changing the bottom height.
To maintain constant conditions on the movable floor
surface during continuous injection, each time the bottom
height was changed, the surface was wiped with a paper rag
moistened with purified water, and then the surface was
wiped dry again with a dry paper rag. When fixing the
movable floor, the height of the installed surface was checkedwith a digital caliper, and a level was used to check the levelat the part protruding from the channel before conducting theexperiment.
The air velocity in the air flow channel during the droplet
atomization experiment was 1.8 m/s, which has been adopted
in conventional experiments on inkjet olfactory displays [1].
A PWM-controllable PC fan was utilized to provide aconstant airflow in the experimental airflow channel. The
selection of the fan to be used is based on the maximum
airflow rate, and the fan is required to sufficiently satisfy thetarget air velocity over the entire experimental air flow
channel cross section. Blowing air by rotating fans generates
a low velocity near the center of the air velocity distribution,
asymmetrical bias, and turbulence. To improve the

2.61
0.21

2.70
0.12

2.70
0.06

3.5
3.0

150

Wind velocity (m/s)

Measurement position (mm)

200

2.80
0.21

100
50

0

2.5
2.0
1.5

1.0
0.5
0.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

Wind velocity (m/s)

0

5

10

15

20

25

30

Measurement position (mm)Figure 3: Wind speed distribution (h = 200 mm).
3.5
3.0
75

Wind velocity (m/s)

Measurement position (mm)

100

50
25

0

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

Wind velocity (m/s)

2.5
2.0

1.5
1.0
0.5

0.0

0

5

10

15

20

25

30

Measurement position (mm)Figure 4: Wind speed distribution (h = 100 mm).
To verify the maximum air velocity and the rectification
effect in the experimental air flow channel and to check

26

International Workshop on Informatics ( IWIN 2022 )whether the use of the movable floor caused a significant
disturbance of the air velocity distribution in the air flow
channel cross section, the air velocity distribution wasmeasured at the vertical and horizontal centers of the crosssection at the airflow channel end when the experimental air
flow channel height was h = 200 mm and h = 100 mm withthe movable floor at the output under the maximum airflow
rate of the fan.
In both cases, the vertical wind velocity distribution wasalmost flat, with an average of approximately 2.7 m/s, and thehorizontal wind velocity distribution was almost symmetrical,
with the fastest near the center and gradually slowing down
as it approached the wall (Figures 3 and 4). Table 1 presents
the average wind velocity and standard deviation of the crosssection of the experimental air flow channel.
The obtained results indicate that the experimental air flow
channel can be utilized for this experiment because the target
air velocity of 1.8 m/s is sufficiently satisfied, and negligible
turbulence exists in the air velocity distribution owing to theasymmetric flow caused by the fan rotation and the influenceof the movable floor.

calculated via fluid simulation, using the particle method topredict the internal state of the atomized substance.
The particle method is a computational technique that ismainly adopted for fluid simulation, in which a continuum is
discretely represented as a set of movable particles. The
governing equations determine the forces acting on eachparticle, and the fluid phenomena can be calculated bycontinuously updating the coordinates of the particles overtime. Compared with the lattice method, which describes
fluid motion by dividing the space with a fixed grid andcalculating the fluid inflow and outflow at the grid points, theparticle method has the advantages of not requiring the space
to be cut with a mesh and being able to easily track individual
fluid elements as they flow in. In addition, by explicitly
obtaining all terms of the governing equations, parallel
computation is facilitated and benefits from accelerated
calculation using GPUs. Therefore, the MPS explicit method,a particle method, is employed in this study to calculate theair flow in an air flow channel [10].
The air in the air flow channel and the atomized substances
treated in this study were sufficiently slower than the speedof sound to be treated as incompressible fluids. The NavierStokes equation in Equation (1) and the continuity equationin Equation (2) are adopted as the governing equations forincompressible fluids.

 6LPXODWLRQ&RQILUPDWLRQRIYRUWH[JHQHUDWLRQDQGUHWHQWLRQRIDWRPL]HGVXEVWDQFHRZLQJWRWKHVKDSHRIWKHIUDJUDQFHWDQNFRQQHFWLRQ

஽࢛
஽௧
஽ఘ
஽௧

ଵ

ൌ  െ ‫ ࡼ׏‬൅ ‫׏ݒ‬ଶ ࢛ ൅ ࡲ

(1)

ൌͲ

(2)

ఘ

where u, ȡ, P, v, F, and t denote the flow velocity, massdensity, pressure, kinematic viscosity, external force, andtime, respectively. Equation (2) refers to the law ofconservation of mass in dynamics, and it was adopted to
calculate the particle dynamics.
In this study, calculations were performed using air and avaporized banana liquid fragrance as particles in the MPS
method with two parameters. The air particle parameters referto the JIS standard pneumatic fluid power - standard reference
atmosphere, with a humidity of 65%, a temperature of 20 °C,
and a pressure of 1 atm as the setting environment [11]. Fromthe setting environment, a density of 1.198 kg/m3 andkinematic viscosity of 1.505E-5 m2/s were adopted asparameters for air particles [12].
The parameters of the particles of the atomized substanceare strictly a phenomenon of gradual vaporization in air fromthe liquid state immediately after atomization. However, forcomputational simplicity, the present simulation was basedon the assumption that the particles enter the air in a gas state
immediately after atomization. The substance to be atomized
was a mixture of 5 % isoamyl acetate, 4.25 % ethanol, 0.94 %glycerin, 0.09 % propylene glycol, and 89.72 % water asbanana fragrance, based on the composition ratio of a banana
liquid fragrance solution that can be adopted in thepiezoelectric olfactory display by Nakamura and colleagues[2]. Based on the molecular weight of each component andthe mixing ratio, we obtained an average molecular weight of
25.57 for the banana liquid fragrance mixture. Assuming that
all the liquid mixtures with the average molecular weight

Figure 5: Simulation environment (unit: mm).
To obtain an appropriate shape for the connection between
the liquid fragrance tank and the airflow channel, we checked
the occurrence of vortexes inside it and compared the effectof its shape on the retention of the atomized substance. The
shape of the connection between the air flow channel andliquid fragrance tank cannot be directly observed because thematerial of the liquid fragrance tank does not allow light topenetrate under the current experimental environment. If theatomized substance remains inside, the droplets will
evaporate inside it and will be invisible. Therefore, in thisstudy, the motion of the atomized substance was numerically27

International Workshop on Informatics ( IWIN 2022 )were vaporized, the density of the particles to be sprayed was
1.0631 kg/m3 according to Avogadro's law, which states that
1 mol of gas has a volume of 24.04 L at a temperature of 20 °C
and 1 atm. The kinematic viscosity was assumed to be
1.505E-5 m2/s, the same as that of air particles. Figure 5
illustrates the computational environment.

during the 1-s period from 0.883 s to 1.883 s after the end ofspraying, when the banana liquid fragrance particles were
fully discharged from the channel without retention, wascompared with the volume of the target particles that flowed
late due to retention. Based on the above results, the mostsuitable shape for olfactory display among the shapes to becompared was considered.

 5(68/7
 ([SHULPHQW&RQILUPDWLRQRIWKH
UHTXLUHGDLUIORZFKDQQHOERWWRPGLVWDQFH
DWWKHWLPHRISXULILHGZDWHUDWRPL]DWLRQ
Figure 6: Comparison cavity shape (unit: mm).

Liquid fragrance tank

The computational air flow channel has an inflow of air
particles in an air flow channel cross section with a width of
30 mm and a height of 70 mm, and the initial velocity was set
to 1.8 m/s. The computational air flow channel length was
170 mm, the center of the inflow point of the particles of theatomized banana liquid fragrance was 50 mm from the inflow
point of air, and the cross section of the computational air
flow channel at 100 mm from the inflow point was the surface
from which the flow rate of the particles of atomized banana
liquid fragrance was measured.
Figure 6 presents the shape of the cavity at the connection
between the liquid fragrance tank and the airflow channel inthe comparison model. The area of the trapezoidal rotator at
the top of the cavity shape in Figure 6 represents the dimple
in the frame protecting the piezoelectric device of the liquidfragrance tank, which was 5 mm high and fixed in all cases,
with its top surface in contact with the atomizing surface ofthe piezoelectric device. The variables include the height ofthe cylindrical areas that are in contact with the dimple of theliquid fragrance tank and the height of the rectangular areathat matches the size of the bottom of the liquid fragrance
tank, Thereby creating a cavity model with heights of 15 mm,
5 mm, and 1 mm for the cylindrical and rectangular areas,
respectively. The cavity shapes of the six patterns werecompared.
The initial inflow velocity of banana liquid fragrance
particles is calculated from the distance traveled by thepiezoelectric device during the atomization of purified water,which was recorded as a 240-fps movie, at the frame when
the atomization is first observed, and in this calculation, it was
11.322 m/s. Banana liquid fragrance particles flowed in a
vertical direction from the center of the depression in thefragrance tank toward the bottom of the flow path. The
banana liquid fragrance particles inflow was 0.3 s from 0.5 s,
to 0.8 s from the start of the simulation, which represents asingle pulse of injection. The calculation covered a period of
up to 2.0 s, which is more than 1.0 s after the end of the inflow
of banana liquid fragrance particles.
The interior of the cavity shape was observed as the banana
liquid fragrance particles flowed and vortex generation waschecked. Banana liquid fragrance particles passing throughthe flow measurement surface were counted to obtain the timeseries variation of the volume flow rate per second.
Furthermore, the total volume of the target particles passed

Adhered dropletsAtomized water

Spirit level

Movable floor

Anemometer

Figure 7: Illustration of experimental results (h = 34 mm).
An experimental air flow channel was fabricated by 3D
printer and acrylic sheet fabrication. Figure 7 illustrates theatomization at h = 34 mm. A digital camera was positioned at
the center of the screen at an angle that captured the entire
movable floor surface, and any droplet adhesion on thebottom of the channel during purified water spraying was
illuminated by the white LED light adopted as illuminationand visualized in white.
Figures 8, 9, and 10 present the experimental results for h =34, 63, and 64 mm, respectively, during 1 s of purified water
atomization, and the time series changed with the atomization
start frame set to 0 s. These images were clipped only inside

0.017 s0.500 s1.000 s1.100 sFigure 8: Experimental results (h = 34 mm, Colordifference from 0 s and after contrast
adjustment, Clipping only in air flow channel).

28

International Workshop on Informatics ( IWIN 2022 )form of a thin line. In Figure 10, h = 64 mm, the atomized
substance flowing in the channel was confirmed, as illustrated
in Figure 9; however, no bottom adhesion was observed, andthe same result was obtained in a total of six subsequent trials
conducted for confirmation.

0.017 s 6LPXODWLRQ&RQILUPDWLRQRIYRUWH[JHQHUDWLRQDQGUHWHQWLRQRIDWRPL]HGVXEVWDQFHGXHWRWKHVKDSHRIWKHIUDJUDQFHWDQNFRQQHFWLRQ

0.500 s1.000 s1.100 sFigure 9: Experimental results (h = 63 mm, Colordifference from 0 s and after contrast
adjustment, Clipping only in air flow channel).

Figure 11: Total volume passing through the measurement
p to 1 s after the end of atomization.
surface up

0.017 s0.500 sFigure 12: Simulation results Comparison of flowrate through shape (A) and (F).
Based on the computational environment set up, Prometech
Particleworks 7.1 was employed as the simulation software
for the calculations. The computing environment was a CPU:Intel core i9-10900K, memory:128 GB, OS: Microsoft
Windows 10 Pro, and GPU: Nvidia RTX 8000. The
computation time was approximately 30–35 h for each casein 2 s of real time. The number of particles in the entire
channel was approximately 10 million.
The calculation results were observed; in all cases, a vortexlike flow was observed in the cavities connecting the liquidfragrance tank. Figure 11 compares the total volume of theatomized substance passing through the measurement area
between 0.883 and 1.883 s after the end of the atomization.
The result of (A), a circular shape with a height of 15 mm,
which is equivalent to that of the conventional device,
exhibited the highest delayed flow, while the result of (F), a
rectangular shape with a height of 1 mm, exhibited the lowest
delayed flow. At the same height, the rectangular shapetended to have slightly less delayed flow than the circularshape. Figure 12 presents the time series variation of thevolume of the atomized substance passing through themeasurement area of geometry (A), (F), where the dotted line
represents the inflow volume of the atomized substance.

1.000 s1.100 sFigure 10: Experimental results (h = 64 mm, Colordifference from 0 s and after contrast
adjustment, Clipping only in air flow channel).
the airflow channel, and the difference in color values
between the atomizing start frame and each frame was createdusing Adobe Photoshop CS5. The color values were output
only for the areas that changed from the initial frame, and theareas that did not change are output in black.
At h = 34 mm in Figure 8, droplet adhesion is clearlyobserved over a wide area of the movable floor surface.
Droplets remained at the bottom of the air flow channel for
1.1 s after the atomization was completed and the atomized
substance had flowed out of the air flow channel. In Figure 9,
at h = 63 mm, linear droplet adhesion is observed in the centerof the movable floor in front of the air flow channel in the29International Workshop on Informatics ( IWIN 2022 ) $1$/<6,6 ([SHULPHQW&RQILUPDWLRQRIWKH
UHTXLUHGDLUIORZFKDQQHOERWWRPGLVWDQFH
DWWKHWLPHRISXULILHGZDWHUDWRPL]DWLRQThe air flow channel height of the conventionalpiezoelectric olfactory display (h = 34 mm) was deduced to
be insufficient because of the large amount of droplet
adhesion on the bottom surface. Bottom adhesions wereobserved up to h = 63 mm, but no bottom adhesions wereobserved above h = 64 mm, even after repeated atomization. The required air flow channel bottom distance at the time of
purified water atomization under a flow velocity of 1.8 m/s,
when vibrating at 100 kHz using a piezoelectric device with
100 micro-holes of 9 μm, was verified to be 70 mm, including
the dimple in the liquid fragrance tank.
In all cases, contact was observed on the sides duringatomization; therefore, the 30-mm air flow channel width
should be reviewed in the future.

 6LPXODWLRQ&RQILUPDWLRQRIYRUWH[JHQHUDWLRQDQGUHWHQWLRQRIDWRPL]HGVXEVWDQFHGXHWRWKHVKDSHRIWKHIUDJUDQFHWDQNFRQQHFWLRQThe simulation results demonstrated that the delayed flowrate of the atomized substance from the 20-mm high circular
cavity was the largest, whereas the delayed flow rate from the
6-mm high rectangular cavity was the smallest. It was alsoinferred that, for cavities of the same height, the delay flowrate tended to be smaller for the rectangular cavity than forthe circular cavity. Extending the cavity in the connection tomaintain the distance between the piezoelectric device andthe bottom of the airflow channel is not considered goodpractice in the design of piezoelectric device olfactory
displays.
This suggests that, among the shapes compared in this study,
the 6-mm high rectangular cavity is the most suitable for the
shape of the connection of the piezoelectric olfactory display.
The shape of the fragrance tank adopted in this study had a 5mm high dimple to protect the surface of the piezoelectric
device; however, this shape was also inferred to be a factor in
the generation of internal vortexes. From this perspective,
there is room for further improvement in the shape of theouter frame of a liquid fragrance tank.

the center, and when air flow was present in the channel at an
air speed of 1.8 m/s toward the outlet, the distance from theatomized position to the bottom surface was ≥ 70 mm, andthe contact of the atomized droplets to the bottom surface was
not confirmed.
The obtained simulation results indicate that if a cavity
exists at the connection between the droplet atomization
mechanism and the air flow channel, a vortex may be createdinside, which may cause the atomized droplet to remain in the
air flow channel even 1 s after the end of injection. Thissuggests that, among the shapes compared in this study, the
6-mm high rectangular cavity is the most suitable for the
shape of the connection to the piezoelectric olfactory display.
In the future, we will create a novel air flow channel structure
designed based on the results of this study. Furthermore, wewill conduct experiments on human subjects to verify theeffectiveness of the piezoelectric device olfactory display inexperiments and investigate the characteristics of humanolfaction.
The piezoelectric olfactory display with a novel air flow
channel structure designed considering these two results is
expected to reduce the difference between the time whenfragrance components actually reach the subject and that set
by the system, compared to the conventional model.
The effectiveness of this method will be verified throughexperiments on test subjects and measurements using gas
sensors.

5()(5(1&(6[1] A. Kadowaki, J. Sato, Y. Bannai, and K. Okada,
Measurement and Modeling of Olfactory Responses toPulse Ejection of Odors, J. Japan Association on Odor
Environment, Vol.39, No.1, pp.36-43 (2008).
[2] S. Nakamura, and Y. Bannai, Development of An
Olfactory Display Using a Piezoelectric Element and
Measurement of Olfactory Detection Threshold, VRSJ
Research Report. Vol.25, No.SBR-1, pp.1-6 (2020).
[3] Y. Seta, M Makino, Y. Bannai, and M. Hattori, Proposalof Air-Flow Channel Structures of The Olfactory
Display Using the Piezoelectric Element Considering
Fluid Behavior, VRSJ Research Report, Vol.27,
No.SBR-1, pp.9-14 (2022).
[4] Y. Bannai, "Olfactory display using piezoelectricelement," Odor sensing, analysis and its visualizationand quantification, pp.518-529, Technical Information
Institute, ISBN 978-4-86104-810-4 (2020).
[5] J. Sato, A. Kadowaki, K. Ohtsu, Y. Bannai, and K.
Okada, Scent Presentation Technique by Pulse Ejection
to avoid Olfactory Adaptation, Transactions ofInformation Processing Society of Japan㸪Vol.49, No.8,
pp.2922-2929 (2008).
[6] A. Aruga, Y. Bannai, and T. Seno, Investigation of the
Influence of Scent on Self-Motion Feeling by Vection,
International Journal of Informatics Society, Vol.11,
No.2. pp.65-73 (2019).
[7] K. Fujita, F. Takahashi, T. Yonemura, N. Kawabata, and
T. Ohta, Study of the Flow Structure of a Newtonian
Fluid through a Rectangular Channel with a Cavity,

 &21&/86,21
Purified water atomization experiments were conducted todesign a piezoelectric olfactory display by varying the bottom
distance. To determine the appropriate cavity shape forconnecting the liquid fragrance tank to the airflow channel,
fluid simulations were conducted to determine the presenceof vortex flow and its effect on the flow.
The obtained results demonstrated that when purified water
was atomized perpendicularly to the bottom surface for 1.0 s
from a piezoelectric device with 100 micropores of 9 ȣm in30

International Workshop on Informatics ( IWIN 2022 )Research reports of National Institute of Technology,
50th Anniversary Issue, No.49, pp.1-8 (2015).
[8] A. Aruga, S. Nakamura, and Y. Bannai, Fine Droplet
Discharge Type Olfactory Display, 24th Annual
Conference of the Virtual Reality Society of Japan, O02 (2019).
[9] Y. Seta, N. Mori, M. Makino, Y. Bannai, and M. Hattori,
Comparison of Rectification Effects of Grid Shapes inan Air-Flow Channel for Droplet Spray Olfactory
Displays, Proceedings of the 84th National Conventionof IPSJ, Information Processing Society of Japan, 2F-05(2022).
[10] M. Oochi, S. Koshizuka and M. Sakai, Explicit MPS
Algorithm for Free Surface Flow Analysis, Transactionsof the Japan Society for Computational Engineering andScience, Vol.2010, p.20100013 (2010).
[11] Japanese Industrial Standards: Pneumatic fluid power−
Standard reference atmosphere, JIS B 8393: 2000
(2000).
[12] National Astronomical Observatory of Japan,
Chronological Scientific Tables 2019, Maruzen (2019).

31

International Workshop on Informatics ( IWIN 2022 )Proposal of Potted Flower Area Segmentation Method Using Hough Transform
Shun Fujii† , Yoshitaka Nakamura‡ , Hiroshi Inamura*, and Shigemi Ishida*†

Graduate School of Systems Information Science, Future University Hakodate, Japan‡
Faculty of Engineering, Kyoto Tachibana University, Japan* School of Systems Information Science, Future University Hakodate, Japan{g2121048, inamura, ish}@fun.ac.jp, nakamura-yos@tachibana-u.ac.jp

Abstract The impact of plant diseases on agricultural production issigniﬁcant. Therefore, plant diseases must be removed or otherwise treated as soon as they are detected. Conventional methods
have attempted to improve the accuracy of disease detection.
However, the current situation is costly in terms of collecting alarge amount of image data for disease detection, installing andmanaging monitoring equipment, and extracting disease images.
In this study, we propose a method to segment an image of theentire farm ﬁeld using image processing based on the Houghtransform to isolate individual plants. We aim to improve the accuracy of disease detection by performing disease identiﬁcation
on cropped plant images using the proposed method. In the evaluation, we conﬁrmed that the proposed method can trim a single
plant with an F-measure of 72.5%, which is approximately 7
percentage points higher than that obtained by segmenting plantparts using CNNs.

on disease detection from images of the entire farm has not been
established. Therefore, we propose to construct a system thatcan extract speciﬁc parts of a ﬁeld, such as leaves, from theentire farm ﬁeld image.
In our study, we focus on potted ﬂowers, which are arranged
in a grid pattern and thus each plant is easily extracted. We
propose a 2-step disease detection method, in which each plantis extracted from the entire farm ﬁeld image by segmenting thepotted ﬂower area, and then each plant is identiﬁed for disease
identiﬁcation. In this paper, we consider a method for potted
ﬂower segmentation that uses feature extraction of grid patterns
that appear in the ﬁeld image.
This paper is organized as follows. Section 2 reviews related
studies on plant disease detection, and Section 3 describes the
proposed method for detecting plant diseases from images ofthe entire farm ﬁeld. Section 4 evaluates the proposed methodand discusses the evaluation results, and Section 5 concludes thepaper.

Keywords: smart agriculture, plant disease, image processing, Hough transform

2 Related Studies
Methods for detecting diseases in vegetables can be classiﬁed
into 2 categories: methods using deep learning based on CNNs,
and methods using image processing.

1 Introduction
Zadoks et al. reported that more than one-third of agricultural
production is wasted due to plant diseases [1]. Since plantdiseases are spread by infection among plants, it is necessary
to remove or otherwise treat plant diseases as soon as they aredetected.
Several types of research conducted on this issue, aiming
at automating the prediction and detection of plant diseases.
Several methods using deep learning based on Convolutional
Neural Networks (CNNs) have been proposed. These researches
include the detection of cucumber and tomato diseases [2], [3].
In addition to vegetables, there are also researches to detect
diseases of rice plant [4].
Since these researches target the plant disease itself, theyrequire data on speciﬁc areas such as diseased leaves. Therefore,
more than 10,000 images need to be collected, and the cost of
installing and maintaining monitoring equipment and extracting
images of diseased areas cannot be ignored. Suwa, et al. propose
a method for detecting diseases by extracting plant leaves fromimages of the entire ﬁeld [3].
As described above, although disease detection has been conducted for speciﬁc parts such as diseased leaves, but the research

2.1 Plant Disease Detection Method Using Deep
Learning Based on CNNs
CNNs are widely used in image analysis and are also widelyused for plant disease detection. Fujita, et al. achieve 83.2% ofidentiﬁcation accuracy in the classiﬁcation of 9 classes of plantdiseases on cucumber, including 8 types of diseased leaves andhealthy leaves [2]. Since these methods use image data trimmed
to a single leaf as training data, it is necessary to collect data on
leaves suspected to be diseased. Regarding the leaf trimming
problem, Suwa, et al. detected plant diseases from images of theentire farm ﬁeld and achieved disease detection rate of 91.1%[3]. In this method, a 2-step detection process is used: leaf
extraction from the ﬁeld image, and disease identiﬁcation on theextracted leaves.
Although the evaluation results of disease detection using
CNNs were reported to be generally highly accurate, these methods require a large amount of data, and the cost of data collection
is high. About 8,000 images were used in Ref.[2], and about

33

International Workshop on Informatics ( IWIN 2022 )Due to this feature, grid lines appear when background images
other than potted ﬂowers are extracted. The main idea of thismethod is to extract each potted ﬂower by trimming using theintersection points of these grid lines. The grid lines that appear
between each potted ﬂower are detected using the Hough transform[7]. The process of segmenting the potted ﬂower area using
the Hough transform and trimming individual potted ﬂowers isshown in Fig.2.

Image of Entire
Farm Field

Potted Flower Area
Segmentation

3.1 Area extraction for applying Houghtransform

DiseaseIdentification

The pre-processing for detecting grid lines from the farm
ﬁeld images in the proposed system is as follows. Since gridlines appear in background images other than individual potted
ﬂower images, this process identiﬁes background images. The
proposed method focuses on the green color of the leaf of theindividual potted ﬂowers and considers all other colors as the
background.
A method for extracting green areas of individual plants use
the RGB-based Vegetative Indexes (RGBVI) shown in Expression (1) This Expression (1) makes it possible to extract thefeatures of green areas with high accuracy.

Result of Disease
IdentificationdiseaseFigure 1: Overview of Proposed System
50,000 images in Ref.[3]. The disease detection rate decreased
to 38.9% when images taken in a diﬀerent farm ﬁeld from theone used for the training data were applied to the detector [3].

2.2 Plant Disease Detection Method Using
Image Processing

G2 − (R × B)
G2 + (R × B)

Several types of research use image processing to detect diseases with a small number of data. Jadhav, et al. proposed amethod to extract only the image area of the disease on leaves[5]. This research also used images of only a single leaf asdata like these researches [2], which does not reduce the cost.
Kawasaki proposed a method for trimming a single leaf from animage of the entire farm ﬁeld using image processing [6]. However, this method failed in trimming a leaf image. The reason
for this failure is stated to be that the image channels used forfeature acquisition are not suitable for leaf detection.
As shown in Ref.[5], it is possible to identify the type andinfection level of a disease by extracting only the diseased area if
the disease to be detected has prominent features. However, only
image processing is diﬃcult to extract leaves from the entire ﬁeld
image. While image processing methods can detect diseases
without requiring a large amount of data, features common to
all images are required for disease detection.

(1)

Our proposed system uses this method to extract green areasfrom the RGB channels of the entire ﬁeld image. The image ofthe entire ﬁeld has 3 channels, R (red), G (green), and B (blue),and all pixel values are in the range of 0 to 255. After applying
Equation(1), the pixel value of −1 to 1 is set to 0 to 255 andconverted to a gray-scale image.
The converted gray-scale image is a multi-stage image, andthe Hough transform cannot be applied. Therefore, image binarization is required. Our proposed system applies Otsu’s binarization algorithm[8]. After binarization, the farm ﬁeld image isrepresented as white for individual potted ﬂowers and black forthe background. There are white spots in the individual potted
ﬂower area and black spots in the background area, which arenoise. The proposed method removes these noises by applying expansion processes to the white areas in the image. Since
the Hough transform detects white areas as straight lines, thebinarized image is inverted to black and white.
This process makes individual potted ﬂower areas and background areas separate from the entire farm ﬁeld image.

3 Disease Detection System Using Potted Flower
Area Segmentation
In this study, we propose a 2-step system that identiﬁes diseases after individual plants are extracted, to enable disease
detection from images of the entire farm ﬁeld. The proposedsystem is applied to potted plants, which are arranged in a grid
pattern and thus each plant can be easily extracted. Figure 1
shows an overview of the proposed system.
In order to segment potted ﬂower areas using image processing, this method takes advantage of the feature that potted ﬂowers
are arranged and grown in a grid pattern over the entire ﬁeld.

3.2 Line detection by Hough transformIn the image shown in Fig.2(1), the background area is displayed as a linear white area with many line segments and lines
mixed in. If the Hough transform is directly applied to the image,
many line segments and lines unrelated to potted ﬂowers are also
extracted. Therefore, it is diﬃcult to extract the optimal lines for
dividing the individual potted ﬂower area and the background34

International Workshop on Informatics ( IWIN 2022 )Image of Entire Farm Field

Potted Flower Area Segmentation
Area extraction for applying
Hough transform(1)

Line detection by Hough transformIs there any oblique line present?

(2)

(3)

(5)

(6)





Removal of oblique lines(4)

Determination of area segmentation lines

Trimming at the intersectionof area segmentation lines(7)

Disease Identification

Figure 2: Process of Potted Flower Area Segmentation
area. The proposed system ﬁrst adjusts the parameters of theHough transform so that line segments with start and end points
in the background area are output. The Hough transform after
parameter adjustment is called the segment Hough transform.
The segment Hough transform produces line segments that are
candidates for line segments that divide the individual potted
ﬂower area and the background area. In order to trim the individual potted ﬂower area, these line segments must be straights
line that spans the entire image. The overlapping line segments
are good candidates for boundaries between areas. Therefore,
the line segments outputted by the segment Hough transform
are thinned, and then the thinned line segments are expanded.
These processes produce straight lines that divide the individual
potted ﬂower area and the background area in the entire farm
ﬁeld image. This is called the line Hough transform. When theline calculated here is merged into the input RGB image, it canbe seen that a line can be detected in the background.

classiﬁed using a clustering method to separate the oblique lines.
The proposed method uses a density-based clustering method,
DBSCAN [9], as the clustering method. Since oblique lines arefewer than straight lines in the horizontal and vertical directions,lines belonging to the top 2 clusters in terms of the number oflines are used as area segmentation lines, and lines belonging
to the remaining clusters are removed. If all lines are classiﬁed
into 2 clusters, no oblique lines are judged.
As an example, the result of clustering the slope of lines for theimage shown in Fig.2(2) is shown in Fig.3. Fig.3 explains that thevertical axis represents the slope of the lines, and the horizontal
axis is set to 0 as the speciﬁed value since the clustering is based
only on the slope of the lines. The result shows that the clusters
are classiﬁed into clusters with 38 lines, clusters with 5 lines,
and clusters with 29 lines. The 5 straight lines belonging to
the middle cluster are separated as oblique lines and removed(Fig.2(4)).

3.3 Removal of oblique lines

3.4 Determination of area segmentation lines

If the image after applying the line Hough transform (Fig.2(2))
contains oblique lines, these lines are noise unrelated to theindividual potted ﬂower areas and must be removed. In theprocess of the Hough transform, the slope and intercept of thedetected line are obtained. Based on the slopes, each line isSince the image after the line Hough transform has multiple
overlapping lines, these lines cannot be used as area segmentation lines for trimming individual potted ﬂowers. Therefore, all
the lines are thickened, and the overlapped lines are clustered.
If the lines are too thin, they cannot be classiﬁed into the same

35

International Workshop on Informatics ( IWIN 2022 )Table 1: Potted Flower Area Segmentation
Proposed CNNs-based
MethodMethodPrecision [%]
78.5
94.3
Recall [%]
67.3
49.5
F-measure[%]
72.5
65.0

TP
TP + FP(2)

TP
TP + FN(3)

Precision =Recall =Figure 3: Result of DBSCAN with the Slope of Linescluster. If the lines are too thick, lines that should be in diﬀerent
clusters are classiﬁed into the same cluster. For this reason, thethickness of the lines should be set to about 20 pixels. The line
passing through the center of the clustered lines is used as thearea segmentation line (Fig.2(5)). The calculated area segmentation line is found to be the best line for segmenting individual
potted ﬂowers(Fig.2(6)).

F-measure =2 × Precision × Recall
Precision + Recall(4)

In this evaluation, TP (True Positive) is the case where potted
ﬂowers are trimmed from the test data image, FN (False Negative) is the case where no potted ﬂower is trimmed from the test
data image, and FP (False Positive) is the case where the test
data image has been trimmed to include parts of the image thatare not potted ﬂowers. Therefore, precision is the probability
that the trimmed image is a single potted ﬂower, and recall is theprobability that multiple potted ﬂower plants are cropped as asingle plant. F-measure is the harmonic mean of precision andrecall.
Table 1 shows the results of potted ﬂower area segmentation. The proposed potted ﬂower area segmentation method
achieved 78.5% precision, 67.3% recall, and 72.5% F-measure,while the CNNs-based segmentation method achieved 94.3%
precision, 49.5% recall, and 65.0% F-measure. The proposed
method achieves higher recall and F-measure than the CNNsbased method, while the CNNs-based segmentation method
achieves better precision. From the results, the proposed method
reduces the number of missed potted ﬂower plants compared to
the CNNs-based segmentation method.
The reason for the 78.5% precision of the proposed method
may be due to the low accuracy of segmentation in the denseareas of potted ﬂowers. The images of unsuccessful segmentation are shown in Fig.4. From the left, the input image, thepre-processed image, and the image after the second Houghtransform. The reason for the low segmentation accuracy isthat the background area could not be detected due to the distortion from the center to the edges of the image. Recall isalso considered to be limited to 67.5% due to the low accuracy of image segmentation as well as precision. Therefore, if
the segmentation accuracy of the individual potted ﬂower area
can be improved by correcting the distortion of the image andadjusting the parameters of the Hough transform, the overall accuracy of the potted ﬂower area segmentation method can also
be improved.

3.5 Trimming at the intersection of area
segmentation lines
Trimming at the intersections of the area segmentation line
obtained allows the potted ﬂower to be divided into individual plants. The proposed method trims the largest rectangle
with 4 intersections. In this way, the trimmed individual potted
ﬂower image retains the maximum amount of image information(Fig.2(7)).

4 Evaluation
We evaluate the potted ﬂower area segmentation method. For
the evaluation of the potted ﬂower area segmentation method,
we compare the accuracy of the proposed method with that of
the CNNs-based segmentation method.
The data used for the evaluation were 78 image. Evaluation
data is taken so that "one image shows about 60 plants".

4.1 Evaluation of the Potted Flower Area
Segmentation Method
The evaluation criterion of the potted ﬂower area segmentation method is whether or not only one whole plant is trimmed.
The term "only one whole plant" here refers to the state in which
one whole plant is captured, and only a portion smaller than halfof the other plants are captured.
This evaluation criterion are the precision (Equation(2)), therecall (Equation(3)), and the F-measure (Equation(4)).

36International Workshop on Informatics ( IWIN 2022 )

(a)

(b)

[6] Y. Kawasaki: Leaves Defection and Recognition Method
for Automated Plant Disease Diagnosis, Bulletin of Graduate Science and Engineering, Engineering Studies of Hosei
University, Vol.58, pp.1–4 (2017).(in Japanese)[7] D. H. Ballard: Generalizing the Hough Transform to Detect Arbitrary Shapes, Pattern Recognition, Vol.13, No.2,
pp.111–122 (1981).
[8] N. Otsu: A Threshold Selection Method from Gray-Level
Histograms, IEEE Transactions on Systems, Man, and Cybernetics, Vol.9, No.1, pp.62–66 (1979).
[9] M. Ester, H.-P.Kriegel, J. Sander, and X. Xu: A DensityBased Algorithm for Discovering Clusters in Large Spatial Databases with Noise, Proceedings of the 2nd International Conference on Knowledge Discovery and Data
Mining (KDD’96), pp.226–231 (1996).

(c)Figure 4: Unsuccessful Segmentation((a): Input image, (b):Pre-processed image, (c): Image after line Hough transform)

5 Conclusion
In this paper, we proposed a method to improve the accuracyof plant disease detection by image processing using the Houghtransform. The results of trimming individual potted plant areas
in a grid pattern feature using the Hough transform from imagesof the entire farm showed that one potted plant was trimmedwith 72.5% F-measure, which is about 7 points higher than thatof the segmentation method using CNNs.
As the results, proposal method achieved high accuracy in thesegmentation with 136 images than R-CNN. And the system iscapable of segmenting individual potted ﬂowers as long as thetarget object is green and arranged in a grid pattern.

REFERENCES[1] J. C. Zadoks: Crop Production and Crop Protection: Estimated losses in major food and cash crops, Agricultural
Systems, Vol.51, No.4, pp.493–495 (1996).
[2] E. Fujita, H. Uga, S. Kagiwada, and H. Iyatomi: A Practical Plant Diagnosis System for Field Leaf Images andFeature Visualization, International Journal of Engineering & Technology, Vol.7, No.4.11, pp.49–54 (2018).
[3] K. Suwa, Q. H. Cap, R. Kotani, H. Uga, S. Kagiwada, andH. Iyatomi: A Comparable Study: Intrinsic Diﬃculties ofPractical Plant Diagnosis from Wide-Angle Images, Proceedings of the 2019 IEEE International Conference on
Big Data (Big Data), pp. 5195–5201 (2019).
[4] Q. Yao, Z. Guan, Y. Zhou, J. Tang, Y. Hu, and B. Yang:
Application of Support Vector Machine for Detecting Rice
Diseases Using Shape and Color Texture Features, Proceedings of the 2009 International Conference on Engineering Computation, pp.79–83 (2009).
[5] T. Jadhav, N. Chavan, S. Jadhav, and V. Dubhele: A Review
on Plant Disease Detection using Image Processing: International Research Journal of Engineering and Technology(IRJET), Vol.6, No.2, pp.2526–2530 (2019).

37

International Workshop on Informatics ( IWIN 2022 )$)HDWXUH'DWD'LVWULEXWLRQ6FKHPHIRU3HUVRQ 7UDFNLQJ 6\VWHPVZLWK0XOWLSOH&DPHUDV
Satoru Matsumoto* and Tomoki Yoshihisa**

Cybermedia Center, Osaka University, Japan{smatsumoto, yoshihisa}@cmc.osaka-u.ac.jp

Abstract-Recently, public cameras are widely used and are
installed in various places. These multiple cameras can beused for tracking lost children or criminals. For person
tracking, most systems send feature data such as feature
values or person images to a server. The server compares thedata and judges whether they are the same person. Artificial
intelligence and numerical analyses can be used for thecomparison. However, in this conventional scheme, the dataamount that is sent to the server increases as the number ofcameras increases. Hence, in this research, we propose ascheme for distributing the computational loads of the server
arose in the conventional scheme. We propose two methods
to determine the timing for camera devices to send feature
data to other cameras. We evaluate these proposed methods
and compare their performances. The simulation results showthat the average traffic for each camera device can be reduced
significantly compared to the average traffic under aconventional scheme.
.H\ZRUGV: public cameras, feature data, processing servers,
peer-to-peer

Fig 1. Overview of person re-identification
proposed method, a camera network is built by multiple
camera devices that can communicate with each other. Whena person is captured in a camera's field of view, the feature
values of the person image are calculated. The camera device
then transmits the calculated feature values to the camera
devices where the person is going to be captured next. The
camera device that receives the feature values compares thefeature value of each captured people with those received
before and judges whether the captured person is captured
before by other camera devices. Since each camera device
performs the process of person re-identification using themodel in the camera, our proposed method has a large
possibility to suppress the concentration of the load on theserver as the number of cameras and persons increases.
Furthermore, to evaluate how much communication is
generated when people are tracked using our proposed
method, we create a simulator. We compare the averagecommunication traffic of the server under a conventional
method and that of our proposed method and confirm that our
proposed method can distribute the load.
The organization of the paper is as follows. Section 2
inscribes existing research on person re-identification, aproblem deeply related to this research. Section 3 inscribes
the proposed method, and the evaluation results are shown in
Section 4. Finally, we conclude the paper in Section 5.

 ,1752'8&7,21
Due to the recent trend of Society 5.0 and Smart city, thedevelopment of comfortable cities using IT technology has
attracted great attention. Understanding how and whenpeople move through the city can be useful in solving various
social issues, such as marketing and research on human flow.
Therefore, obtaining the travelling routes of people movingaround the city contributes to the comfortable cities. If it ispossible to track people using many security cameras
installed in cities, we can track many people widely.
Some methods to detect a person in multiple images
obtained from multiple cameras that do not share the same
field of view have been proposed. These methods track thepeople by mapping same person across multiple cameras.
These studies use deep learning or deep distance learning [14] to obtain feature representations with high discriminative
power.
However, when identifying the same person from multiple
images obtained from multiple cameras, a large amount ofcommunication bandwidth is required if all the images aretransmitted to the server. In addition, if all the information
obtained from the video is sent to the server and the person
re-identification process is performed on the server, the loadon the server increases as the number of cameras and persons
tracked increases. Even in the case of using cloud video
recording services, the load on the recording server increases.
Hence, in this paper, we propose a person tracking methodthat does not concentrate the load on the server. In the5(/$7(':25.

Person re-identification is the problem of identifying thesame person from images of people captured by multiple
cameras that do not share the same field of view. Given aquery image, the person re-identification system searches fora person identical to the query image in the gallery images(see Figure 1). Many studies improved the accuracy of person
re-identification. Some of them use distance learning such as
triplet loss, etc. [5-10].
Person re-identification is expected to have a wide range ofapplications in computer vision, such as surveillance,

39

International Workshop on Informatics ( IWIN 2022 )behavior analysis, and person tracking, but on the other hand,
it has a major problem due to gaps between cameras. Whenusing multiple person images captured by multiple camerasthat do not share the same field of view to perform person reidentification, the inter-camera gaps are unavoidable due tothe nature that the person images used were captured bydifferent cameras [6].
Therefore, person re-identification systems that can give ahigher accuracy even when the influences of these varieties
are large.
Many research has been done to overcome these effects. In[5], an adversarial network is used to obtain a more accuratefeature representation that eliminates gaps between cameras
as much as possible. The method proposed in [7] uses
StarGAN to transform the styles of people in images. The
method transforms the images of the people captured by acamera device to the images that consider the shooting
conditions (background, lighting, etc.) of other cameras, thenit uses these images as training data to reduce the influence ofgaps between cameras. Also, there is a study that investigate
how the variety of viewpoints affects the accuracy of person
re-identification, as in [6].
Although many studies have been conducted to reduce theinfluence of above differences in conditions between cameras,
the several problems still exist.
For the case where the camera images overlap, [8] performs
partial figure re-identification using local features. To solve
the problem of cloth changes, [9] systematically investigates
how the accuracy of re-identification of existing models
changes when clothes changes by generating pedestrian
images with other clothes. In [10], a method that person reidentification with removing the external information ofclothes and focuses on body shape information is proposed.
However, the systems that adopt these existing methods
need to collect all camera images to a computational server.
This causes a large communication and processing loads on
the server. Even in the conventional method under that thecameras send only the feature values of the identified people
to the server, the loads concentrate on the server. We aim torelief this loads for person re-identification in the paper.

person tracking scheme in which features are transmitted
among cameras. In our proposed method, a camera network
is built using multiple camera devices that can communicate
with each other, and the travelling paths of people in the target
area are tracked by repeatedly sending and receiving feature
values between camera devices and re-identifying people.
If re-identification fails, the system cannot track the person.
Thus, the tracking performance can deteriorate comparedwith the system that a server manages all the cameras.
However, our proposed system can distribute thecommunication and processing loads arose on the server in
the above system.

 7UDFNLQJ0HWKRGIn this section, we describe the process flow of person
tracking using camera device network. The proposed method
is based on the following four assumptions.
͌ All camera devices that are connected to the camera
device network can communicate with each other andsend feature values.
͌ All camera devices have a neural network model that
calculate the feature values of a captured person. The
input of the model is him/her image. Each camera device
gets the images from their connected cameras.
͌ The locations and the angles of the cameras are fixed, andthe positioning of all cameras is assumed to be known inadvance.
͌ All camera devices can estimate the direction of
movement of a person using the coordinate and theinterframe information.
Under the above assumptions, the camera devices
connected to the computer network track the travelling pathof a person by repeatedly transmitting feature values andjudging whether the person is the same person. The following
is an overview of the process flow when a person is
successfully tracked between Camera A and Camera B.
1. Camera A captures a new person X.
2. Camera A detects a person, acquires a person image, andcalculates the feature of the person X using a neuralnetwork model.
3. The destination camera device is determined by thedestination determination method (detailed in 3.4) and thefeature X is sent.
4. Camera B adds the feature X to the gallery.
5. A person moves and is captured by Camera B.
6. Camera B computes the feature values and compares themwith the feature values X in the gallery to determine thatthey are the same person or not.
7. The fact that the person captured by Camera A was also
captured by Camera B indicates that the person moved
from A to B
.Figure 2 shows an image of a person re-identification
process. In Figure 2, there are two separate images of people
on the left side, and they are input to the same neural networkmodel (NN model). The distance between the output features
is calculated. The distance is between the features is used tojudge whether the persons in the images are the same person
or not.

 352326('0(7+2'
In this section, we first provide an overview of the
proposed person tracking method. After that, we explain thedetail.

 6XPPDU\Consider the case of a person tracking using surveillancecameras deployed throughout a city or a facility, using a
conventional person re-identification method as described in
Section 2. In this case, the method of transferring the images
captured by the cameras to a server via a computer networkand processing the person re-identification on the server
requires a large communication traffic for the transmission of
the images. Moreover, the methods in which information
obtained from the video is transferred to the server and theperson re-identification process is performed on the server
increases the load on the server as the number of cameras or
persons increases. To solve these problems, we propose a40

International Workshop on Informatics ( IWIN 2022 )Human image
A

11PRGHO)HDWXUHGDWD)HDWXUHGLVWDQFHLVXQGHUWKH
WKUHVKROG"

Human image
B

11PRGHO<HV

6DPHSHUVRQ)HDWXUHGDWD

Fig 2. An image of a person re-identification process

Fig 3. Time to start communication under our proposed
method 3URFHVVHVIRU(DFK&DPHUD

after a certain time has elapsed, preventing features that are
not used for tracking from remaining in the gallery.
Based on the above approach, we propose two types ofmethods for determining the transmission destination. The
image of each method is shown in Figure 3. The first one isto transmit feature values to all the neighboring cameras
within N hops when a person is captured by a camera device.
The transmission timing is when the moving direction of theperson is predicted. After the reception, the camera devices
that are not likely to capture the person need to delete thefeature values from the gallery (the proactive method). The
other one does not transmit feature values when a person is
captured by a camera device but transmits feature values tothe camera devices that exist in the destination direction when
the direction of the person is predicted (the reactive method).

The flow of processes executed by each camera device isshown below.
1. A person is captured by the camera.
2. Obtain bounding boxes and calculate features with NNmodels.
3. Person identification by comparing the calculated featureswith those in the gallery.
4. If these match, go to 7.
5. If these do not match, the feature values are sent to acamera device that is determined using the destination
determination method (see 3.4) because the person is a
newly detected person. The received camera device adds
the feature values to the gallery.
6. Return to 1.
7. Notifies the server that a person has been detected.
8. The features of the person are again sent to the camera
device determined using the destination determination
method. The received camera device adds the feature
values to the gallery.
9. Return to 1.

 3URDFWLYH0HWKRG
The flow of the proactive method is shown in Figure 4. The
gray areas in the figure represent roads. The people walk on
those areas. For simplicity, the roads are grid-shaped asshown in the figure, but the same process can be applied toroads that are not grid-shaped. The camera devices are
assumed to be located at each intersection, and the locationsof the camera devices are marked with the numbers (1 to 6).
The parameter N is set to 2, which indicates how manycameras are to transmit the feature values to the next camera.
Procedure 1 shows how the features are transmitted whena person is detected by Camera 1. Camera 1 sends the feature
values of the person to the surrounding N (= 2) camera
devices when it detects a person. (Cameras with red numbers
are the those hold the feature values.) In Procedure 2, theperson moves from the area that Camera 1 shoots to the areathat Camera 2 shoots. Camera 1 judges that Camera 2 is thecamera that may capture the person in the next based on its
direction.In Procedure 3, Camera 4 and Camera 6 are notified
to remove the feature values from the gallery. This avoids thecameras that are unlikely to capture the person fromcontinuing to have the feature values and reduces the numberof candidates for the person re-identification. Figure 5 shows
the processes for each camera device in the proactive method.
In the proactive method, when a new person is captured in thefield of view of a camera, it calculates the feature values ofthe person. Then, the camera device determines whether thecaptured person is the same person that other camera devices
capture before, by calculating the distance among feature
values. When it finds the same person, it sends only the
information that the person was captured to the server. In the 'HVWLQDWLRQ'HWHUPLQDWLRQ0HWKRGVIn this section, we describe a method for determining thedestination camera device for transmitting feature values toanother camera. In order to track a person, it is consideredthat when a person is captured by a camera, the feature values
should be transmitted to its neighboring cameras. This isbecause the cameras neighboring to the camera device that aperson captured is likely to be captured in the next. However,in the method where the feature values are transmitted only
to the neighboring cameras, there is a possibility that the
tracking of a person fails if the neighboring cameras fail todetect the person. One of the solutions for avoiding the
failures is transmitting the feature values to further
neighboring camera devices (the neighboring camera devicesof the neighboring camera devices, etc.). Therefore, in the
proposed method, we introduce a parameter N that indicatesthe number of the communication hops from the source
camera device to send the feature values.
As described in section 3.2, camera devices can predict thedirection of moving people, and therefore, it is possible tolimit the transmission destinations by using the direction.
That is, the direction of movement can be used to limit thetransmission destination. The transmitted features are deleted

41

International Workshop on Informatics ( IWIN 2022 )example for the proactive method in the previous subsection.
Unlike the proactive method, the reactive method starts
transmitting features after the moving direction of the person
is found.
Procedure 1 shows the movement of the person from thearea that Camera 1 shoots to that of Camera 2. In Procedure
2, the feature values are transmitted only to the camera device
that exist in the direction of the person when Camera 1 detects
it. We assume that the direction is predictable based on thetravelling path of the person in the camera's field of view,
such as the trajectory of the person and the position at which
the person frames out.
The reactive method has the advantage of reducing theamount of communication because each camera device
predicts the direction in which a person is moving andtransmits the feature values only to the camera devices thatexist in the direction. On the other hand, if the direction of theperson cannot be predicted correctly, the feature values are
not transmitted to the camera devices in the direction of theperson, thus the tracking fails. If the terrain is complex, or if
it is considered difficult to correctly predict the direction of aperson due to the positional relationship among cameras, theprobability of tracking failures can be high.
Figure 7 shows the processes for each camera device in thereactive method. In the reactive method, as in the proactive
method, each camera device calculates feature values and reidentifies people when a new person is captured in the fieldof view. However, the feature values are not transmitted
immediately, but only to the N neighboring cameras in the
direction of their movement after predicting them based on
their trajectories.

Fig 4. The flow of the proactive method
A person frames-in
Calculate feature values andperson re-identification
Transmit feature values tocamera devices (9$/8$7,21
To evaluate the amount of communication traffic generated
when tracking a person under our proposed method, we
created a simulator and measured the performances.

Predict direction of person
Notify feature value
deletions

Fig 5. Processes for each camera device in the proactive
method6LPXODWLRQ6SHFLILFDWLRQVTo systematically evaluate the performance of our
proposed methods, we assume that the roads are grid-shape.
A camera network is built with camera devices that cancommunicate with each other and are located at eachintersection.
In order to simulate different map sizes, we use threedifferent maps as a 4 × 5 grid with 20 cameras, a 5 × 6 gridwith 30 cameras, and a 6 × 7 grid with 42 cameras. One
section of the grid is fixed by 10 meters. The map becomeslarger as the number of cameras increases.

proactive method, after that, the camera device transmits thefeature value of the captured person as well as own Camera
ID to N (at maximum) neighboring camera devices. In thiscase, the system can avoid duplicate transmissions because it
is possible to find the camera devices to which the feature has
been transmitted in the past from the list of camera IDs. If thesame person is not found, it is assumed that the person is a
new person and the feature is sent to all camera devices to N
(at maximum) neighbors. If the direction of the person ispredictable from the direction and the location information at
the time of frame-out, the number of galleries for person reidentification can be reduced by notifying the camera devices
to delete the features stored that is not likely to capture theperson.

 3DUDPHWHUVWe change the following five parameters in the simulator.
͌ The parameter to determine the number of the camera
devices that receive the feature values. When the value isN, the feature values are transmitted to N neighboring
camera devices.
͌ The number of persons flowing into the tracking area per
a second.

 5HDFWLYH0HWKRG
The flow of the reactive method is shown in Figure 6. The
road and the camera devices deployment are the same as the42International Workshop on Informatics ( IWIN 2022 ) (YDOXDWLRQ,WHPVThe change in the communication traffic under thecondition of different number of the camera devices anddifferent people inflow.
The change in the tracking success rate in the proposed
method changing the communication bandwidth. The
tracking success rate is the rate that the number of the people
that are tracked from the time to enter the tracking area to thetime to exit divided by the number of the entered people.
When the communication delays among the camera devices
are all shorter than the one block travelling time of a person,
the person is tracked in the tracking area.
Comparison of the average communication traffic of theserver in a conventional method, in which the camera devices
transmit the feature values to the server with that under our
proposed method.

Fig 6. The flow of the reactive method &RPPXQLFDWLRQ7UDIILFIRU
7UDQVPLWWLQJ)HDWXUH9DOXHVA person frames-in
Calculate feature values andperson re-identification

We evaluated how the communication traffic changes
when the number of cameras is changed to 20, 30, or 42, andwhen the number of people per second is changed to 1, 2, 3,
or 4. In this evaluation, we assume that the communication
traffic for one set of feature values is 1.648 [Kbit] assuming
a 50-dimensional vector of float32 as the feature data. Also,
three 16-bit regions are assumed to be reserved to record the
ID identifying the person and the cameras that have passed
through the area). This is an example setting. The detection
probability for each camera device is set to 0.8.
The results are shown in Figure 8. The vertical axis
represents the communication traffic for transmitting feature
values. The unit is Kbps. The horizontal axis represents thenumber of people per second. From the results, it can beconsidered that there is a proportional relationship between
the number of people and the communication traffic. The
communication traffic ranges from 15 [Kbps] to 170 [Kbps]
when the number of the camera devices is between 20 and 42
and the number of people per second is between 1 and 4.
We confirmed that there is a proportional relationshipbetween the number of cameras and the amount ofcommunication.

Predict direction of person
Transmit feature values tocamera devices

Fig 7. Processes for each camera device in the reactive
method͌ The number of the camera devices deployed in the
tracking area. As described in the previous section, thenumber is one of 20, 30, or 42.
͌ Since a person entering an intersection is not always
detected by the camera, the detection probability can bechanged as a parameter ranging from 0.0 to 1.0. This value
depends on perspective, lighting, and resolutions in realsituations, but these conditions are various and thus wegive the probability as a parameter.
͌ We set the communication bandwidth available fortransmitting feature values. By setting the communication
bandwidth, we can calculate the delay time fortransmission.

 7UDFNLQJ6XFFHVV5DWH

People walk at a speed of 1 meter per a second. Since one
section of the grid is 10 meters long, the time between one
camera capturing the person and the next is 10 seconds. A
person enters the map at the upper left corner and exits at thelower left, the upper right, and the lower right corners. The
number of people exiting from each exit is adjusted to be thesame.

In order to track a person without tracking failures due tolatency, it is necessary to provide more bandwidth than theamount of communication generated. If the amount ofcommunication per second generated by tracking exceeds the
bandwidth provided, the delay in sending features will
increase as tracking continues, and the delay will diverge toinfinity. If the bandwidth is not sufficient for the number ofpeople travelling through the area, the tracking will fail due
to delay.

 (YDOXDWLRQ5HVXOWV &RPSDULVRQRI&RPPXQLFDWLRQ7UDIILF

 3HUVRQ7UDYHOOLQJ0RGHOWe simulated and compared the average communication
traffic of the server (the amount of data received per unit of

We get the results under the following situations.

43International Workshop on Informatics ( IWIN 2022 )devices to send feature data to other cameras. We developed
a simulator for the evaluation and simulated the situation inthat the number of the camera devices is 20, 30 or 42, and thetracking area is a grid-shape roads. Our simulation evaluation
revealed that the average traffic per camera device under our
proposed method can be reduced significantly compared tothe average traffic on the server.
Future work includes the simulation of a more realistic
environment and the evaluation on a real situation.

$&.12:/('*(0(17This research was supported by a Grants-in-Aid for
Scientific Research(C) numbered 21H03429, 20K11829,
20H00584 and by G-7 Scholarship Foundation.
Fig 8. Communication traffic changing the number ofcamera devices and the people inflow (left: the proactive
method, right: the reactive method)

5()(5(1&(6[1] D. Wu, S.-J. Zheng, X.-P. Zhang, C.-A. Yuan, F. Cheng,
Y. Zhao, Y.-J. Lin, Z.-Q. Zhao, Y.-L. Jiang, and D.-S.
Huang, "Deep learning-based methods for person reidentification:acomprehensive
review,"
Neurocomputing, vol. 337, pp. 354-371, 2019.
[2] S. Liao and L. Shao, "Graph sampling based deep metriclearning for generalizable person re-identification,"
arXiv:2104.01546, 11pages, 2021.
[3] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. Hoi,
"Deep learning for person re- identification: A survey andoutlook," IEEE Transactions on Pattern Analysis andMachine Intelligence, 20pages, 2021.
[4] W. Wu, D. Tao, H. Li, Z. Yang, and J. Cheng, "Deepfeatures for person re-identification on metric learning,"
Pattern Recognition, vol. 110, 12pages, 2021.
[5] Y. Li, R. Xue, M. Zhu, J. Xu, and Z. Xu, "Angular triplet
loss-based camera network for reid," in 2021 International
Joint Conference on Neural Networks (IJCNN), pp. 1-7,
2021.
[6] X. Sun and L. Zheng, "Dissecting person re-identificationfrom the viewpoint of the viewpoint," in Proceedings ofthe IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pp. 608-617, 2019.
[7] Y. Lin, Y. Wu, C. Yan, M. Xu, and Y. Yang,"Unsupervised person re-identification via cross- camera
similarity exploration," IEEE Transactions on Image
Processing, vol. 29, pp. 5481- 5490, 2020.
[8] X. Zhang, Y. Yan, J.-H. Xue, Y. Hua, and H. Wang,"Semantic-aware occlusion-robust net- work for
occluded person re-identification," IEEE Transactions
on Circuits and Systems for Video Technology, vol. 31,
no. 7, pp. 2764-2778, 2021.
[9] F. Wan, Y. Wu, X. Qian, Y. Chen, and Y. Fu, "Whenperson re-identification meets change- ing clothes," inProceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition Workshops, pp. 830831, 2020.
[10] X. Qian, W. Wang, L. Zhang, F. Zhu, Y. Fu, T. Xiang,
Y.-G. Jiang, and X. Xue, "Long-term cloth-changing
person re-identification," in Proceedings of the Asian
Conference on Computer Vision, 17pages, 2020.

Fig 9. Comparison of communication traffictime) and that under the proposed method. These traffic arises
when all the feature values of the people captured by a camera
device are transmitted to the server or other camera devices.
The simulation results are shown in Figure 9. The horizontal
axis is the person detection probability explained in Section
4.1.1.
The communication traffic for each camera device in our
proposed method is reduced about one-twelfth at mostcompared to the average communication traffic under theconventional method, in which the server identifies the same
person on the server. This indicates that the load that wasconcentrated on the server in the conventional method is
distributed to each camera device under our proposed method.

 &21&/86,21
A human tracking scheme in which each camera device
sends the camera image to a server causes a large
communication and processing loads on the server. This
lengthens the delay for tracking and deteriorates the tracking
success rate. Hence, in this research, we proposed a humantracking method in which each camera device transmits
feature values of captured people among camera devices. We
proposed two methods to determine the timing for camera

44

International Workshop on Informatics ( IWIN 2022 )6WDWLRQDU\+XPDQ5HFRJQLWLRQE\0XOWLSOH9HUWLFDOO\$UUDQJHG'/L'$5V
7DNX\D:DWDQDEH <X\D6DZDQR <RVKLDNL7HUDVKLPD 5\R]R.L\RKDUD͊*UDGXDWH6FKRRORI.DQDJDZD,QVWLWXWHRI7HFKQRORJ\-DSDQ
6RND8QLYHUVLW\-DSDQ͊.DQDJDZD,QVWLWXWHRI7HFKQRORJ\-DSDQ$EVWUDFW  6HYHUDO PDQXIDFWXUHUV KDYH VWDUWHG WKHGHYHORSPHQW RI DXWRPDWHG YHKLFOH WHFKQRORJLHV ZKLFK FDQEH DGRSWHG IRU YDULRXV DXWRPDWHG URERWV RU VPDOO FDUWV+XPDQ UHFRJQLWLRQ WHFKQRORJLHV LV RQH VXFK NLQG RI
DXWRPDWHG YHKLFOH WHFKQRORJ\ ,Q VHYHUDO FDVHV VWDWLRQDU\KXPDQUHFRJQLWLRQLVUHDOL]HGXVLQJSRLQWFORXGGDWDIURPDWKUHHGLPHQVLRQDO ODVHU LPDJLQJGHWHFWLRQDQGUDQJLQJ '
/L'$5 +RZHYHULWUHTXLUHVQXPHURXVFRPSXWHUUHVRXUFHVVXFK DV D &38 DQG PHPRULHV ,Q WKLV VWXG\ D QHZ KXPDQUHFRJQLWLRQ PHWKRG XVLQJ SRLQWFORXG GDWD IURP PXOWLSOHYHUWLFDOO\ DUUDQJHG '/L'$5V LV SURSRVHG DQG LWVSHUIRUPDQFHLVHYDOXDWHG

zz

7RVDWLVI\WKHVHUHTXLUHPHQWVWKHUHDUHIROORZLQJLVVXHVzz

7KHGHWHFWLRQRIKXPDQV
7KHDFFXUDF\RIORFDOL]DWLRQ

7LOO GDWH VHYHUDO KXPDQ GHWHFWLRQ PHWKRGV KDYH EHHQSURSRVHGDVWKHVHWHFKQRORJLHVDUHUHTXLUHGIRUDXWRQRPRXVGULYLQJ +RZHYHU D PDMRULW\ RI WKHVH WHFKQRORJLHV DVVXPHWKDWKLJKVSHFLILFDWLRQFDPHUDVDUHLQVWDOOHGDQGWKHGLVWDQFH
FDQ EH GHWHFWHG E\ PXOWLSOH FDPHUDV RU D VWHUHRFDPHUD ,Q
UHDOLW\LWLVGLIILFXOWWRVHQVHWKHGLVWDQFHXVLQJDWPRQRFXODUFDPHUD,IWKHVL]HRIREMHFWLVNQRZQWKHGLVWDQFHDWZKLFKWKHKXPDQRUDQRWKHUREMHFWLVSUHVHQWFDQEHHVWLPDWHG+RZHYHU LW FDQQRW DOZD\V EH HVWLPDWHG 7KHUHIRUH LW LV
GHVLUDEOH WR EH DEOH WR GHWHFW KXPDQV DQG PHDVXUH WKHGLVWDQFH XVLQJ RQO\ /L'$5 7KH SRLQWFORXG GDWD IURP WKH'/L'$5 DUH TXLWH ODUJH DQG UHTXLUH H[WHQVLYH FRPSXWHUUHVRXUFHV7KHUHIRUHDKXPDQGHWHFWLRQPHWKRGLVSURSRVHG
E\XVLQJSDUWVRIWKHSRLQWFORXGGDWDIURPWKH'/L'$5
WKDWLVPXOWLSOHYHUWLFDOO\DUUDQJHG'/L'$5V
7KH LVVXH RI GHWHFWLQJ VWDWLRQDU\ KXPDQV LV WKH IRFXV RIWKLV VWXG\ DV WKHUH DOUHDG\ H[LVWV H[WHQVLYH UHVHDUFK IRU WKHGHWHFWLRQ RI ZDONLQJ KXPDQV ,Q 6HFWLRQ  WKH SURSRVHG
SURMHFW DQG WKH IHDWXUHV RI WKH /L'$5 DUH H[SODLQHG
3UHYLRXV UHODWHG VWXGLHV DQG WKH SURSRVHG PHWKRG DUH
SUHVHQWHGLQ6HFWLRQVDQGUHVSHFWLYHO\,QVHFWLRQWKH
H[SHULPHQWLVH[SODLQHGDQGWKHHYDOXDWLRQVDUHSUHVHQWHGLQVHFWLRQ7KHILQDOVHFWLRQLQFOXGHVDEULHIVXPPDU\.H\ZRUGVVWDWLRQDU\ KXPDQGHWHFWLRQ/L'$5PXOWLSOHYHUWLFDOO\DUUDQJHG'/L'$5VSRLQWFORXGDXWRHQFRGHU ,1752'8&7,21
7KH .DQDJDZD ,QVWLWXWH RI 7HFKQRORJ\ LV SURPRWLQJ D
UHVHDUFK SURMHFW ZKLFK LV FDOOHG WKH ³.$,7 0RELOLW\
5HVHDUFK&DPSXV3URMHFW´7KHSXUSRVHRIWKLVSURMHFWLVWRGHYHORS WHFKQRORJLHV IRURSHUDWLQJ VPDOO DXWRQRPRXV URERWFDUWV 6$5&  ZKLFK FDQ PRYH IUHHO\ RQ FDPSXV E\
LQWHJUDWLQJ VHYHUDO WHFKQRORJLHV VXFK DV $, VHQVLQJ DQGFRPPXQLFDWLRQWHFKQRORJLHVHWF
7KH6$5&VKRXOGKDYHFHUWDLQVHQVRUVIRUWKHUHFRJQLWLRQ
RI WKH RXWVLGH DQG LQVLGH VWDWXV 7KH FDPHUD DQG WKUHH
GLPHQVLRQDO ODVHU LPDJLQJ GHWHFWLRQ DQG UDQJLQJ '
/L'$5  DUH XVHG IRU REVWDFOH RU KXPDQ GHWHFWLRQ ZKLFKUHTXLUH VDIHW\ GXULQJ RSHUDWLRQV DQG VLPXOWDQHRXVORFDOL]DWLRQDQGPDSSLQJ 6/$0 WHFKQRORJLHV,Q WKLV SURMHFW LW LV DVVXPHG WKDWWKH 6$5&V DUH DSSOLHG
WRWKHGHOLYHU\URERWVIRUWKHGRFXPHQWVDQG OXJJDJH LQWKH
GD\WLPH WR SDWURO WKH URERWV DW PLGQLJKW DQG WR JDWKHU WKHIDOOHQOHDYHVLQWKHPRUQLQJ)RU WKLV SXUSRVH WKH 6$5&V DOZD\V HVWLPDWH WKHLU RZQSRVLWLRQ XVLQJ WKH 6/$0 WHFKQRORJLHV ZKLFK HQDEOHVVHQVLQJDURXQGWKHPYLDWKH'/LG$5V6/$0VDYHV
WKH SRVLWLRQ LQIRUPDWLRQ DQG VHQVLQJ GDWD DURXQG LW DV WKHPDSGDWD0RUHRYHULWDOZD\VFRPSDUHVLWVVHQVLQJGDWDDQG
PDSGDWDZKHUHLWKDVYLVLWHGWRHVWLPDWHLWVSRVLWLRQ
7KH 6$5&V UHTXLUH WKH IROORZLQJ IXQFWLRQV IRU WKHVH
SXUSRVHVz

7KH 6$5& PXVW DYRLG FROOLVLRQ ZLWK WKH REVWDFOHV
DQGKXPDQV
7KH6$5&PXVWNHHSDFHUWDLQGLVWDQFHIRUH[DPSOHDSSUR[LPDWHO\PIURPDKXPDQWRDYRLGLVVXHV .$,702%,/,7<5(6($5&+&$0386
 2EMHFWLYHRIWKH3URMHFW
7KH ³.$,7 PRELOLW\ UHVHDUFK FDPSXV´ LV HQYLVLRQHG DV D
SURMHFW LQ ZKLFK DXWRQRPRXV VPDOO FDUWV PRYH IUHHO\ E\WKHPVHOYHVLQWKHFDPSXVDQGPDNLQJOLIHDQGOHDUQLQJULFKDQGHQMR\DEOH
7KHFDPSXVKDVKLJK OHYHO LQIUDVWUXFWXUHZKLFK KDYH EHHQ
FUHDWHG XVLQJ FRPPXQLFDWLRQ DQG LQIRUPDWLRQ SURFHVVLQJWHFKQRORJLHV ,Q WKLV FDPSXV 6$5&V ZKLFK KDYH YDULRXV

7KHDFFXUDF\RIWKHORFDOL]DWLRQPXVWVDWLVI\WKDWWKH
6$5&FDQEHPRYHGDFFRUGLQJWRWKHSODQ

47

International Workshop on Informatics ( IWIN 2022 )UROHV PRYH DERXW HYHU\GD\ E\ WKHPVHOYHV H[DPSOHV DUHVKRZQ LQ )LJXUH   7KH VWXGHQWV DQG WHDFKHUV FDQ XVHZDWFKDQGEHJXDUGHGXVLQJWKHVH6$5&V$WWKHVDPHWLPH
WKH\FDQGHYHORSWKHLURZQ6$5&DQGH[SHULPHQWZLWKLWPDWHULDOVDUHHVWLPDWHGXVLQJWKHDWWHQXDWLRQUDWH7KHUHIRUH
WKHW\SHVRIGDWDREWDLQHGIURPD/L'$5DUHPRUHWKDQWKH
W\SHV RI GDWD REWDLQHG IURP LPDJHV FDSWXUHG E\ D FDPHUD
7KHKLJKSUHFLVLRQRIWKHDURXQGVWDWXVE\D/L'$5FDQEH
XVHGIRUWHUUDLQPHDVXUHPHQW5HFHQWO\WKLVWHFKQRORJ\KDV
EHHQ DSSOLHG IRU WKH ORFDOL]DWLRQ RI DXWRQRPRXV FDUWV
GHWHFWLRQ RI REVWDFOHV FROOLVLRQ DYRLGDQFH IRU YHKLFOHVDXWRQRPRXVGULYLQJHWF+RZHYHU LQ EDG ZHDWKHU FRQGLWLRQV WKH REWDLQDEOH
GLVWDQFHLVGHFUHDVHGRUWKHUHDUHIDOVHGHWHFWLRQV(VSHFLDOO\
LQ WKH FDVHV ZKHUH WKH OLJKW UHIOHFWLRQ FDQQRW EH UHFHLYHG
WKH GDWD FDQQRW EH JDWKHUHG IRU H[DPSOH LI WKH JODVV GRHVQRWUHIOHFWWKHOLJKW)XUWKHULWLVGLIILFXOWWRGHWHFWDPLUURURUDOXPLQXPVKHHWEHFDXVHLWGRHVQRWVFDWWHUWKHOLJKW>@
7KHUH DUH WZR W\SHV RI /L'$5V VROLG W\SH DQG URWDWLRQ
W\SH 7KH URWDWLRQ W\SH KDV D ZLGH UDQJH RI VHQVLQJ ZKLFKFUHDWHV PRGXOHV ZKLFK LUUDGLDWH WKH ODVHU OLJKW URWDWLQJ LW
+RZHYHULW KDV ORZGXUDELOLW\7KHVROLGW\SHKDVQRSK\VLFDOO\ PRYLQJ SDUWV +RZHYHU WKH GHWHFWLRQ UDQJH LV
QDUURZHU WKDQ WKH URWDWLRQ W\SH )XUWKHUPRUH /L'$5V FDQEH FODVVLILHG DV '/L'$5V DQG '/L'$5V $ '/LGDUKDV PXOWLSOH ODVHU OLJKWV LQ ZKLFK HDFK OLJKW KDV D GLIIHUHQWKHLJKW7KH SRLQWFORXG GDWD DUH VKRZQ LQ )LJXUHV  DQG )LJXUHVKRZVDQH[DPSOHRIWKHSRLQWFORXGGDWDREWDLQHG
IURPD'/L'$5ZKLFKFDQRQO\JDWKHUDVPDOODPRXQWRI
GDWD 7KLV PDNHV LW GLIILFXOW WR UHFRJQL]H DQ\ REMHFW DV D 6$5&$UFKLWHFWXUH
7KH 6$5& DUFKLWHFWXUH LV KLHUDUFKLFDO PRELOH LQWHOOLJHQFHDUFKLWHFWXUH ZKLFK LV LQWHJUDWHG XVLQJ VHYHUDO WHFKQRORJLHVVXFK DV $, VHQVLQJ UHFRJQL]LQJ DQG FRPPXQLFDWLRQWHFKQRORJLHVDVVKRZQ LQ )LJXUH$FFRUGLQJWRWKHVWDWXVDURXQG WKH 6$5& LW LV FRQWUROOHG SURSHUO\ E\ KLHUDUFKLFDO
PRELOH LQWHOOLJHQFH 7KLV DUFKLWHFWXUH LV FRPPRQ EHWZHHQ
6$5&VZKLFKKDYHPXOWLSOHUROHV /L'$5
7KH/L'$5LVWKHPRVWLPSRUWDQWDQGHVVHQWLDOVHQVRUIRUD
6$5&,WLVDNLQGRIVHQVRUZKLFKFDQILQGDQGPHDVXUHWKH
GLVWDQFH EHWZHHQ REMHFWV XVLQJ ODVHU OLJKW 7KH REMHFWV DUHGHWHFWHGXVLQJVFDWWHUHGOLJKWDQGWKHGLVWDQFHEHWZHHQWKHPLV PHDVXUHG XVLQJ WKH SKDVH GLIIHUHQFH 0RUHRYHU WKH&OHDQLQJ'HOLYHULQJ

6LJQDJH)LJXUH,PDJHRIH[DPSOHVRI6$5&)LJXUH([DPSOHRISRLQWFORXGGDWDXVLQJD'
/L'$5

ŐůŽďĂůďĞŚĂǀŝŽƌ
;ŶĂǀŝŐĂƚŝŽŶͿƉƌĞĚŝĐƚŝŽŶďĞŚĂǀŝŽƌ
;ŽďƐƚĂĐůĞƐĂŶĚŚƵŵĂŶƐ ĂǀŽŝĚĂŶĐĞͿƌĞĨůĞǆďĞŚĂǀŝŽƌ
;ĐŽůůŝƐŝŽŶĂǀŽŝĚĂŶĐĞͿ)LJXUH([DPSOHRISRLQWFORXGGDWDXVLQJD'
/L'$5

)LJXUH+LHUDUFKLFDOPRELOHLQWHOOLJHQFHDUFKLWHFWXUH

48

International Workshop on Informatics ( IWIN 2022 )KXPDQ IURP VROHO\ LWV VKDSH +RZHYHU WKH HTXLSPHQW FRVWIRU D '/L'$5 LV ORZ )LJXUH  VKRZV DQ H[DPSOH RI WKHSRLQWFORXGGDWDREWDLQHGIURPD'/L'$5WKDWFDQJDWKHUD ODUJHU DPRXQW RI GDWD IRU SXUSRVHV VXFK DV DXWRQRPRXVYHKLFOHVZKLFKUHTXLUHDKLJKHUDFFXUDF\7KH'/L'$5LV
5H7KHSRLQWFORXGGDWDIURPD'/L'$5GRHVQRWUHTXLUH
H[WHQVLYHFRPSXWHUUHVRXUFHV
7KHUHIRUH LQ WKLV VWXG\ LW LV DVVXPHG WKDW PXOWLSOH '
/L'$5V ZKLFK DUH YHUWLFDOO\ DUUDQJHG FDQ EH XVHG IRUVWDWLRQDU\KXPDQGHWHFWLRQ$ S\URHOHFWULF LQIUDUHG VHQVRU LV RQH RI WKH VHQVRUV ZKLFKFDQGHWHFWSHRSOH>@+RZHYHULWDVVXPHVWKDWKXPDQVZDON([DPSOHVRIVWDWLRQDU\KXPDQVDUHVKRZQLQ)LJXUH
LQ YDULRXV VLWXDWLRQV DQG GLVWDQFHV 7KLV W\SH RI PHWKRGPLJKW EH DEOH WR GHWHFW KXPDQV GHSHQGLQJ RQWKH VLWXDWLRQ,QWKHFDVHRIDXWKRUL]HGSHRSOHKDYLQJVPDUWSKRQHVDQGDQDSSOLFDWLRQ IRU DXWKHQWLFDWLRQ LV LQVWDOOHG LQ WKH SKRQH
KXPDQV FDQ EH HDVLO\ GHWHFWHG E\ EOXHWRRWK ORZ HQHUJ\%/(  WHFKQRORJ\ >@ +RZHYHU WKLV FDQ RQO\ EH IRUDXWKRUL]HGSHRSOHZLWKVPDUWSKRQHV$ KXPDQGHWHFWLRQ PHWKRGXVLQJ'SRLQWFORXGGDWDDQG
DGHHSOHDUQLQJWHFKQRORJ\ LVSURSRVHG>@7KLVDSSURDFKLVDSSOLHGLQZKLFKKXPDQFUDIWHGIHDWXUHVLVLQSXWWHGWRWKH
RQHFODVV FODVVLILFDWLRQ PRGHO WR LGHQWLI\ ZKHWKHU LW LV D
KXPDQ RU QRW +RZHYHU WKLV WHFKQRORJ\ DOVR DVVXPHG WKDWKXPDQVZDON,Q WKLV VWXG\ D VWDWLRQDU\ KXPDQ UHFRJQLWLRQ PHWKRG LV
SURSRVHG $ PHWKRG IRU GHWHFWLQJ KXPDQV XVLQJ ' SRLQWFORXG GDWD ZLWK /L'$5 >@ ZDV SURSRVHG LQ D SUHYLRXVVWXG\ E\ WKH DXWKRUV 7KH UHVXOWV RI WKH VWXG\ VKRZHG WKDWKLJKDFFXUDF\FRXOGEHREWDLQHGXQGHUFHUWDLQFRQGLWLRQV 5(/$7('678',(6+XPDQ GHWHFWLRQ DQG WUDFNLQJ KDYH EHHQ VWXGLHG
H[WHQVLYHO\ LQ UHFHQW \HDUV >@ 5*%FDPHUDV FDQ GHWHFWKXPDQV E\ UHFRJQL]LQJ WKH LPDJH GDWD >@ $GGLWLRQDOO\
5*%' FDPHUDV FDQ UHFRJQL]H KXPDQV >@ +RZHYHUWKHVH FDPHUDV UHTXLUH FHUWDLQ EULJKWQHVV FRQGLWLRQV
7KHUHIRUHWKH\FDQQRWEHXVHGRXWGRRUVDWQLJKW
7KHUH KDYH EHHQ YDULRXV VWXGLHV IRU WKH GHWHFWLRQ RI
KXPDQV XVLQJ /L'$5 >@ +XPDQV FDQ EH UHFRJQL]HGWKURXJK WKHLU VKDSH ZKLFK FDQ EH GHWHFWHG XVLQJ D '
/L'$5 DQG WLPH VHULHV REVHUYDWLRQ LQIRUPDWLRQ >@+RZHYHU WKH WLPH VHULHV LQIRUPDWLRQ RI VWDWLRQDU\ KXPDQVFDQQRWFKDQJHHYHU\WLPH2WKHUZLVHLWFDQEHDVVXPHGWKDWWKHKXPDQLVZDONLQJ>@ LVEDVHGRQ'/L'$5VZKLFKWUDFNWZROD\HUVZDLVWV
DQG NQHHV RI SHRSOH DQG FDQ GHWHFW DQG IROORZ PXOWLSOHZDONLQJKXPDQV,QWKLVVWXG\WKHLQGRRUHQYLURQPHQWDQGDZDONLQJ KXPDQ DUH DVVXPHG 7KHUHIRUH LW GRHV QRWUHFRJQL]H WKH KXPDQ XVLQJ WKH IHDWXUH RI KXPDQ VKDSHKRZHYHU LWUHFRJQL]HVWKHPRYLQJREMHFWVDUHKXPDQV7KH.DOPDQ ILOWHU LVXVHG IRUWKHSUHGLFWLRQRIZDONLQJKXPDQV
7KH VWDWLRQDU\ KXPDQV DUH WUHDWHG DV EDFNJURXQG LPDJHV
DQGGHOHWHG
7KHUHDUHDIHZVWXGLHV>@WKDWKDYHGHWHFWHGREMHFWVE\
VHQVRU IXVLRQ WHFKQRORJLHV ZKLFK XVH PXOWLSOH GLIIHUHQW
VHQVRUV IRU H[DPSOH D /L'$5 DQG FDPHUD 7KH LPDJH
SURFHVVLQJ WHFKQRORJ\ FDQ GHWHFW WKH  REMHFWV IURP WKH
LPDJH GDWD DQG WKH /L'$5 WHFKQRORJ\ FDQ PHDVXUH WKH
GLVWDQFHE\PDWFKLQJERWKVHWVRIGDWD
7KHVHNLQGVRIDSSURDFKHV PLJKWEHDEOHWRGHWHFWREMHFWV
RU KXPDQV ZLWK KLJK DFFXUDF\ +RZHYHU WKH FRVW LV KLJK7KHUHIRUH WKHLU LQVWDOODWLRQ LQ DXWRQRPRXV YHKLFOHV PLJKW
EH SRVVLEOH ,Q FRQWUDVWWKH FRVW LV WRR KLJK IRU LQVWDOODWLRQ
LQDXWRQRPRXVFDUWV 352326('0(7+2'
,QDSUHYLRXVVWXG\>@WKHDXWKRUVIRFXVHGRQWKHIHDWXUH
RI WKH DUP DQG ZDLVWWR GHWHFW KXPDQV 7KH UHVXOWV VKRZHG
 DFFXUDF\ IRU KXPDQ GHWHFWLRQ +RZHYHU DV SHRSOHFKDQJH WKHLU FORWKHV GHSHQGLQJ RQ WKH VHDVRQ WKHLU VKDSHVFDQIXUWKHUFKDQJHGHSHQGLQJRQWKHSUHVHQFHRUDEVHQFHRIOD\HUV RI FORWKHV 0RUHRYHU WKHUH DUH FRXQWOHVV W\SHV RIERG\ VKDSHV 7KHUHIRUH LW LV QHFHVVDU\ WR JDWKHUPRUH GDWDUHJDUGLQJWKHGLIIHUHQWERG\VKDSHV7KLVZLOOLPSO\WKDWWKH
GDWD ZKLFK PXVW EH OHDUQHG ZLOO LQFUHDVH DORQJ ZLWK WKHFDOFXODWLRQ FRVW $V D UHVXOW WKH DPELJXLW\ PLJKW LQFUHDVH
DQGWKHDFFXUDF\PLJKWGHFUHDVH
,WLVDVVXPHGWKDWWKHVKDSHRIWKHKXPDQVKLQZKLFKLVQRW
GHSHQGHQWRQWKH VHDVRQ FDQ REWDLQ KLJK DFFXUDF\ ZLWK WKH
VDPHYROXPHRIOHDUQLQJSRLQWFORXGGDWD+HQFH D KXPDQ GHWHFWLRQ PHWKRG ZLWK PXOWLSOH SRLQWFORXGGDWDIURP PXOWLSOH YHUWLFDOO\DUUDQJHG'/L'$5V LV
SURSRVHG DV VKRZQ LQ )LJXUH  ,I ERWK VHWV RI GDWD DUHKLJKO\ UHODWHG D KLJK DFFXUDF\ RI GHWHFWLRQ RI KXPDQV LV
H[SHFWHG+RZHYHULWPXVWEHLPSOHPHQWHGFDUHIXOO\

ZDLVW

/L'$5

/L'$5

VKLQV

NQHHV
DQNOHV)LJXUH0XOWLSOHYHUWLFDODUUDQJHG'/L'$5VV

)LJXUH([DPSOHGDWDRIS\URHOHFWULFLQIUDUHGVHQVRU

49

International Workshop on Informatics ( IWIN 2022 )FRUUHFW
5HVXOWRIHVWLPDWLRQKXPDQQRWKXPDQ

KXPDQ

$

%

&

'

QRW
KXPDQ

)LJXUH$FFXUDF\RIKXPDQGHWHFWLRQ

7KHUH DUH FHUWDLQ UHODWLRQV EHWZHHQ WKH DFWXDO FRUUHFWDQVZHU DQG WKH HVWLPDWLRQ UHVXOWV DV VKRZQ LQ )LJXUH *HQHUDOO\ UDWLRV $ DQG ' VKRXOG EH KLJK DV GHSLFWHG LQ
)LJXUH  7KHUH FDQ EH WZR LPSOHPHQWDWLRQ PHWKRGV 7KHILUVW PHWKRG LV WKDW ERWK WKH GDWD VHWV DUH OHDUQLQJ DW WKH
VDPHWLPH7KHVHFRQGPHWKRGLVWKDWERWKWKHGDWDVHWVDUH
OHDQLQJ LQGHSHQGHQWO\ 7KH ODWWHU PHWKRG LV PRUHVWUDLJKWIRUZDUGDQGUHTXLUHVOHVVFDOFXODWLRQFRVW+RZHYHU
$ PLJKW EH KLJK DQG ' PLJKW EH ORZ 7KHUHIRUH LW LV
VXJJHVWHGWKDWWKHIRUPHUPHWKRGEHDGRSWHG)LJXUH$QH[DPSOHRIVLPLODUREMHFWLQVLGHFDPSXV %DVLF([SHULPHQW
7KH H[SHULPHQWV ZHUH FDUULHG RXWWR VLPSO\ HVWLPDWH WKHDFFXUDF\RI KXPDQGHWHFWLRQWKURXJKWKHSURSRVHG PHWKRG
7KHUH DUH VLPLODU REMHFWV WR WKH KXPDQ VKLQ IRU LQVWDQFHYDULRXVREMHFWVZKLFKKDYHWZRIHHWZHUHIXUWKHUXWLOL]HGLQ
WKHH[SHULPHQW$QH[DPSOHRIDQREMHFWZLWKWZRIHHWVHHQRXWVLGH RI FDPSXV LV VKRZQ LQ )LJXUH  )LJXUH  VKRZV D
VLPLODU REMHFW LQVLGH WKH FDPSXV 7KH GDWD FROOHFWHG IURPWKHVHREMHFWVXVLQJ'/L'$5ZHUHXVHGDV LQFRUUHFWGDWD
7KH GDWD FROOHFWHG IURP KXPDQV XVLQJ '/L'$5 ZHUHFRQVLGHUHG DV WKH FRUUHFW GDWD $ ODUJH YROXPH RI GDWD LV
UHTXLUHG IRU OHDUQLQJ 7KH SRLQWFORXG GDWD ZHUH JDWKHUHG
FRYHULQJRIWKH/L'$5+RZHYHURQO\KDOIRIWKHGDWD
FDQ EH GHOHWHG DVWKH GDWD IRU OHDUQLQJ PLJKW EH V\PPHWULF
LQVKDSH)LUVWWKH HVVHQWLDO LQIRUPDWLRQ LV JDWKHUHG WKURXJK FHUWDLQH[SHULPHQWV7KHH[SHULPHQWHQYLURQPHQWZDVSUHSDUHGIRUJDWKHULQJ WKH FRUUHFW GDWD LH WKH VKLQ LQIRUPDWLRQ XVLQJ'/L'$5 DV GHSLFWHG LQ )LJXUH  7KH H[SHULPHQWDOFROODERUDWRUVZHUHDVNHGWRVWDQGRQWKHEHQFK7KHGDWDIRU
HDFKPHDVXUHRIGLVWDQFHLHDQGPZHUHFROOHFWHG
0RUHRYHUWKHH[SHULPHQWDOFROODERUDWRUVKDGWRVWDQGLQWKH
IURQW  GLDJRQDO WR WKH ULJKW DQG OHIW DQG WRZDUGV WKHEDFN6XEVHTXHQWO\WKHLQFRUUHFWGDWDZHUHDOVRJDWKHUHGDVVKRZQLQ)LJXUH)RU FRQILUPDWLRQ RI WKH SURSRVHG DSSURDFK H[SHULPHQWV
ZHUH FDUULHG RXW RQ D VPDOO VFDOH %RWK WKH FRUUHFW DQGLQFRUUHFWGDWDZHUHREWDLQHGIURPSHRSOHDQGREMHFWV
7KHWRWDODPRXQWRIGDWDZDVTXLWHVPDOO+RZHYHULWFRXOG
EHXVHGWRFRQILUPWKHSRVVLELOLW\ (;3(5,0(17 (9$/8$7,21

)LJXUH$QH[DPSOHRIVLPLODUREMHFWRXWVLGHFDPSXV (YDOXDWLRQ

50

International Workshop on Informatics ( IWIN 2022 )

7DEOH'HWHFWLRQUDWLRIRUHDFKDQJOH







7DEOHVKRZWKHGHWHFWLRQUDWLRIRUHDFKDQJOH7KHDQJOHWRWKH/L'$5ZDVFKDQJHGWR,QWKHFDVH
RIWKHGHWHFWLRQUDWLRZDV%HFDXVHLQWKHFDVHRILWLVGLIILFXOWWRGHWHFWWZRVKLQV7KHGDWDVKRZVXSDV
RQH SROH IRU WZR VKLQV 7KHVH UHVXOWV VKRZ LW LV GLIILFXOW WR
GHWHFWKXPDQVXVLQJRQO\WKHVKLQ ',6&866,21$1'&21&/86,21
7KHUHVXOWVRIWKLVVWXG\VKRZWKDWKXPDQVFDQEHGHWHFWHG
E\ WKH GDWD RI PRUH WKDQ WZR '/L'$5V 7KHUHIRUH WKHKXPDQ GHWHFWLRQ IXQFWLRQ VKRXOG SHHS LQWR WKH SRLQWFORXGGDWD RI '/L'$5 IRU 6/$0 ,W VKRXOG EH DEOH WR VHOHFWVXLWDEOH SRLQWFORXG GDWD IURP DOO GDWD )XWXUH VWXGLHV ZLOOXWLOL]HWZRNLQGVRIPRUHWKDQWZRGDWDRI'/L'$5V7KH
ILUVW RQH ZLOO EH '/L'$5V E\ GLIIHUHQW KHLJKWV DQG WKHVHFRQG RQH ZLOO EH '/L'$5V RQ PRUH WKDQ WZR URERWFDUWV

)LJXUH([SHULPHQWHQYLURQPHQWIRUJDWKHULQJFRUUHFWGDWD

5()(5(1&(6>@ 7 1LVKLGD ) 2KNDZD + 0L\DJDZD 0 2EDWD
³'HYHORSPHQW RI D 6HQVRU 6\VWHP IRU DQ 2XWGRRU
6HUYLFH 5RERW´ ,17(&+ 2SHQ $FFHVV 3XEOLVKHU>@ & 3UHPHELGD - &DUUHLUD - %DWLVWD 8 1XQHV
³3HGHVWULDQ GHWHFWLRQ FRPELQLQJ 5*% DQG GHQVH/,'$5 GDWD´ ,Q 3URFHHGLQJV RI WKH  ,(((56LQWHUQDWLRQDO FRQIHUHQFH RQ LQWHOOLJHQW URERWV DQG
V\VWHPV ,526 SS±
>@ $ *RQ]iOH] * 9LOODORQJD - ;X ' 9i]TXH] -
$PRUHV $0 /ySH] ³0XOWLYLHZ UDQGRP IRUHVW RI
ORFDO H[SHUWV FRPELQLQJ 5*% DQG /,'$5 GDWD IRU
SHGHVWULDQGHWHFWLRQ´,Q3URFHHGLQJVRIWKH,(((LQWHOOLJHQW YHKLFOHV V\PSRVLXP ,9  SS ±>@ 7/LQGHU6%UHXHUV%/HLEH.2$UUDV³2QPXOWLPRGDO SHRSOH WUDFNLQJ IURP PRELOH SODWIRUPV LQ YHU\FURZGHG DQG G\QDPLF HQYLURQPHQWV´ ,Q 3URFHHGLQJVRIWKH,(((LQWHUQDWLRQDOFRQIHUHQFHRQ URERWLFV
DQGDXWRPDWLRQ ,&5$ SS±
>@ 0 0XQDUR ( 0HQHJDWWL ³)DVW 5*%' SHRSOHWUDFNLQJ IRU VHUYLFH URERWV´ $XWRQRPRXV 5RERWV SS±
>@ / 6XQ = <DQ 60 0HOODGR 0 +DQKHLGH 7
'XFNHWW ³'2) SHGHVWULDQ WUDMHFWRU\ SUHGLFWLRQOHDUQHG IURP ORQJWHUP DXWRQRPRXV PRELOH URERW
GHSOR\PHQW GDWD´ ,Q SURFHHGLQJV RI WKH  ,(((
LQWHUQDWLRQDO FRQIHUHQFH RQ URERWLFV DQG DXWRPDWLRQ,&5$ 
>@ 0 +DVKLPRWR 7 .RQGD = %DL . 7DNDKDVKL/DVHUEDVHG WUDFNLQJ RI UDQGRPO\ PRYLQJ SHRSOH LQ
FURZGHG HQYLURQPHQWV  ,((( ,QWHUQDWLRQDO)LJXUH([DPSOHRILQFRUUHFWGDWD

 7KH WHVW GDWD DUH FODVVLILHG WR WZR FODVVHV XVLQJ WKHVXSSRUW YHFWRU PDFKLQH 690  >@ 7KH GDWD UHJDUGLQJ D
V\PPHWULFREMHFWRUKXPDQZHUHHYDOXDWHGE\XVLQJKDOIRISRLQWFORXG GDWD 7KH GHWHFWLRQ UDWLR ZDV DSSUR[LPDWHO\  ZKLFK LV QRW H[FHOOHQW 0RUHRYHU WKH GDWD ZHUHFODVVLILHGE\GLIIHUHQWGLVWDQFHVLHRUPZDVFODVVLILHG
DVFRUUHFWGDWDDQGRUPZDVFRQVLGHUHGDVLQFRUUHFWGDWD
IRUGHWHFWLRQRIRQO\VKLQGDWD7KHGHWHFWLRQUDWLRZDVDERXWZKLFKLVPXFKORZHUWKDQWKHSUHYLRXVRQH

51

International Workshop on Informatics ( IWIN 2022 )&RQIHUHQFH RQ $XWRPDWLRQ DQG /RJLVWLFV SS 

>@ 0/LDQJ%<DQJ<&KHQ5+X58UWDVXQ
³0XOWL7DVN0XOWL6HQVRU)XVLRQIRU'2EMHFW'HWHFWLRQ´3URFHHGLQJVRIWKH,(((&9)&RQIHUHQFH
RQ&RPSXWHU9LVLRQDQG3DWWHUQ5HFRJQLWLRQ &935 SS
>@ -.RFLü1-RYLþLüDQG9'UQGDUHYLü6HQVRUVDQG
6HQVRU)XVLRQLQ$XWRQRPRXV9HKLFOHVWK
7HOHFRPPXQLFDWLRQV)RUXP 7(/)25 SS
>@ -<XQ66/HH³+XPDQ0RYHPHQW'HWHFWLRQDQG,GHQWLILFDWLRQ8VLQJ3\URHOHFWULF,QIUDUHG6HQVRUV´
6HQVRUV9R1RSS
>@ ;/XR%6KHQ;*XR*/XR*:DQJ+XPDQWUDFNLQJXVLQJFHLOLQJS\URHOHFWULFLQIUDUHGVHQVRUV,(((,QWHUQDWLRQDO&RQIHUHQFHRQ&RQWURODQG$XWRPDWLRQSS

>@ <.RKDUDDQG01DND]DZD+XPDQ7UDFNLQJRI
6LQJOH/DVHU5DQJH)LQGHU8VLQJ)HDWXUHV([WUDFWHG
E\'HHS/HDUQLQJ7ZHOIWK,QWHUQDWLRQDO&RQIHUHQFHRQ0RELOH&RPSXWLQJDQG8ELTXLWRXV
1HWZRUN ,&08 SS
>@ <1DJDL<6DZDQR<7HUDVKLPD76X]XNLDQG5
.L\RKDUD$0HWKRGIRU'HWHFWLQJ+XPDQE\'/L'$5,(((,QWHUQDWLRQDO&RQIHUHQFHRQ&RQVXPHU(OHFWURQLFV ,&&( GRL
,&&(
>@ 690BSFOKWWSVJLWKXEFRPEHOORQHPDXUR
690BSFOBGHPRDFFHVVHG-XQ!
>@ =-&KRQJ%4LQ7%DQG\RSDGK\D\0+$QJ
( )UD]]ROL ' 5XV ³0DSSLQJ ZLWK V\QWKHWLF '/,'$5 LQ ' XUEDQ HQYLURQPHQW´ ,(((56,QWHUQDWLRQDO &RQIHUHQFH RQ ,QWHOOLJHQW 5RERWV DQG
6\VWHPV SS  GRL ,526



52

International Workshop on Informatics ( IWIN 2022 )'HYHORSPHQWDQG(YDOXDWLRQRID%URDGFDVWHU6XSSRUW0HWKRGXVLQJ
056WDPSLQGHJUHH,QWHUQHW/LYH%URDGFDVWLQJ
Yoshia Saito* and Kei Sato**

Graduate School of Software and Information Science, Iwate Prefectural University, Japany-saito@iwate-pu.ac.jp$EVWUDFW - In this study, we investigate the use of MR(Mixed Reality) stamp which supports broadcasters to
reduce communication errors from viewers to a broadcasterin 360-degree Internet live broadcasting. There is a problem
that the broadcaster is unable to grasp the viewer's POV
compared to the conventional broadcasting method. We
have confirmed that this problem could be reduced by
combining an equirectangular video and 2D stamp which
presents the viewer's interests in a simple image on thevideo. On the other hand, the 2D stamp system remains
three issues. The issues are (1) the stamp cannot be fixed on
the target object, (2) the broadcaster must check a PC
display to see the stamp, and (3) it is difficult to understanda position of the stamp in the real space.
In this study, we propose MR stamp which can bedisplayed and fixed on the real space, which enables thebroadcaster to check a holographic stamp on the real space
through an MR device. To realize our proposal, we
implemented the proposed system using Microsoft's
HoloLens 2 which is an HMD-type MR device. It can show
holograms in the real space by recognizing real space. We
also evaluated the effectiveness of the MR stamp comparedwith the 2D stamp and found that the MR stamp with thespatial audio solved the three issues of the 2D stamp.

the 360-degree Internet live broadcasting, it uses anomnidirectional camera which has a wide-angle lens or
multiple lens. It prevents the broadcaster to see what theviewers are watching by direction of the lens.
There are many studies about the role of gaze informationin the remote communication [1][2]. In the studies, it
concludes that the communicatee’s gaze information
indicates the target of interest or center of the topic. The
gaze information in the remote communication is similar tothe viewers’ POV in the 360-degree Internet live
broadcasting. The broadcaster sometimes cannot understandthe context of the viewers’ comments and it causes
communication errors between the broadcaster and theviewers.
To solve the problem of the communication errors, wehave proposed stamp functions in the 360-degree Internet
live broadcasting [3]. In this study, the stamp functions help
the broadcaster to understand what the viewers talk about
and find the object. However, it remains three issues in the
proposed system, which are (1) the stamp cannot be fixed on
the target object, (2) the broadcaster must check a PC
display to see the stamp, and (3) it is difficult to understanda position of the stamp in the real space. These issues should
be solved to realize smooth communication between the
broadcaster and the viewers.
In this study, we propose MR stamp which is a new stampfunction using MR and can be displayed and fixed on thereal space. It enables the broadcaster to check a holographic
stamp on the real space through an MR device. The MR
stamp can solve the issues of the previous stamp function.
The contributions of this paper are summarized asfollows:.H\ZRUGV: 360-degree Internet live broadcasting, Mixed
Reality, MR stamp, Broadcaster support method

 ,1752'8&7,21
YouTube started a 360-degree Internet live broadcasting
service from 2016 and it enables anyone to easily use the360-degree Internet live broadcasting service now. The 360degree Internet live broadcasting is a service which
combines the Internet live broadcasting with 360-degree
videos using an omnidirectional camera. In the 360-degree
Internet live broadcasting, a broadcaster can provide a 360degree video to viewers in real time without care about theview angle of the camera. The viewers can change the pointof view (POV) according to their own interests andcommunicate with the broadcaster using text chat.
The 360-degree Internet broadcasting, however, has a lotof new problems. One of the problems is that thebroadcaster cannot be aware of the viewers’ POV. In theconventional Internet live broadcasting, it uses a web
camera which has a single lens and the single lens definitely
shows the viewers’ POV. The broadcaster can see what they
are watching by direction of the lens. On the other hand, inyWe proposed a new broadcaster support method called
MR stamp in 360-degree Internet live broadcasting.

y

We developed and evaluated a prototype system of the
MR stamp using HoloLens 2.

y

We clarified that the MR stamp enabled thebroadcaster to find a target object in a relatively shorttime and spatial audio could help the broadcaster tofind the MR stamp.

The rest of this paper is organized as follows. Section 2
describes our previous work about stamp functions in the360-degree Internet live broadcasting. Section 3 describes
an overview of the proposed system and a use case of the
MR stamp. Section 4 describes implementation of the
proposed system. Section 5 describes a first evaluation

53

International Workshop on Informatics ( IWIN 2022 )experiment to clarify the effects of the proposed system andreveal its problem. Section 6 describes a second evaluation
experiment improving the proposed system. Section 7
summarizes this study.

 35(9,286678'<
We proposed have stamp functions in 360-degree Internet
live broadcasting to support communication between the
broadcaster and the viewers. In this section, we explain thestamp functions in the previous study and its known issues.

/RRN VWDPS*R VWDPS

Figure 1: Stamps in the previous study 6WDPSIXQFWLRQVLQGHJUHH,QWHUQHW
OLYHEURDGFDVWLQJGHJUHH9LGHRLQVSKHULFDOIRUPDW

There are two stamps which are “Look” and “Go” stampin the previous study as shown in Fig. 1. Figure 2 shows auser interface of the viewer in the previous study. The
viewers can watch the 360-degree live video in sphericalformat and change the POV as they want. The viewers arealso able to use Look and Go stamp by selecting the kind ofstamp and clicking on the video. The stamps are shown insame place of the video on the user interface of the viewers
and the broadcaster. Figure 3 shows a user interface of thebroadcaster. The 360-degree video is displayed inequirectangular format. The broadcaster can check thestamps which are sent from the viewers without changing
the POV.
We evaluated the previous stamp function and found theiradvantages. The stamp function improved easiness ofcommunication between the broadcaster and the viewers
compared with the case that the stamp was not used. The
broadcaster could easily understand what the viewer talked
about and found the object. Moreover, the stamp function
increased the frequency of communication between the
broadcaster and the viewers.

6WDPSV

6HOHFWLRQRIVWDPSV

Figure 2: User Interface of viewer in the previous study
GHJUHH9LGHRLQHTXLUHFWDQJXODUIRUPDW

6WDPSV .QRZQLVVXHV
Although the previous study has advantages, there arethree issues as follows.
(1) The stamp cannot be fixed on the target object.
A stamp only has information of a direction from theomnidirectional camera at a particular time. Therefore,
the stamp cannot be fixed on the target object and it
causes positional shift of the stamp when theomnidirectional camera is moved.

Figure 3: User Interface of broadcaster in theprevious study
it is inconvenient not to be able to use a hand whichkeeps the laptop PC.

(2) The broadcaster must check a PC display to see thestamp.
In the previous study, the broadcaster has to carry alaptop PC and check stamps from the viewers throughthe display. The check of the laptop PC display
consumes time and prevents smooth communication
between the broadcaster and the viewers. Besides that,
it is dangerous to see the laptop PC while walking and(3) It is difficult to understand a position of the stamp in
the real space.
The stamps are shown on the 360-degree video in
equirectangular format. When a stamp is appeared on
the video, the broadcaster must find a position of thestamp in the real space. The distortion of the 360degree video in equirectangular format makes it
difficult to find the direction of the stamp.

54

International Workshop on Informatics ( IWIN 2022 ) 352326('6<67(0 8VH&DVH

The MR stamp aims to solve the issues of the previous
study using MR and realize smooth communication between
the broadcaster and the viewers.

In this study, we suppose that an omnidirectional camera
is fixed on an arbitrary position and 360-degree Internet live
broadcasting is performed indoors. The reason is that theusage environment should be simple for the first step of thisstudy. We also suppose it is used at the showroom and theexhibition.
Figure 5 shows a use case of the proposed system. The
viewers can send a MR stamp to the broadcaster if they want
to see a particular object in a showroom. The broadcastercan understand the request from the viewers easier thanwhen only comments are used. Since 2000, online virtual
events have been increasing because of COVID-19. In the
online virtual events, it is said that more opportunity for real
two-way communication between the broadcaster andviewers [6]. In online conferences, it is reported that it
cannot have smooth relationship with each other between
participants [7]. The MR stamp would realize smooth twoway communication between the broadcaster and theviewers in various online virtual events.

 (IIHFWLYHQHVVRI05
MR is a technology which displays holograms of virtual
objects in the real space and the users can interact with theholograms. Several researches show effectiveness of MR inremote communication between users.
Lee [4] developed a MR remote collaboration system
which shared 360-degree live video. In this system, ahologram of remote user’s hand is displayed on the real
space through the MR device. The hand gestures by thehologram help to understand each other’s focus and improvetheir communication. Johnson [5] studied the effect of MR
guidance. An experiment was conducted to understand how
providing explicit spatial information in collaborative MR
environment. From the experiment result showed the MR
guidance realized effective referencing through deixis.
From the related work, the reduction effect ofcommunication errors can be also expected by introducing
the MR technology for the stamp function in 360-degree
Internet live broadcasting.

ůŝĐĞ͗/ůŝŬĞŝƚ͘Žď͗ / ĂŐƌĞĞ͘ĂƌŽů͗/ƚΖƐĂůƵǆƵƌǇŽŶĞ͘sŝĞǁĞƌƐ

 6\VWHP0RGHO%URDGFDVWHU

056WDPS&RPPHQWVƌŽĂĚĐĂƐƚĞƌ͛Ɛ ƐŝŐŚƚ ƚŚƌŽƵŐŚ ƚŚĞDZĚĞǀŝĐĞ

ƌŽĂĚĐĂƐƚĞƌǁŝƚŚĂDZĚĞǀŝĐĞ

Figure 5: A use case of the proposed system

Figure 4 shows a model of the proposed system. The
proposed system is included in the existed 360-degree
Internet live broadcasting system. A broadcaster provides
360-degree live video to viewers using the 360-degree
Internet live broadcasting system. The viewers can send a
2D stamp which is implemented in the previous study to thebroadcaster. The proposed system receives 2D position
information of the stamp and transforms it to 3D position
information. The proposed system displays a MR stamp
using the 3D position information and the broadcaster cancheck the MR stamp in the real space through a MR device.
The proposed system can solve three issues in theprevious section. The first issue which is that “The stamp
cannot be fixed on the target object” can be solved by fixing
the stamp on the real space using MR. The second issue
which is that “The broadcaster must check a PC display to
see the stamp” can be solved by using a MR device which isa type of head-mounted display. The third issue which isthat “It is difficult to understand a position of the stamp in
the real space” can be solved by directly displaying thestamp on the target object in the real space using MR.
GHJUHH
OLYHYLGHRdŚŝƐŝƐĂǀĞƌǇƉŽƉƵůĂƌďĞĚ͊/ĂŵĐƵƌŝŽƵƐĂďŽƵƚƚŚŝƐďĞĚ

GHJUHH,QWHUQHW
OLYHEURDGFDVWLQJ
V\VWHP3URSRVHG
6\VWHP

We implemented a prototype system of the MR stamp
using Hololens2. In this section, we describe the architectureof the prototype system and its main application.



6\VWHP$UFKLWHFWXUH

The prototype of the proposed system is based on 360degree Internet live broadcasting system of the previous
study. Figure 6 shows the architecture of the prototype
system. The red square shows new implementation in thisstudy and the other parts are diverted from the previous
study. A broadcaster can start 360-degree Internet live
broadcasting using the client for broadcaster on a web
browser. The 360-degree Internet live broadcasting server
distributes it to viewers. The viewers can watch the 360degree live video and send 2D stamp and comments in thesame manner as the previous study. The stamp/comment
server forwards it to all clients for viewers and a MR deviceof the broadcaster. We use Microsoft HoloLens 2 as the MR
device. In the HoloLens2, the MR stamp application isrunning. The application presents the MR stamp andcomments to the broadcaster. The broadcaster can check the
MR stamp in the real space through the HoloLens 2.

GHJUHH
OLYHYLGHR

' 6WDPS&RPPHQWV,03/(0(17$7,21

9LHZHUV

Figure 4: A model of the proposed system

55

International Workshop on Informatics ( IWIN 2022 )KŵŶŝĚŝƌĞĐƚŝŽŶĂůĂŵĞƌĂ
ϯϲϬͲĚĞŐƌĞĞůŝǀĞǀŝĚĞŽ

ϯϲϬͲĚĞŐƌĞĞ/ŶƚĞƌŶĞƚ
ůŝĞŶƚ ĨŽƌϯϲϬͲĚĞŐƌĞĞ ůŝǀĞďƌŽĂĚĐĂƐƚŝŶŐƐĞƌǀĞƌ ϯϲϬͲĚĞŐƌĞĞƌŽĂĚĐĂƐƚĞƌ
ůŝǀĞǀŝĚĞŽ
ůŝǀĞǀŝĚĞŽ
EŽĚĞ͘ũƐ
;tďƌŽǁƐĞƌͿƌŽĂĚĐĂƐƚĞƌ

05VWDPS&RPPHQWV

DZ ĚĞǀŝĐĞ;,ŽůŽ>ĞŶƐϮͿh/ĨŽƌsŝĞǁĞƌƐ

<ƵƌĞŶƚŽ DĞĚŝĂ^ĞƌǀĞƌ

h/ĨŽƌƌŽĂĚĐĂƐƚĞƌ

ϮƐƚĂŵƉ͕ŽŵŵĞŶƚƐ

DZ ^ƚĂŵƉƉƉůŝĐĂƚŝŽŶ

^ƚĂŵƉͬŽŵŵĞŶƚ
^ĞƌǀĞƌ
^ƚĂŵƉĂŶĚŽŵŵĞŶƚ
^ĞŶĚͬZĞĐǀ
&ƵŶĐƚŝŽŶ

ůŝĞŶƚĨŽƌ sŝĞǁĞƌƐ
䠄tďƌŽǁƐĞƌ䠅Ϯ ƐƚĂŵƉ͕ŽŵŵĞŶƚƐ
ϮƐƚĂŵƉ͕ŽŵŵĞŶƚƐ

^ƚĂŵƉĂŶĚŽŵŵĞŶƚ
^ĞŶĚͬZĞĐǀ
&ƵŶĐƚŝŽŶ

sŝĞǁĞƌƐ

Figure 6: Architecture of the prototype system 05VWDPSDSSOLFDWLRQThe MR stamp application receives 2D position
information of the stamp and needs to transform it to 3D
position information for the MR stamp. To realize thecoordinate transformation, we use a Raycast function in
Unity, which irradiates a 3D ray from an origin point to atarget direction and detects intersecting collisions. The
origin point is the coordinates of the omnidirectional camera.
The target direction can be given by the coordinates of thestamp. The HoloLens 2 have a function of spatial mapping.
The spatial mapping provides a detailed representation ofreal space surfaces in the environment around HoloLens 2.
The raycast detects the intersecting collisions with the real
space surfaces and returns the 3D coordinates. The MR
stamp is displayed on the 3D coordinates.
Figure 7 shows the example of the MR stamp. The MR
stamp is shown as a 3D square frame so that it is easy tocheck the target object from the broadcaster. The MR stamp
disappear after the period of time (10 seconds in this
implementation).
The comments from the viewers are displayed on acomment window as shown in Fig. 8. The comment window
tracks the broadcaster’s sight.

Figure 8: Comment window through HoloLens 2

 ),567(9$/8$7,21
We conducted an experiment to evaluate effectiveness of
the MR stamp using the implemented prototype system. The
purpose of the evaluation is to check whether the threeissues of the previous study can be solved or not by using
the MR stamp.

 (QYLURQPHQWDQG3URFHGXUH
We compare the prototype system (MR stamp system)with the stamp system of the previous study (2D stamp
system). In the experiment, A broadcaster performs 360degree Internet live broadcasting and two viewers watch thebroadcasting. In the room of the broadcaster, there arevarious objects. The viewers talk about an object in the
broadcaster’s room using the MR/2D stamp and comments.
The broadcaster looks for the target object andcommunicates with the viewers. The evaluation items are as
follows; (1) time to find the target object, (2) number ofcommunication errors, and (3) subjective easiness to find thetarget object.

Figure 7: An example of the MR stamp56

International Workshop on Informatics ( IWIN 2022 )The experiments were conducted 4 times. There were one
broadcaster and two viewers per time and the participants
were students in our university. The broadcasting time was
30 minutes. Figure 9 shows the procedure of the experiment.
At first, a viewer sends a stamp to the broadcaster. The
target object and the viewer who performs the task arepredetermined by the task instruction. The broadcaster looks
for the target object referring to the stamp. After the target
object is found, the broadcaster confirms whether the object
is correct or not by speaking to the viewer. The viewer
replies whether it is correct or not. If it is not correct, thebroadcaster continues to find the target object. The searchtask with stamp is performed 2 times. At last, we ask thebroadcaster how easy to find the target object in 5-point
scale by a questionnaire. This procedure was performed forboth the MR stamp system and the 2D stamp system. The
order of the MR stamp and 2D stamp system was random tokeep fairness.
Figure 10 shows the target objects used in the experiment.
Target A and C are comprehensible ones to find because
there is nothing around the object (hereafter, single object).
Target B and D are mistakable ones to find because there areseveral similar objects around the object (hereafter, multiple
objects). In addition to these target objects, there are several
dummy objects in the broadcaster’s room. Figure 11 shows
the location of all objects in the broadcaster’s room.



GHVN

GHVN

GHVN




GHVN



ʁ7DUJHW$ VLQJOHREMHFWʁ'XPP\ VWXIIHGDQLPDO
ʁ7DUJHW% PXOWLSOHREMHFWV ʁ'XPP\ SLOHRIERRNV
ʁ7DUJHW& VLQJOHREMHFW ʁ'XPP\ WRROFDVH
ʁ7DUJHW' PXOWLSOHREMHFWV  ʁ'XPP\ KHOPHW

Figure 11: Location of all objects in the
broadcaster’s room )LUVW(YDOXDWLRQ5HVXOWVTable 1 shows evaluation result of the first evaluation.
There are the results of 4 broadcasters with 2D and MR
stamp system. In terms of Target, “Single” means single
object such as Target A and C and “Multiple” means
multiple objects such as Target B and D. “Time to find”
denotes time from when a stamp was displayed to when the
broadcaster found the correct target object. “Errors” denotes
number of times which the broadcaster indicated incorrect
objects.
The average of time to find the target object in the 2D
stamp system takes 11.25 seconds in case of the single
object and 20.25 seconds in case of the multiple objects. The
total of the average time in the 2D stamp system is 15.75
seconds. In the MR stamp system, it takes 18 seconds incase of the single object and 13.25 seconds in case of themultiple objects. The total of the average time in the MR
stamp system is 15.625 seconds. The average of number of
errors in the 2D stamp system shows 0.25 and 0.5 in the
single and multiple objects respectively. The total of theaverage number of errors in the 2D stamp system is 0.375.
The average of number of errors in the MR stamp systemshows 0.25 and 0.25 in the single and multiple objects
respectively. The total of the average number of errors in the
MR stamp system is 0.375. We conducted a Student’s t-test
for the results of the total average and there is no significant
difference both in the average time to find (p = 0.98) and theaverage number of errors (p = 0.63) between the 2D and MR
stamp systems.
Table 2 shows the result of the questionnaire which isabout subjective easiness to find the target object in 5-point
scale. The average points of the 2D stamp system are 4.5
and 3.25 in the single and multiple objects respectively. The
total average in the 2D stamp system is 3.85. The average
points of the MR stamp system are 4.25 and 3.75 in the
single and multiple objects respectively. The total average in
the MR stamp system is 4. We conducted a Student’s t-test
for the results of the total average and there is no significant
difference about the subjective easiness to find the target
object (p = 0.83).

 $IWHU WKHWDUJHWREMHFWLV IRXQG WKHEURDGFDVWHUVD\Vn,VWKLVZKDW\RXWDONDERXW"|
 ,IWUXHWKHYLHZHUVHQGVDFRPPHQWn<(6| ,IIDOVHWKHYLHZHUVHQGVDFRPPHQWn12|DQGWKHEURDGFDVWHUFRQWLQXHVWRILQGWKHWDUJHWREMHFW ,WLV ILQLVKHG DIWHU WZR WDUJHW REMHFWV DUH IRXQG

ᶇ 4XHVWLRQQDLUH
7KHEURDGFDVWHUHYDOXDWHVVXEMHFWLYHHDVLQHVVWRILQGWKHWDUJHW
REMHFW LQSRLQWVFDOH

Figure 9: Procedure of the experiment

7DUJHW&ʁ&RIIHHPDNHU&DPHUD
DQG3&



ᶆ %URDGFDVWHUORRNVIRUWKHWDUJHWREMHFWUHIHUULQJWRWKHVWDPS

7DUJHW$ʁ%ODFNEDFNSDFN



GHVN

 7KHWDUJHWREMHFWDQGWKHVHQGLQJYLHZHUDUHSUHGHWHUPLQHG

z





ᶅ 9LHZHUVHQGVDVWDPSWLPHVFKDQJLQJWKHV\VWHP

GHVN

7DUJHW%ʁ%OXHODEHOHGIROGHU

7DUJHW'ʁ6WXIIHGZKLWHWLJHURQDFDUGERDUGER[

Figure 10: Target objects

57International Workshop on Informatics ( IWIN 2022 )find the MR stamp easily and conduct a second evaluation
using the improved prototype system.

Table 1: Evaluation result of the first evaluation%URDGFDVWHU

6\VWHP

7DUJHW

'6WDPS$
056WDPS
'6WDPS%
056WDPS
'6WDPS&
056WDPS
'6WDPS'
056WDPS

'6WDPS$YHUDJH
056WDPS

7LPHWRILQG(UURUV

VHF

1RRIWLPHV

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH 6\VWHP,PSURYHPHQW

6LQJOH

0XOWLSOH





7RWDO






In order to help the broadcaster to find the MR stamp, we
introduce spatial audio into the MR stamp. The spatial audio
enables the broadcaster to perceive sound all around his/herand be aware of the location of the sound source. Titus [8]studies the effectiveness of spatial audio for finding thelocation of a maker. He finds that the spatial audio helps theuser with a head-mounted display to find a rough location of
the maker although it is not enough to specify the exact
location of the maker. In our MR stamp system, thebroadcaster would find the MR stamp if he/she can be awareof a rough location of the stamp.
We implemented the spatial audio to the MR stamp using
a spatial audio function of MRTK (Mixed Reality Toolkit).
When the MR stamp is displayed, the spatial sound isgenerated from the location of the MR stamp. The
broadcaster can hear the spatial sound through the
HoloLens2 and find the MR stamp quickly.



6LQJOH

0XOWLSOH





7RWDO



 6HFRQG(YDOXDWLRQ5HVXOWVTable 2: Questionnaire result of the first evaluation

We conducted a second evaluation using the improved
prototype system with spatial audio. The environment andthe procedure are same as the first evaluation.
Table 3 shows evaluation result of the second evaluation.
The average of time to find the target object in the 2D stamp
system takes 17.5 seconds in case of the single object and19.75 seconds in case of the multiple objects. The total of
the average time in the 2D stamp system is 18.625 seconds.
In the MR stamp system, it takes 5 seconds in case of thesingle object and 5 seconds in case of the multiple objects.
The total of the average time in the MR stamp system isabout 5 seconds. The average of number of errors in the 2D
stamp system shows 0.25 and 0.5 in the single and multiple
objects respectively. The total of the average number of
errors in the 2D stamp system is 0.375. The average of
number of errors in the MR stamp system shows 0 and 0 in
the single and multiple objects respectively. The total of theaverage number of errors in the MR stamp system is 0. We
conducted a Student’s t-test for the results of the total
average. There is a significant difference in the average time
to find (p = 0.01) and a significant trend in the average
number of errors (p = 0.08) between the 2D and MR stamp
systems.
Table 4 shows the result of the questionnaire which isabout subjective easiness to find the target object in 5-point
scale. The average points of the 2D stamp system are 3 and2.25 in the single and multiple objects respectively. The
total average in the 2D stamp system is 2.625. The average
points of the MR stamp system are 4.75 and 4.5 in the single
and multiple objects respectively. The total average in the
MR stamp system is 4.625. We conducted a Student’s t-test
for the results of the total average and there is a significant

6XEMHFWLYHHDVLQHVVWR
6\VWHP

7DUJHW

ILQGWKHWDUJHWREMHFWSRLQWVFDOH

6LQJOH DYJ'6WDPS

056WDPS0XOWLSOH DYJ

7RWDODYHUDJH

6LQJOH DYJ

0XOWLSOH DYJ

7RWDODYHUDJHFrom these results, we didn’t find the effectiveness in the
MR stamp system. In the free descriptive answer of thequestionnaire, there were several same answers that “It wasdifficult to find the MR stamp”. If the MR stamp can help
the broadcaster to find the target object, the broadcastercannot find the location of the MR stamp itself in theexperiment. We expect that an additional function to findthe location of the MR stamp improves the effectiveness of
the MR stamp system.

 6(&21'(9$/8$7,21:,7+6<67(0,03529(0(17
In the first evaluation, there was no difference about theeffectiveness between the 2D stamp system and the MR
stamp system. Since the prototype system of the MR stamp
had a problem that it is difficult to find the location of the
MR stamp itself, we try to improve the prototype system to58International Workshop on Informatics ( IWIN 2022 )through an MR device. We implemented a prototype systemof the MR stamp with special audio. In the evaluation of thisstudy, we verified the effectiveness of the MR stamp
compared with the 2D stamp using the prototype system.
We found that the MR stamp with the spatial audio made iteasier for the broadcaster to intuitively grasp the location of
the stamp. In addition, a comparison with the 2D stamp
system revealed that there was a significant difference in thetime to find the target object and the subjective easiness tofind the target object. There was also a significant trend in
the average number of errors between the 2D and MR stamp
systems. From the result, we found that the MR stamp could
solve the issues of the 2D stamp. In the future, it isnecessary to improve the accuracy of the stamp display.

Table 3: Evaluation result of the second evaluation%URDGFDVWHU

6\VWHP
'6WDPS$
056WDPS
'6WDPS%
056WDPS
'6WDPS&
056WDPS
'6WDPS'
056WDPS

7DUJHW$YHUDJH
056WDPS(UURUV

VHF

1RRIWLPHV

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH

6LQJOH

0XOWLSOH



6LQJOH'6WDPS

7LPHWRILQG

5()(5(1&(6[1] Roel Vertegaal: The GAZE groupware system:
mediating joint attention in multiparty communicationand collaboration, CHI '99 Proceedings of the SIGCHI
conference on Human Factors in Computing Systems,
pp.294-301 (1999).
[2] David M. Grayson, Andrew F. Monk: Are you looking
at me? Eye contact and desktop video conferencing,
ACM Transactions on Computer-Human Interaction(TOCHI) Volume 10 Issue 3, September 2003, pp.221243 (2003).
[3] Yoshia Saito, Aoi Kuzumaki, Yasuhiro Yahata and Dai
Nishioka: Development of a Communication Support
System with Stamp Functions in 360-degree Internet
Live Broadcasting, Proc. of DICOMO2019, pp. 895900 (2019) (in Japanese).
[4] Gun A. Lee, Theophilus Teo, Seungwon Kim andMark Billinghurst: Mixed Reality Collaboration
through Sharing a Live Panorama, Proc. of SA’17,
pp.1-4 (2017).
[5] Janet G Johnson, Danilo Gasques, Tommy Sharkey,
Evan Schmitz and Nadir Weibel: Do You Really Needto Know Where “That” Is? Enhancing Support for
Referencing in Collaborative Mixed Reality
Environments, Proc. of CHI '21, pp. 1-14 (2021).
[6]
StateOfVirtualEvents2021,
availablefrom<https://s3.amazonaws.com/media.mediapost.com/uplo
ads/StateOfVirtualEvents2021.pdf> (accessed 2022-610).
[7] Chris Misa, Dennis Guse, Oliver Hohlfeld,
Ramakrishnan Durairajan, Anna Sperotto, Alberto
Dainotti and Reza Rejaie: Lessons Learned Organizing
the PAM 2020 Virtual Conference, ACM SIGCOMM
Computer Communication Review, Vol.50, Issue 3,
pp.46-54 (2020).
[8] Titus J. J. Tang and Wai Ho Li: An Assistive EyeWear
Prototype that interactively converts 3D Object
Locations into Spatial Audio, Proc. of ISWC’14,
pp.119-126 (2014).






0XOWLSOH





7RWDO



6LQJOH

0XOWLSOH





7RWDO





Table 4: Questionnaire result of the second evaluation
6XEMHFWLYHHDVLQHVVWRILQG
6\VWHP

7DUJHW

WKHWDUJHWREMHFWSRLQWVFDOH

6LQJOH DYJ'6WDPS

056WDPS0XOWLSOH DYJ

7RWDODYHUDJH

6LQJOH DYJ

0XOWLSOH DYJ

7RWDODYHUDJH



difference about the subjective easiness to find the target
object (p = 0.0007).
From these results, we can find the effectiveness in the
MR stamp system with special audio. In terms of the threeissues of the previous study, the first issue which is that
“The stamp cannot be fixed on the target object” is solved
because the number of the errors decreases in the proposedsystem. The second issue which is that “The broadcastermust check a PC display to see the stamp” is solved because
the time to find the target object decreases. The third issue
which is that “It is difficult to understand a position of thestamp in the real space” is solved because both the numberof the errors and the time to find the target object decreases.

 &21&/86,21
In this study, we proposed MR stamp which can bedisplayed and fixed on the real space, which enables thebroadcaster to check a holographic stamp on the real space

59

International Workshop on Informatics ( IWIN 2022 )Object Recognition Method Using Locus of Gestures Detected by YOLOv5
Tsukasa Kudo††

Faculty of Informatics, Shizuoka Institute of Science and Technology, Japan
kudo.tsukasa@sist.ac.jp

Abstract - To recognize relatively small objects in an image, it is ﬁrst necessary to perform object detection. In recent years, research has been actively conducted to utilize
deep learning to simultaneously perform object detection andrecognition, in which real-time object recognition has been
also enabled by such as You Only Look Once (YOLO). However, they are based on deep learning, there are application
issues that training data for all targets must be prepared fortraining the model. In this study, to detect target objects, I
propose a method to detect the locus indicated by gestures invideos using YOLOv5, which uses only a single object suchas a hand, to extract the target area. In this method, to improve
the accuracy of the target area, the false object detection results are eliminated and the locus is corrected, by using themedian and moving average of consecutive video frames respectively. Furthermore, it is shown that simple object recognition methods such as template matching can be used by detecting the size and tilt of the target based on the detected areaby this method.
Keywords: YOLOv5, Deep learning, Object detection, Gesture recognition, Template matching

1 INTRODUCTION
In recent years, object recognition for videos and images
has been actively studied, and its applications are expandingin various ﬁelds such as immigration control by face recognition and automatic car driving. However, when the area ofthe target (hereinafter, target area) in the image is small, it isnecessary to perform object detection ﬁrstly to specify the target area before object recognition. For example, in the caseof face recognition, face detection is performed using Haarlike features, and then face recognition is performed on thedetected target area [1].
Using deep learning, various methods have been proposed
to efﬁciently perform both object detection and recognition.
For example, You Only Look Once (YOLO) detects the target
as a bounding box and simultaneously recognizes the target[2]. Furthermore, it has been shown that even videos can beprocessed in real-time [3].
However, these methods require the preparation of modeltraining data for each target object, which is a signiﬁcant burden when the types of target objects are large. On the other
hand, for still objects that can be photographed from a speciﬁc
direction, such as the cover of a book, object recognition canbe performed by a simple method such as template matchingif the target area can be identiﬁed.
In this paper, I propose a method to extract the target areafrom the gesture in videos by using YOLO for object detection. The gesture is performed so that the target area is a

closed area surrounded by the locus of the gesture, and thetarget area is extracted by using this locus. The important
point is that, since gestures can be performed by a certain partof the body such as a hand, only one type of training data
is required for a variety of object detections in this method.
Also, while other object detection methods extract the target
area as a bounding box, this method can extract the target areaaccording to the shape of the target object. That is, for example, when performing template matching, the method canestimate and correct the target’s tilt or suppress the inﬂuencesof background areas.
However, since the above locus is created by continuously
detecting a speciﬁc part of the gesture in a video, it causes
some challenges. The locus contains wrong points due tofalse detections (hereafter, noises); there may be a double andmissing part of the locus at the beginning and end of gestures. To address these challenges, this method eliminates
these noises and corrects the locus by utilizing the medianand moving average of the locus points. And, I show that thismethod can extract the target area through experiments.
Furthermore, in order to investigate the effects of size speciﬁcation and tilt correction on recognition accuracy in template matching, which is one of the simpliﬁed object recognition methods, I evaluated the improvement of recognition
accuracy for books. The purpose of this evaluation was toclarify the required accuracy in the target area extraction. The
results show that when the vicinity of the target region is extracted, with a size error of less than 10% and a tilt error ofless than 10◦ , the detection is correct.
The remainder of this paper is organized as follows. Section 2 presents related works and the aim of this study, andSec. 3 proposes a target area extraction method based on gestures in a video. Section 4 shows the implementation and experimental results of target area extraction, and Sec. 5 evaluates the accuracy of the proposed method and its effectivenessfor template matching. Section 6 discusses on the evaluation
results, and Sec. 7 concludes this paper.

2 RELATED WORKS AND AIM OF THIS
STUDY
In recent years, the effectiveness of object recognition basedon deep learning for images and videos has been widely recognized and applied to various ﬁelds. On the other hand,when the target area in an image is relatively small, recognition accuracy deteriorates. So, it is necessary to extract thetarget area ﬁrstly and then perform object recognition.
So, various methods for simultaneously detecting and recognizing objects have been proposed. Faster R-CNN performed both of them in a lump by collective end-to-end train-

61

International Workshop on Informatics ( IWIN 2022 )ing of both models [4], and YOLO executed them with a single neural network to improve efﬁciency [2]. Concerning different scale objects, SSD made it possible to process themcollectively [5], and RetinaNet improved efﬁciency by introducing the Feature Pyramid Network (FPN) and improving
the loss function [6], [7]. Then, M2Det has further improved
accuracy and efﬁciency by introducing the new FPN and loss
function [8].
Among these methods, YOLO has been improved repeatedly through version upgrades, and several models are currently available as YOLOv5 [9]. YOLO estimates the bounding box surrounding the target area and the probability of containing the target when the center of the target area is locatedin a grid cell. The grid cell is a part of the image divided bygrids. And, YOLO is known to have high detection efﬁciency
and accuracy. And, it has been shown that YOLO can be applicable to real-time object detection and recognition [3].
However, because the above methods use deep learning,
it is necessary to prepare training data consisting of images
and correct labels for model training. For example, YOLO
requires not only the preparation of images for training themodel but also the corresponding correct labels indicating thelocation and classiﬁcation of bounding boxes for each object
contained in each image. Therefore, when targeting a large
number of object types, the burden of creating these labels
increases, which is a major obstacle in practical applications.
On the other hand, trained models and training data for
YOLOv5 for various objects, such as the coco dataset, areavailable on the Internet [9]–[11]. Therefore, when targeting
speciﬁc objects, YOLO can be easily used for the object detection and recognition from videos in real-time.
The motivation for this study is the idea that the target areaof an arbitrary object can be extracted, by detecting the locus
of a speciﬁc object such as the tip of a hand indicated by agesture in a video. This locus can be detected in real-time by
YOLO, targeting only one type of object, that is, training themodel is easy. In addition, using the target area extracted inthis way, the target size and tilt can also be estimated from thearea. In other words, when recognizing still objects viewed
from a speciﬁc direction, such as back covers of books ona shelf, it is expected that a simpler method such as templatematching can be used instead of the methods using deep learning.
Several applications have been proposed for hand gesturerecognition using deep learning, such as conversation and device control [12]–[14]. However, I could not ﬁnd application
studies to extract the target area of an object. Furthermore, incontinuous object detection using gestures in videos, it is necessary to eliminate noises due to object detection errors andto correct a double or missing part near the beginning and endof the locus.
The aim of this study is to propose a method for extracting the target area with high accuracy using gesture locus andclarify its effects and practical issues for applying it to object
recognition.

Figure 1: Right hand object detection and recognition using
YOLOv5

3 PROPOSAL OF TARGET AREA
EXTRACTION METHOD USINGGESTURE
3.1 Target Video Frame Images
I propose a method for extracting the target area by using
gestures in videos. In this method, the locus of the gesture is
detected using YOLOv5 (hereinafter, YOLO) sequentially foreach frame of the video, and extracts the area surrounded by
the locus for the target area. In order to correctly detect thelocus indicated by the gesture, we perform noise eliminationand locus correction as mentioned in Sec. 2.
Figure 1 shows an object detection and recognition (hereafter, object detection) result image of the right hand by YOLO
from a video frame. The detected target is indicated by abounding box, and the class of the target and the recognition
accuracy are indicated above the upper side of the box. In
this image, my right hand (“myright”) is detected with an accuracy of 77% (“0.77”) in the center. On the other hand, thelower-left bounding box is falsely detected noise, and its accuracy is 26%. Therefore, in this case, the former is adopted.

3.2 Target Area Extraction Procedure
The procedure to extract the target area from such as theimage of Fig. 1 is shown as follows. Valid coordinates of thelocus are selected, and noises are eliminated using the coordinates median of adjacent frames. Then, the locus is correctedusing the moving average.
3.2.1 Selection of Valid Coordinates in Locus
In this method, one of the vertices of the bounding box shownin Fig. 1 is selected as a point constructing the locus of thegesture. In the following, it is assumed that the coordinatesof the top-left vertex are selected. The valid bounding box isselected in each image with the following condition: its accuracy is the highest in the image and greater than the thresholdnamely the speciﬁed value. And, its coordinate of the top-leftvertex is adopted for the locus as the valid coordinate.
If each set of the coordinate and accuracy of the i-th frame
is indicated by cij and aij , the valid coordinates si shown in62

International Workshop on Informatics ( IWIN 2022 )㼥㻺㼛㼕㼟㼑㻌㻔㼜㼓㻕㻹㼑㼐㼕㼍㼚 㼏㼛㼛㼞㼐㼕㼚㼍㼠㼑

㼥

㼙㼓
㼜㼓㻗㻝
㼜㼓㻙㻞

㻾㼑㼙㼍㼞㼗㼟㻕㻌㻌㻌㻌㻌㻦㻌㻯㼑㼚㼠㼑㼞㻌㼛㼒㻌㼓㼞㼍㼢㼕㼠㼥㻌㻔㻳㻕㻧㻌㻌㻌㻌㻌㻌㻦㻌㻹㼛㼢㼕㼚㼓㻌㼍㼢㼑㼞㼍㼓㼑㻌㼜㼛㼕㼚㼠㻌㻔㼢㼍㻕㻧㻦㻌㻼㼛㼕㼚㼠㼟㻌㼛㼚㻌㼘㼛㼏㼡㼟㻌㼣㼕㼠㼔㻌㼚㼛㼕㼟㼑㻌㼑㼤㼏㼘㼡㼐㼑㼐㻌㻔㼜
䠺㼍㻕㻚

㼜㼓㻗㻞

㼜㼓㻙㻝
㻯㼘㼛㼟㼑㼟㼠㻌㼜㼛㼕㼚㼠 㼤㻔㻝㻕㻌㻲㼍㼘㼟㼑㻌㼠㼞㼍㼖㼑㼏㼠㼛㼞㼥

㼪㼜㼓㻗㻝
㼪㼜㼓㻙㻞

㻳

㼪㼜㼓㻗㻞

㼀㼍㼞㼓㼑㼠㻌㼍㼞㼑㼍

㻾㼍

㼪㼜㼓㻙㻝

㼤
㻔㻞㻕㻌㻯㼛㼞㼞㼑㼏㼠㼑㼐㻌㼠㼞㼍㼖㼑㼏㼠㼛㼞㼥

㼢㼍

䠺㼜㼍

Figure 2: Noise reduction by median coordinate
the Eq. (1) is selected.
cik (∃k(aik ≥ L ∧ aik = max({aij }))si =∅(∀k(aik < L))㻔㻞㻕㻌㼀㼍㼞㼓㼑㼠㻌㼍㼞㼑㼍㻌㼑㼤㼠㼞㼍㼏㼠㼕㼛㼚㻔㻝㻕㻌㼃㼔㼛㼘㼑㻌㼘㼛㼏㼡㼟

Figure 3: Target area extraction using moving average(1)Here, ∅ indicates that the coordinates of the frame are notselected; L indicates the threshold; {aij } indicates the set ofaij . In the case of Fig. 1, if L = 0.4, then the upper-leftcoordinate of the center bounding box is selected because its
accuracy is the highest and greater than the threshold.
3.2.2 Noise Elimination Using Median Coordinates
To eliminate noises, the median coordinates are calculated
from the valid coordinates of the previous and next frames.
Let pg (g = 1, 2, 3, · · · ) be the ordered set of coordinates witheliminating si if si = ∅ from {si } the set of si in Eq. (1).
Figure 2 (1) shows an example where pg is a noise.
The median of pg is constructed using the interval before
and after the index g. Let Rg indicates the set of indices of thisinterval, and let pRx and pRy indicate the set of x-coordinates
and y-coordinates, respectively. I deﬁne the median coordinate of this interval by mg = (median(pRx ), median(pRx )).
Here, median is the function to get the median value of thecoordinates. And, in the case of Fig. 2, the coordinate “mg”
with the median of each of the x-y coordinates is selected.
The noises are eliminated by using these median coordinates. As shown in Eq. (2), if pg is not the closest coordinate
to mg for the interval Rg , then it is converted to a coordinatep̃g with empty ∅; else pg is adopted for p̃g .

pg (m̃g = pg )
p̃g =
(2)
∅ (m̃g = pg )Here,

The other coordinates of are set as p̃n = pn (r ∈ Rg ). As
a result, a locus without the noise is constructed as shown in
Fig. 2 (2).
3.2.3 Target Area Extraction by Using Moving Average
To compensate for doubles or missing near the beginning andend of the locus, a moving average of the coordinates along
the angle from the center of gravity is created. First, the x-y
coordinate of the center of gravity G is obtained as a simple average of the x-coordinate and y-coordinate of the set ofcoordinates {p̃g } excluding noises (∅), respectively.
Next, as shown in Fig. 3 (1), the coordinates of each point
are transformed into a pair p̂a = (θa , ra ) of angle and distancefrom G. Let Ra be the interval to calculate the moving averagecorresponding to p̂a and deﬁne the moving average va by Eq.
(3).
va = (θa , r̄a )
(3)
Here,
r̄a = (

ru )/n(u ∈ Ra )The n is the number of coordinates contained in Ra , and it is
5 in the case of Fig. 3 (1). In other words, r̄a is the average ofthe distances between G and the coordinates p̂a (a ∈ Ra ).
By connecting the coordinates of this moving average set{va } along the angle, the target area is extracted as shown in
Fig. 3 (2).

4 IMPLEMENTATION AND
EXPERIMENTS
4.1 Implementationm̃g = {ph |∃h(dist(ph ,mg ) = min(dist(pn , mg ))∧∀n ∈ Rg )}dist(pn , mg ) indicates the distance between pn and mg . In
other words, m̃g denotes the coordinate of pn (r ∈ Rg ) that isclosest to the median coordinate mg ; and, if the corresponding point pg is not this coordinate, then it is set to ∅. The
coordinates of the locus without noises are obtained by eliminating ∅ from the set of coordinates {p̃g }.
In the case of Fig. 2, the closest coordinate to the medianm̃g is pg−1 , so the coordinate p̃g is set to ∅ and eliminated.

The experimental system was constructed to verify that the
proposed method can extract the target area. This system was
implemented on a Windows 10 PC, Python Ver. 3.8.13 as the
program, and Pytorch Ver. 1.7.1 with CUDA Ver. 11.5 to use
YOLO, OpenCV-Python Ver. 4.5.5.64 for image and video
manipulation.
YOLOv5s, a highly efﬁcient model of YOLO, was used
and was implemented by adding the necessary functions to
the publicly available program [9]. Similarly, the publicly
available “Egohand Dataset” [11] was used for the training
data of the model. This is the data for training the model to63International Workshop on Informatics ( IWIN 2022 )㻔㻝㻕㻌㻸㼛㼏㼡㼟㻌㼛㼒㻌㼙㼑㼐㼕㼍㼚

Figure 5: Original gesture locus detected by YOLO

Figure 4: Gesture locus and target area extraction experimentusing proposed method
detect four types of hands, the left and right hands of oneself
and the other party, and the number of data is 3,840.
Using a model trained with this data, I implemented a program to extract the target area from a video of hand gestures.
First, the hands are detected at each frame, and the valid coordinate in locus is selected using the procedure shown in Sec.
3.2.1. In this implementation, the right hand was used, andthe upper-left corner was assumed to be the tip of the hand, as
shown in Fig. 1.
Next, the procedure mentioned in Sec. 3.2.2 and 3.2.3 isused to eliminate noises by median coordinate and extract thetarget area by moving average. Five points were used to calculate each median coordinate, including the target point andits front and rear points. Since there was no point on one
side of the endpoints, their median coordinates are omitted.
For the next point, the median coordinate was calculated withthree points instead of ﬁve points. The moving average isalso calculated using 5 points, and the set of moving averages{va } is obtained. The target area is extracted from {va } using
OpenCV’s ﬁllConvexPoly function.
Finally, the bounding box containing the target area is extracted, setting the target area to the frame image and the outside to white.

4.2 Experiments
Using the implemented program, I conducted an experiment to extract the locus of the tip of my hand (hereinafter,hand) captured on video. To conﬁrm that the program candetect even in the case of complex backgrounds, I used thebookshelf shown in Fig. 1. I used a SONY FDRX3000 action
camera, and shot videos at 1, 920 × 1, 080 pixels and 30 fps.
The hands were moved in a clockwise circular motion starting from the lower right in the image. The accuracy threshold
L was set to 0.4.
Figure 4 shows the locus constructed by selecting the coordinates with the highest accuracy for each frame. That is, thisis the original locus detected by YOLO. For the backgroundin Fig. 4, the image of Fig. 1 is used. In addition, in thecase of this frame image, the hand detected in the center was
adopted, so the left-top vertex of its bounding box is on thelocus. As shown in Fig. 4, since there were noises due to false
detections, the target area could not be extracted directly fromthis locus.

㻔㻞㻕㻌㻸㼛㼏㼡㼟㻌㼛㼒㻌㼙㼛㼢㼕㼚㼓㻌㼍㼢㼑㼞㼍㼓㼑

Figure 6: Target area extracted by proposed method
Figure 5 (1) shows the locus after selecting only the valid
coordinates and eliminating noises by using median as described in Sec.3.2.1 and 3.2.2, respectively. Note that only
the vicinity of the locus has been extracted from the whole
image. The noises were eliminated, but the locus was doubled near its beginning and end. Figure 5 (2) shows correctedlocus from the one shown in Fig. 5 (1) by using the moving
average mentioned in Sec. 3.2.3. The moving average corrected the locus to the place between the doubled loci, anda closed area could be constructed. However, near the endpoints, since the hand locus deviated from the target area, an
extra area was included as shown right-lower part.
Figure 6 shows the bounding box of the target area extracted using the locus of Fig. 5 (2). The outside of the target
area has been transformed to white.
We performed the above procedure three times in the sameenvironment to examine the number of frames detected ateach stage of the procedure. Figure 7 shows the results, andFigure 7: Corrected locus by proposed method

64International Workshop on Informatics ( IWIN 2022 )

㻔㼍㻕

㻔㼎㻕

㻔㼏㻕

㻔㼐㻕

㻔㼑㻕

㻔㼒㻕

Figure 9: Objects used for template matching
Figure 8: Evaluation of target area extraction accuracy
Figs. 4 to 6 correspond to “Case2”. The vertical axis indicatesthe number of the detected frames, and the horizontal axis indicates the stage. “Total” shows the total number of frames,“YOLO” shows the number of detected by YOLO including
noises, and “Valid” shows the number of accuracies above
the threshold (0.4). “Median” shows the number obtained byusing medians, that is, the number of frames after eliminating noises, and the number after the moving average is also
the same. Note that since the two points at both endpoints of
the locus are excluded in the Median stage, as mentioned in
Sec.4.1, two points are also excluded for the numbers in theother stages.
As shown in Fig. 7, there was a large difference in the proportion of detections even under similar conditions. In Case3,
the proportion detected by YOLO was less than half that ofCase2; conversely, Case2 had the highest number of points
eliminated due to accuracy under the threshold at the Validstage. The number of points judged as noise in the Medianstage was 7 in Case1 while it was 3 in Case2. The former’s
percentage of the total (48), was 14.6%.

5 EVALUATION OF TARGET AREA
EXTRACTION AND OBJECT
RECOGNITION ACCURACY
5.1 Evaluation of target area extraction
accuracy
To evaluate the accuracy of extracting the target area whenusing hand gestures, I evaluated the extraction accuracy using
a round wall clock. In this experiment, the camera was ﬁxed
and the hand was moved while watching the monitor in orderto evaluate the accuracy of the locus indicated by hand. The
used camera was a Nikon COOLPIX A1000, and the resolution and frame rate were the same as in the experiment of Sec.
4.2.
In Fig. 8, the red line shows the locus constructed by using
the median; the green line shows the one by using the moving
average. The accuracy of the target area is low with respect tothe target clock, and in this case, it is outside. Furthermore, atthe lower-right part namely near the endpoints of the gesture,
it is quite outwardly displaced. The former was caused by using the hand for the gesture, which was too large compared tothe target object. The latter was caused by the detection of ex-

㻔㼎㻕

㻔㼍㻕

㻔㻝㻕㻌㼃㼔㼛㼘㼑㻌㼍㼞㼑㼍㻌㼛㼒㻌㼕㼙㼍㼓㼑㻔㼏㻕㻔㻞㻕㻌㼀㼍㼞㼓㼑㼠㻌

Figure 10: Images for evaluations of size errors
tra hand movement near the beginning and end of the gesture,
namely the movement between the target and the external of
the image.

5.2 Evaluation of Object Recognition
Accuracy Improvement
To investigate the impact of target area extraction on object
recognition accuracy, I evaluated the recognition accuracy ofbooks stored on bookshelves using template matching. The
purpose of these evaluations is to clarify the accuracy required
to extract the target area by the gesture.
For template matching, the “matchTemplate” function of
OpenCV was used with the normalized squared difference
matching method. The books to be recognized are the six
books shown in Fig. 9. For these books, we evaluated thevariation in accuracy when there are errors in size and tilt between the template and the target objects in the image, andwhen the range of the images was narrowed to the vicinity ofthe target.
Figure 10 shows the images to evaluate the case of size
error. Figure 10 (1) shows the whole image; Fig. 10 (2) shows
the image with the narrowed area. Though the latter size isenlarged in this ﬁgure, both images are the same size in thisexperiment. The numbers (a), (b), and (c) below each object
correspond to Fig. 9. Similarly, Fig. 11 shows an image toevaluate the case of tilt error, and note that the margins createdby the rotation are ﬁlled in with white. In addition, the images
in Figs. 10 and 11 were shot in a different environment fromthat of the template image in Fig. 9.
Figure 12 shows the template matching results of all theimages in Fig. 9 against Figs. 10 (1) and (2), namely the caseof size errors. The “Whole” and “Vicinity” in Fig. 12 corre-

65

International Workshop on Informatics ( IWIN 2022 )

㻔㼐㻕

㻔㼑㻕

㻔㻝㻕㻌㼃㼔㼛㼘㼑㻌㼍㼞㼑㼍㻌㼛㼒㻌㼕㼙㼍㼓㼑㻔㼒㻕㻔㻝㻕㻌㻿㼕㼦㼑㻌㼑㼞㼞㼛㼞㻌㻩㻌㻜㻑㻔㻞㻕㻌㻿㼕㼦㼑㻌㼑㼞㼞㼛㼞㻌㻩㻌㻞㻜㻑㻔㻞㻕㻌㼀㼍㼞㼓㼑㼠㻌㼢㼕㼏㼕㼚㼕㼠㼥

Figure 14: Template matching results on size error
Figure 11: Images for evaluations of tilt errors㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼐㼑㼠㼑㼏㼠㼕㼛㼚㼟㻿㼕㼦㼑㻌㼑㼞㼞㼛㼞

Figure 12: Improvement results for size errors
spond to (1) and (2) in Fig. 10, respectively. “0%” indicatesthe case where the template size is adjusted to the target ofFig 10, while “10%” and “20%” indicate the case where thetemplate is enlarged to this size, respectively. As shown in
Fig. 12, the recognition accuracy degraded as the size error
increased, and in the case of the “Whole”, no image was recognized at “20%”. On the other hand, the recognition accuracy in the case of “Vicinity” was improved, and two images
were recognized even at “20%”.
Similarly, Fig. 13 shows the result of evaluating the tilt errors, in which “Tilt error” corresponds to the magnitude ofthe error between the objects in Fig. 9 and Fig. 11. “0◦ ” indicates the case where the image is rotated so that the books
are vertical, while “5◦ ” and “10◦ ” indicate the case where therotation is insufﬁcient by this angle, respectively. As shownin Fig. 13, the recognition accuracy degraded as the tilt er㻺㼡㼙㼎㼑㼞㻌㼛㼒㻌㼐㼑㼠㼑㼏㼠㼕㼛㼚㼟㻔㻝㻕㻌㼀㼕㼘㼠㻌㼑㼞㼞㼛㼞㻌㻩㻌㻜㼻㻔㻞㻕㻌㼀㼕㼘㼠㻌㼑㼞㼞㼛㼞㻌㻩㻌㻝㻜㼻

Figure 15: Template matching results on size error
ror increased, and in the case of the “Whole”, no image wasrecognized at “10◦ ”. On the other hand, similar to Fig. 12,
the recognition accuracy of the “Vicinity” was improved, andtwo images were recognized even at “10◦ ”.
In addition, in the case shown in Fig. 12, the books (d) to(f) shown in Fig. 11 (2) are not detected; conversely, in thecase shown in Fig. 13, the books (a) to (c) shown in Fig. 10(2) were not detected. This is because the tilt of books was
too large against the corresponding template shown in Fig. 9,
respectively.
Figures 14 and 15 show examples of the matched imagesfor (2) of Figs. 10 and 11, where the matched places are indicated by the rectangles. As shown in (1) of Figs. 14 and15, when the errors of scale and tilt are small, the extractions
were accurate. However, when these errors were large, theaccuracies were degraded due to the inclusion of areas otherthan the target, as shown in (2) of Figs. 14 and 15.

6 DISCUSSION

㼀㼕㼘㼠㻌㼑㼞㼞㼛㼞

Figure 13: Improvement results for tilt errors

In this study, I am trying to clarify the effectiveness of the
proposed target area extraction method and the issues for its
practical applications.
As shown in Fig. 4, several noises tend to be included in
the locus due to false object detections used for the gesture.
As shown in Fig. 5 (2) and Fig. 8, the proposed method
was able to extract the target area by eliminating these noises.
Therefore, I consider that the proposed method is effective for66International Workshop on Informatics ( IWIN 2022 )the purpose of target area extraction.
As mentioned in Sec. 2, object recognition accuracy can be
improved by specifying the target area in the image, namely
by object detection. In addition to this, this method can extract arbitrarily shaped target areas, so it is possible to detect
the target size and tilt by utilizing the target area. As shownin Figs. 12 and 13, it was possible to improve the recognition
accuracy even in simple object recognition such as templatematching by using this data,
On the other hand, through experiments and evaluations,the issues for practical use were found, too. The ﬁrst issue
is object detection accuracy including the position of the object used in the gesture. By extracting the target area using
gesture, for example, the “vicinity” shown in Figs. 14 and 15
can be applied for the template matching case. In this case,with a size error of less than 10% and a tilt error of less than10◦ , the detection was correct. However, as shown in Fig. 8,
the gesture locus deviated from the target area. In addition, as
shown in Fig. 7, the object detection accuracy differed greatly
for even similar gestures.
The former was due to the hand being too large to specify the target area. The latter was due to the use of existing
training data. In other words, it is considered that there weresome differences in visibility between the images of existing
training data and the images of the gesture, though they are
the same objects namely hands.
To address these problems, the following measures can beconsidered. First, we can use the part of the body that can
specify the target area more precisely, such as the ﬁngertip,
for the gestures. Second, for efﬁciently model training, we
can use transfer learning based on the model trained with theexisting training data used in this study according to the usage
part of his/her body. In addition, for distant objects, it is considered effective to perform gestures while monitoring with awearable camera to suppress the difference in viewpoint between the camera and the operator.
The second issue is that, as shown in Fig. 5 (1), the extra
locus is detected. This method aims to automatically extract
the target area from the continuously shot videos. However,I found that extra hand motions before and after the target
gesture are also detected as a part of the gesture. To address
this issue, for example, it is considered to stop the gesture at
the beginning and end of the target gesture and identify the
extra frames in videos.
The construction and evaluations of these measures are thesubjects of the next study.

7 CONCLUSION
In order to recognize a small object in an image, it is ﬁrst
necessary to detect the object, and various methods for simultaneous object detection and recognition such as YOLO havebeen proposed. However, since these methods utilize deeplearning, there is an application problem that it is necessary
to prepare the training data for each target object.
For this problem, I propose a method to detect the locus indicated by hand gestures in videos using YOLOv5 and extract
the target area. Experiments have shown that this method caneliminate noises due to false detection and improve the accuracy of target area detection. Furthermore, I evaluated the effectiveness of this method for template matching and showedthat the recognition accuracy can be improved by detecting
the target size and tilt in addition to the target area.
However, it was found that further improvement in the target area detection was necessary to improve this recognition
accuracy. So, future studies include improving the accuracyof the target area to be extracted by gestures.

REFERENCES[1] P. Viola, and M. Jones, “Rapid object detection using aboosted cascade of simple features,” Proc. 2001 IEEE
Computer Society Conf. Computer Vision and Pattern
Recognition, I–I (2001).
[2] J. Redmon, S.Divvala, R. Girshick, and A. Farhadi,“You only look once: Uniﬁed, real-time object detection,” Proc. IEEE Conf. Computer Vision and Pattern
Recognition, pp. 779–788 (2016).
[3] Z. Wang, L. Jin, S. Wang, and H. Xu, “Apple stem/calyx
real-time recognition using YOLO-v5 algorithm for
fruit automatic loading system,” Postharvest Biology
and Technology, Vol. 185, No. 111808 (2022).
[4] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN:
Towards real-time object detection with region proposal
networks,” Advances in Neural Information Processing
Systems, pp. 91–99 (2015).
[5] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed,
C. Y. Fu, and A. C. Berg, “SSD: Single shot multibox
detector,” European Conf. Computer Vision, pp. 21–37,
Springer. (2016).
[6] T. Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan,and S. Belongie, “Feature pyramid networks for object
detection,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2117–2125 (2017).
[7] T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar,“Focal loss for dense object detection.” Proc. IEEE Int.
Conf. Computer Vision, pp. 2980–2988.(2017).
[8] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L.
Cai, and H. Ling, “M2det: A single-shot object detector based on multi-level feature pyramid network,” Proc.
AAAI Conf. Artiﬁcial Intelligence, Vol. 33, pp. 9259–
9266.(2019).
[9] G. Jpcher, et.al. “YOLOv5,” https://github.com/
ultralytics/yolov5 (referred May 17, 2022).
[10] T. Y. Lin, et.al., “Microsoft coco: Common objects incontext,” European Conf. Computer Vision, pp. 740–
755 (2014).
[11] S. Bambach, S. Lee, D. Crandall, and C. Yu, “Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions,” IEEE Int.
Conf. Computer Vision (ICCV), pp. 1949-1957 (2015),
https://public.roboﬂow.com/object-detection/hands (referred May 19, 2022).
[12] M. Oudah, A. Al-Naji, and J. Chahl, “Hand gesturerecognition based on computer vision: a review of techniques,” J. Imaging, Vol. 6, No. 8, 73 (2020).
[13] A. Mujahid, M. J. Awan, A. Yasin, M. A. Mohammed,

67

International Workshop on Informatics ( IWIN 2022 )R. Damaševičius, R. Maskeliūnas, and K. H. Abdulkareem, “Real-time hand gesture recognition based on
deep learning YOLOv3 model.” Applied Sciences, Vol.
11, No. 9, 4164 (2021).
[14] Y. Shi, Y. Li, X. Fu, K. Miao, and Q. Miao, “Review ofdynamic gesture recognition,” Virtual Reality & Intelligent Hardwar, Vol. 3, No. 3, pp. 183–206 (2021).

68

International Workshop on Informatics ( IWIN 2022 )An Analysis of Tweets in Disasters for Extracting Rescue Requests
Yuki Koizumi† , Junji Takemasa† , Toru Hasegawa† , and Yoshinobu Kawabe‡† Osaka University, Japan‡ Aichi Institute of Technology, Japan{ykoizumi, j-takemasa, t-hasegawa}@ist.osaka-u.ac.jp, kawabe@aitech.ac.jp
𝑨𝒃𝒔𝒕𝒓 𝒂𝒄𝒕 - During catastrophic disasters like the Japan ﬂoods
2018, phone-based emergency call systems may not work as
expected due to heavy congestion or network disruption. Social media, e.g., Twitter and Facebook, has been playing anessential role as a communication tool to deliver rescue requests in disasters, and it complements phone-based emergency call systems. Understanding rescue requests on social
media is key to realizing the automatic extraction of rescue
requests from a vast amount of social media posts. This paper
analyzes rescue-related tweets, which are tweets containing
rescue-related keywords like #rescue, and proposes a taxonomy for rescue-related tweets. According to the analysis, weconstruct a classiﬁer based on a recurrent neural network andan annotation mechanism to identify why the classiﬁer identiﬁes a tweet as a rescue request. Using these two models, wereveal why there are many false negatives, i.e., the number ofrescue requests identiﬁed as non-rescue requests.

tributed to saving about 50 victims, but on the other hand it
consumed many workers, who may have contributed to othertasks. These tales imply that it is indispensable to extract rescue requests on social media automatically to utilize them for
rescue activities. Machine learning is a promising techniquefor ﬁltering rescue requests on social media [5]–[7].
Rescue-related social media posts, which are deﬁned as social media posts having rescue-related keywords like #rescue,
are a “mixture of good and bad” as we will analyze them in Section 2. Speciﬁcally, many rescue-related social media posts
are not related to rescue requests although they have rescuerelated keywords. One essential issue for extracting rescue
requests from a vast amount of social media posts is variety incontexts of rescue-related social media posts. Rescue-relatedsocial media posts include not only rescue requests but also
disaster information and sympathy. Another crucial issue isambiguity in social media posts due to the nature of free-form
texts. The ambiguity results in blurring the boundary between
rescue requests and disaster information.
Understanding real rescue-related social media posts is a
key to realizing good classiﬁers for extracting rescue requests
from social media automatically. While many existing studies
analyzed social media posts related to natural disasters, suchas hurricanes [8]–[10], few studies analyze rescue requests onsocial media.
The key contributions of this paper are summarized as follows. First, this is the ﬁrst study that analyzes rescue-relatedsocial media posts from the perspective of extracting rescue
requests as far as we know. We captured real social media
posts on Twitter during several recent ﬂoods in Japan and analyze them. Hereafter, we refer to social media posts simply
as tweets since this study focuses on Twitter as social media.
Second, we conduct preliminary experiments of classifying
rescue requests from numerous tweets with machine learning.
Our experiments reveal several lessons to build a classiﬁer
based on machine learning to extract rescue requests fromtweets.
The paper is organized as follows: In Section 2, we analyze
disaster-related and rescue-related tweets. Based on observations found in the analysis, we build a classiﬁer to extract
rescue requests from Twitter in Section 3. Section 4 brieﬂysummarizes related work and Section 5 ﬁnally concludes thispaper.

𝑲𝒆𝒚𝒘𝒐𝒓 𝒅𝒔: Social Media, Twitter, Disaster, Analysis, Machine Learning

1

Introduction

Delivering rescue requests from citizens in need of help tothe right persons, such as rescue authorities and ﬁrst responders, is a key to eﬀective disaster management. Phone-based
emergency call services, however, may not work as expectedduring and after catastrophic disasters because of network disruption and congestion [1].
Circumstances of rescue requests during disasters have beenchanging. Social media, e.g., Twitter and Facebook, plays avital role in delivering rescue requests from victims and it
complements existing phone-based emergency call services.
Japan, for instance, had several catastrophic ﬂoods in 2018and 2019. One is the heavy rain of July 2018, also referred
to as Japan ﬂoods 2018 [2] and another is the 19th typhoon of2019 [3]. We observed that many rescue requests were posted
on Twitter during both the disasters.
Few rescue requests on social media, nevertheless, contributed to actual rescue activities. Through our analysis, discussed later in Section 2, reveals that 312 rescue requests wereposted during the Japan ﬂoods 2018, none of them directly
contributed to rescue activities. Most of the rescue requests
on Twitter were forwarded to phone-based emergency call services by voluntary workers, which caused further congestionof the phone-based emergency call services. As another example, a local government, Nagano Prefecture, deployed several
workers to capture rescue requests from Twitter [4] during
the 19th typhoon. On the one hand this activity ﬁnally con-2

Analysis on Disaster-related Tweets

This section analyzes disaster-related tweets captured during the recent ﬂoods in Japan in order to understand rescue
requests in tweets.

71

Number of Tweets

International Workshop on Informatics ( IWIN 2022 )Table 1: Search keywords used for collecting disaster-related
tweetsCategory Keywords

RescuerequestFirst
responder
Infrastructure

× 105

4
2
0

07

-0
07 1
-0
07 3
-0
07 5
-0
07 7
-0
07 9
-1
07 1
-1
07 3
-1
07 5
-1
07 7
-1
07 9
-2
07 1
-2
07 3
-2
07 5
-2
07 7
-2
9

Heavy rain, rain disaster, disaster, ﬂood, ﬂood
disaster, disaster-hit area, ﬂood-hit area, river
burst, evacuation, and (conﬁrmation of someone’s) safety
Rescue, rescue request, SOS, and help (me)Date (July 2018)(a) Number of disaster-related tweets
Number of Tweets

Raindisaster6

× 104

RescueFirst responder
Infrastructure
Volunteer

4
2
0

07

-0
07 1
-0
07 3
-0
07 5
-0
07 7
-0
07 9
-1
07 1
-1
07 3
-1
07 5
-1
07 7
-1
07 9
-2
07 1
-2
07 3
-2
07 5
-2
07 7
-2
9

Rescue team, ﬁre ﬁghting team, police, Japan selfdefense forces, hospital, and local government
Infrastructure, lifeline, water supply, electricity
supply, gas supply, (network) disconnection, (network) congestion, (network) failure, and recovery
Volunteer Volunteer, support, and relief supply

Date (July 2018)(b) Number of rescue, ﬁrst responder, infrastructure, andvolunteer-related tweets

Table 2: The number of tweets during the Japan ﬂoods 2018

2.1

Category

NumberTotal
Rescue requestFirst responder
Infrastructure
Volunteer

6,978,389
246,807932,605
889,889324935Figure 1: Time series analysis of the number of tweets

2.2

Spatiotemporal Analysis

We investigate how the number of tweets grows according
to situations in a ﬂood disaster. To understand this analysis, we
brieﬂy explain the Japan ﬂoods 2018. From the end of Juneto the middle of July 2018, there was successive heavy rain
in western Japan, and it caused widespread and catastrophic
ﬂoods throughout western Japan, especially in Okayama andHiroshima. From July 5th to 7th, the most severe ﬂoods
occurred in Okayama and Hiroshima.
The time series of the number of disaster-related tweets areplotted in Fig. 1a. The horizontal and vertical axes represent
the date and the number of tweets in each day. The numberof disaster-related tweets steeply increases around July 5th as
ﬂoods became severe, and its peak is on July 6th and 7th, themost intense days of ﬂoods. The numbers of tweets on July
6th and 7th are 591,514 and 584,692, respectively. Note thatthe number of disaster-related tweets again increases at theend of July because another typhoon came to Japan.
Figure 1b indicates the number of rescue, ﬁrst responder,
infrastructure, and volunteer-related tweets. Though rescuerelated tweets also increased as ﬂoods became severe, thetrend is shifted slightly behind compared to that of disasterrelated tweets. That is, the number of rescue-related tweets
suddenly increases on July 7th, and the peak is also on thesame day. We captured that 48,272 rescue-related tweets on
July 7th. First responder and infrastructure-related tweets
were captured almost invariably throughout the disaster. In
contrast, the number of volunteer-related tweets increases in
the aftermath of the disaster.
Next, we analyze the locations where rescue requests wereposted. Because few tweets have geographical metadata likegeotag information, we extract rescue requests (not rescuerelated tweets) that contain a postal address in their text ﬁeld.
In addition, not only retweeted but also copied tweets are
eliminated. We found 312 rescue requests indicating a postal
address, and we put pins pointing the postal addresses on

Data Set

We collected tweets from three ﬂood disasters in Japan: theJapan ﬂoods 2018 [2], the 15th [11], and the 19th typhoons [3]of 2019 in Japan. We use the tweets from the Japan ﬂoods
2018 for the analyses in this section and those from the othertwo disasters for the classiﬁcation in the next section.
The tweets were captured via the Twitter search API [12]by specifying disaster-related keywords, which were selected
so that we could capture as many disaster-related tweets as
possible. Tweets retweeted with the twitter oﬃcial API are
eliminated from the data set. Let us note that all the keywords
are Japanese, and therefore all the tweets are also Japanese.
The keywords are categorized into ﬁve classes: rain disaster,
rescue request, ﬁrst responder, volunteer, and infrastructure,
as summarized in Table 1. We refer to tweets containing atleast one keyword in any of the classes as disaster-related
tweets. In the same way, tweets containing keywords in theclasses of rescue request, ﬁrst responder, infrastructure, andvolunteer are referred to as rescue, ﬁrst responder, infrastructure, and volunteer-related tweets, respectively. Since mostdisaster-related tweets contain keywords in the rain disasterclass, we do not focus on tweets in this class. Each class
may have tweets unrelated to the class because they were captured with keyword search. For instance, tweets that are notrescue requests, although having rescue-related keywords, arecategorized into rescue-related tweets.
The number of collected tweets, i.e., disaster-related tweets,in the Japan ﬂoods 2018 is 6,978,389 and the number of tweets
in each class is summarized in Table 2.

72International Workshop on Informatics ( IWIN 2022 )• Unrelated: A tweet that is not related to the disaster
while it contains rescue-related keywords, which areused in a diﬀerent context from rescue requests, e.g.,
tweets regarding a smartphone game application.
This taxonomy implies that it is challenging to extract rescue
requests via keyword search because many tweets are unrelated
to rescue requests while they contain a rescue-related keyword.
According to a recommendation about rescue requests via
Twitter posted by Twitter Japan (@TwitterLifeline) [13], victims are recommended to post a rescue request indicating thedetailed current situation and the postal address (or geographical location) of victims as well as the hashtag #rescue. We,
however, captured many rescue requests that lack necessary
information like the location of victims. We categorize such
tweets into incomplete rescue requests.
We found 312 original rescue requests, some of which do
not have necessary information like the location of victims, inthe tweet data set. Many voluntary citizens re-posted such an
incomplete rescue request by adding missing information, e.g.,
the hashtag #rescue. Hence, there are many tweets categorized
to the class of incomplete rescue requests.

Flood

Isolated
Landslide
Missing Person

Other

Figure 2: A map pointing rescue requests during the Japanﬂoods 2018a map. Due to privacy issues, we present a coarse-grained
map in Fig. 2. One crucial observation is that rescue requests
were concentrated in severely ﬂood-hit areas, which were very
narrow, in the case of the Japan ﬂoods 2018. In a certain
town in Okayama, for instance, there were 235 rescue requests
indicating a postal address within a 7 × 5 km rectangle area.

2.3

3

Understanding Rescue-related Tweets

Classiﬁcation Based on Machine Learning

In this section, we build a classiﬁer that identiﬁes rescue
requests among rescue-related tweets and evaluate the performance. Moreover, we investigate reasons for false negatives
with an annotation mechanism [7], which explains reasons
why a classiﬁer identiﬁes a tweet as a rescue request.

To understand rescue requests on Twitter, we further picked
5305 tweets, which contains hashtags related with rescue
requests, i.e., #rescue, #rescue request, #SOS, #help, and#help_me, from the aforementioned rescue-related tweets andread all of them. We observe that many of the tweets are
not rescue requests. Based on the observation, we deﬁne ataxonomy of rescue-related tweets.
Our taxonomy is inspired by the study by Alam et al. [9].
While the taxonomy in [9] categorizes disaster-related tweets,
our taxonomy focuses only on rescue-related tweets and analyzes them in detail. Rescue-related tweets are categorized asfollows:• Rescue request: A tweet that indicates all the following
information: a rescue request, situations of victims, andlocations of victims.
• Incomplete rescue request: A tweet that indicates a
rescue request but lacks either or both the situation andthe location of victims.
• Disaster situation: A tweet that reports situations of
the disaster.
• Sympathy: A tweet that prays for the safe rescue ofvictims.
• Advice and supplement: A tweet that gives advice tovictims or adds supplementary information to existing
tweets.
• Volunteer: A tweet that oﬀers or requests voluntary
contributions, e.g., voluntary work and donations ofsupplies like foods, water, and clothes.
• Exploitation: A tweet that mentions something unrelated to rescue requests by intentionally exploiting
rescue-related keywords so that the tweet is widespread.
Such tweets, for instance, include political opinions andadvertises services.

3.1

Data Set

We use the same data set in Section 2. We use tweets
from the Japan ﬂoods 2018 and the 19th typhoon for train
and test data, respectively, because the two disasters caused
more severe damage than recent other typhoons in Japan. In
contrast, we use tweets from all the three disasters for creatinga corpus of disaster-related tweets because we need as many
tweets as possible to create an accurate corpus. For the train
and test data sets, we further extract tweets containing rescuerelated hashtags, i.e., #rescue, #rescue request, #SOS, #help,
and #help me. The train data set has 2000 rescue-related
tweets from the Japan ﬂoods 2018 and the test data set has
2091 rescue-related tweets from the 19th typhoon.

3.2

Model of Classiﬁer

We create a classiﬁer using a neural network as a machinelearning technique. The classiﬁer consists of three layers:a word embedding layer, a recurrent neural network layer,
and a densely connected neural network layer. The wordembedding layer converts a word to a vector of real numbers,
the recurrent neural network layer learns the relation between
tweets and labels, and the densely connected neural network
layer summarizes the output of the recurrent neural network
layer. A schematic diagram of the classiﬁer is illustrated inFig. 3.

73

International Workshop on Informatics ( IWIN 2022 )a recurrent neural network for classifying social media posts.
We adopt gated recurrent unit (GRU) cells in our recurrent
neural network in the same way as Kshirsagar et al. [7]. As
another famous variant of recurrent neural networks, manystudies use long short-term memory (LSTM) cells [16]. Bothcells can model long-term dependencies among values in an
input sequence. For notational simplicity, we refer to a recurrent neural network consisting of GRU and LSTM cells
as an LSTM and a GRU network, respectively. Although
LSTM networks generally outperform GRU networks [17],LSTM networks have more parameters than GRU networks,
and LSTM networks hence require more train data than GRU
networks. We compare a GRU and an LSTM network in thefollowing section. Since we do not have suﬃcient data fortraining an LSTM network due to the limitation of the numberof real rescue requests, we mainly use GRU networks. Finally,
we use a bidirectional recurrent network in the same way as
the model in [7].

orFully Connected Layer

GRU
Unit

GRU
Unit

GRU
Unit

GRU
Unit

GRU
Unit

GRU
Unit

Glove
Model

Glove
Model

Glove
Model

Figure 3: Model of the classiﬁer
3.2.1

Labels of Tweets

A label 𝑦 𝑡 is assigned to each tweet 𝑡 according to the taxonomy
deﬁned in Section 2. Labels are expressed by 8-bit one-hot
binary vectors, where each bit corresponds to each category.
For comparison purposes, one-bit binary labels 𝑧 𝑡 , where 1
indicates 𝑡 is a rescue request and 0 otherwise. For notation
simplicity, the classiﬁer trained with 8 category labels (8bit labels 𝑦 𝑡 ) and that trained with 2 category labels (1-bitlabels 𝑧 𝑡 ) are referred to as the 8-bit and the 1-bit classiﬁer,
respectively.
3.2.2

3.2.5

We use the following four metrics: accuracy, precision, recall,and F-measure. Let TP, TN, FP, and FN be the number oftrue positives, true negatives, false positives, and false negatives, respectively. In our case, a true positive (negative)corresponds to a (non-)rescue request that is (not) identiﬁed
as a rescue request. A false negative corresponds to a rescue
request that is not identiﬁed as a rescue request and a false
positive corresponds to a non-rescue request that is identiﬁed
as a rescue request. The most critical metric for classiﬁers of
rescue requests is recall, which represents the ratio of tweetsthat are identiﬁed as rescue requests among tweets that are
actual rescue requests. A low recall value means that many
rescue requests will be missed, i.e., there are many false negatives. Since rescue requests must not be missed, recall mustbe high for classiﬁers of rescue requests.

Preliminary Processing of Tweet

A tweet is segmented into words, and the words are converted
to their base form with MeCab [14], which is an open-source
text segmentation library for the Japanese language. Since
spaces do not separate words in the Japanese language, sucha tool is necessary for processing Japanese text. Next, URLs,
emoticons, and signs, including hash signs, are eliminated.
Screen names are converted into a special word, SCREENNAME. This process converts a tweet 𝑡 to a sequence of words{𝑤 1,𝑡 , . . . , 𝑤 𝑛,𝑡 }.
3.2.3

3.2.6

Annotation

To understand prediction results made by neural networks, weuse an annotation mechanism, also referred to as an attention
mechanism [7], [18], which analyzes hidden states 𝑎 𝑛,𝑡 produced by the GRU or the LSTM layer. Hidden states 𝑎 𝑛,𝑡 arereferred to as annotation values, hereafter. A high annotation
value represents that the corresponding phrase is related to a
rescue request.

Word Embedding

For constructing a word embedding model, we use GloVe [15],which computes vector representations for words according toword-to-word co-occurrence statistics in a corpus. We create a
corpus from the tweets from all the three disasters. The corpus
has more than 23 million unique tweets, and it contains morethan 0.6 billion words. By eliminating words appearing oncein the corpus, the corpus ﬁnally has 651,423 unique words.
We build a 256-dimensional word embedding model from thecorpus with GloVe, i.e., a word 𝑤 𝑖,𝑡 is mapped to a vector
𝑥𝑖,𝑡 ∈ R256 .
3.2.4

Performance Metrics

3.3

ResultWe ﬁrst evaluate the performance of a GRU and an LSTM
network by using the 1-bit classiﬁer. Table 3 summarizes theclassiﬁcation results and Table 4 shows the four performance
metrics of the two models.
Though both the GRU and the LSTM network achieve highaccuracy, they do not achieve high precision and recall. On the
one hand the LSTM network outperforms the GRU networkin terms of overall performance, i.e., accuracy and F-measure.
On the other hand the GRU network is superior to the LSTM
network in terms of recall. Since rescue requests must not be

Neural Network

We adopt a recurrent neural network as the main body of theclassiﬁer. A recurrent neural network is a kind of artiﬁcial neural network. It is capable of modeling the dynamics of input
values, and hence it is often used as a classiﬁer for time series
data, including text data. Kshirsagar et al. [7], for instance, use

74International Workshop on Informatics ( IWIN 2022 )Table 3: Result of classiﬁcation with the 1-bit classiﬁer
Model Predict\Label
1
0
GRU

1
0

209
72

165
1645

LSTM

1
0

181
100

70
1740

Table 5: Result of classiﬁcation with the 8-bit classiﬁer
Predict\Label
Rescue request Non-rescue request
Rescue request
Incomplete
rescue request
Disastersituation
Sympathy
Advice andsupplement
Volunteer
Exploitation
Unrelated

Table 4: Performance of GRU and LSTM
Model Accuracy Precision Recall F-measure
GRU
LSTM

0.887
0.919

0.559
0.721

0.744
0.644

0.638
0.680

124
83

28
86

51

177

11
8

748
362

0
1
3

1
310
98

Such tweets are misclassiﬁed into the category of either sympathy or advice and supplement.
Finally, we analyze tweets with the annotation mechanism
to investigate what kinds of phrases contribute to being rescue
requests. We use a 1-bit classiﬁer of a GRU network trainedwith tweets from the Japan ﬂoods 2018. Then, we derive
annotation values of phrases, which are generated by deriving
all possible 𝑛-grams (3 ≤ 𝑛 ≤ 10) in tweets from both theJapan ﬂoods 2018 and the 19th typhoon. The following list
summarizes phrases of a high annotation value:• Characteristics of victims like age and sex (e.g., elderly
persons, babies, children, woman, man, octogenarian(80s), septuagenarian (70s), grandfather, and grandmother).
• Situations of ﬂood-hit buildings (e.g., the water level has
been rising gradually and my house is ﬂooded above the
2nd ﬂoor level).
• Location names of ﬂood-hit areas during the Japanﬂoods 2018 (e.g., Mabi-town, Kurashiki-city, Okayama
Prefecture, Japan).
• Numbers in postal addresses (e.g., 𝑥-𝑦 (𝑥 and 𝑦 represent
a block and a house number and they are used like Mabitown 𝑥-𝑦 in Japanese style postal addresses)).
In contrast, the annotation values of phrases often used in
rescue requests (e.g., rescue request, SOS, help, share it if you
can, and retweet it if you can) are very low because they arealso used in non-rescue requests. Furthermore, the annotation
values of location names of typhoon-hit areas during the 19th
typhoon are also very low. If we replace a location name in a
false negative rescue request of the 19th typhoon to that of aﬂood-hit area during the Japan ﬂoods 2018, they are identiﬁed
as a rescue request.

missed, recall must be high. We, hence, use the GRU networkin the following experiments. The result, however, shows thatthe recall of the GRU network is not suﬃciently high, and it
misses approximately a quarter of rescue requests.
To investigate reasons for the misclassiﬁcation, we nextclassify rescue requests into the eight categories, which are
discussed in Section 2, using the 8-bit classiﬁer. We use a
GRU network in this evaluation. Table 5 summarizes the result. Many rescue requests are misclassiﬁed into the following
three categories: disaster situation, sympathy, and advice andsupplement. There are two typical observations about themisclassiﬁcation: The ﬁrst one is that similar expressions areused in both rescue requests and tweets in the three categories
and the other one is that many citizens retweeted rescue requests by adding text, which is likely to belong to the three
categories.
Regarding the ﬁrst observation, some rescue requests, forinstance, use guessing expressions, which are often used in
tweets reporting a disaster situation. The following tweet is atypical example of such rescue requests:
Help me. My house is ﬂooded above the 2nd
ﬂoor level. Nobody comes to my rescue. Roads
appear to be blocked due to landslide.
The last sentence is similar to tweets reporting disaster situations. Such rescue requests are misclassiﬁed into the categoryof disaster situations.
The second observation comes from the fact that there wereincomplete rescue requests during the Japan ﬂoods 2018,as discussed in the previous section. Many voluntary citizens added several pieces of information, such as hashtags,
to incomplete rescue requests to make them complete andretweeted them. On retweeting rescue requests, the citizens
often add comments praying for the safety of the victims. The
following tweet is a typical example:

3.4

Lessons Learned

This section summarizes four lessons learned through theclassiﬁcation experiments. First, the entire text of rescue requests should not be used for training classiﬁers because a partof a rescue request is often unrelated to the rescue request but
it is similar to tweets reporting disaster situations. Second,retweeted rescue requests should be carefully handled in thesame way as the ﬁrst lesson because persons retweeting rescue
requests often add several texts expressing sympathy for thevictims. If texts added on retweeting are unrelated to rescue

I hope the victim will be successfully rescued
ASAP. #rescue RT: Help me. My house is ﬂooded.
Another example is as follows:
Add #rescue in case you need rescue. RT: Helpme. My house is ﬂooded.

75International Workshop on Informatics ( IWIN 2022 )5

requests, they should be eliminated for training a classiﬁer.
Third, the entire text of tweets should not use for predicting
whether they are rescue requests or not for the same reason
behind the ﬁrst and the second lesson. One way to realize
this lesson is to classify tweets using annotation values (𝑎 𝑖,𝑡in Fig. 3) as well as a predicted result (𝑧 𝑡 ) of classiﬁers. Using
annotation values allows us to identify texts related to rescue
requests in a tweet and omit texts unrelated to the rescue requests, which cause misclassiﬁcation, as demonstrated in theprevious section. Finally, location names should be handledindependently for each disaster. One way to follow this lesson
is to convert location names in tweets to a particular reserved
word.

4

Conclusion

Circumstances of rescue requests during disasters have beenchanging. Citizens in need of help use social media, like Twitter, for expressing their rescue requests. To utilize such rescue
requests on social media, it is a key to understand real rescue
requests on social media. This study captured real disasterrelated tweets from several ﬂood disasters in 2018 and 2019
in Japan and analyzed the tweets. We observed that tweets
having rescue-related keywords are classiﬁed into the eightcategories, which include not only rescue requests but also
non-rescue requests. Furthermore, most of the rescue-related
tweets are unrelated to rescue requests. Next, we conductedpreliminary experiments of classifying rescue requests fromtweets. We built a classiﬁer based on GRU and LSTM-based
recurrent neural networks. The experiments revealed several
lessons for building classiﬁers of rescue requests.

Related Work
This section compares the present study with related studies.

REFERENCES

Several studies analyzed disaster-related social media posts,
especially focusing on Twitter. Alam et al. [9] analyzed
disaster-related tweets in three hurricanes in the U.S. in 2017,
i.e., Harvey, Irma, and Maria. They conducted both textual
content analysis and multimedia content analysis on tweets
from the three hurricanes. They deﬁne a taxonomy for disasterrelated tweets and this study inspired us to categorize rescuerelated tweets. Yang et al. [10] also analyzed tweets from thehurricane Harvey and proposed a framework to estimate thecredibility of events reported by tweets. They also captured
disaster-related tweets by searching Twitter for predeﬁned keywords. While those studies focuses on disaster-related tweets,
our study focuses on rescue-related tweets.

[1] M. Jahanian et al., “The evolving nature of disaster management in the internet and social media era,” in Proceedings of IEEE LANMAN, June 2018.
[2] Wikipedia, “2018 Japan ﬂoods (heavy rain of
July 2018).” https://en.wikipedia.org/wiki/2018_Japan_
ﬂoods.
[3] Wikipedia, “Typhoon hagibis (the 19th typhoon of2019 in japan).” https://en.wikipedia.org/wiki/Typhoon_
Hagibis_(2019).
[4] Japan Broadcasting Corporation (NHK), “Naganoprefecture collected rescue requests from Twitter andit contributed to saving about 50 victims.” https://bit.ly/
3741noG, Nov. 2019. (in Japanese).
[5] J. Ma, W. Gao, P. Mitra, S. Kwon, B. J. Jansen, K.-F.
Wong, and M. Cha, “Detecting rumors from microblogs
with recurrent neural networks,” in Processings of International Joint Conference on Artiﬁcial Intelligence,
pp. 3818–3824, July 2016.
[6] N. Ruchansky, S. Seo, and Y. Liu, “CSI: A hybrid deepmodel for fake news detection,” in Processings of International Conference on Information and Knowledge
Management, pp. 797–806, Nov. 2017.
[7] R. Kshirsagar, R. Morris, and S. R. Bowman, “Detecting and explaining crisis,” in Proceedings of ACM Workshop on Computer Linguuistics and Clinical Psychology,
pp. 66–73, Aug. 2017.
[8] B. Takahashi, E. Tandoc, and C. Carmichael, “Communicating on Twitter during a disaster: An analysis of
tweets during typhoon Haiyan in the Philippines,” Computers in Human Behavior, vol. 50, pp. 392–398, Sept.
2015.
[9] F. Alam, F. Oﬂi, M. Imran, and M. Aupetit, “A Twitter tale of three hurricanes: Harvey, Irma, and Maria,”
in Proceedings of International Conference on Information Systems for Crisis Response and Management, May
2018.
[10] J. Yang, M. Yu, H. Qin, M. Lu, and C. Yang, “A Twitter
data credibility framework—hurricane Harvey as a use

Next, we introduce studies that proposed classiﬁers for social media posts with machine learning. Studies in [5] and [6]propose classiﬁers based on neural networks for detecting fakeinformation or rumors. Though both studies adopt recurrent
neural networks, they develop slightly diﬀerent models. Ma
et al. [5] use intervals between social media posts reporting
the same event as input values for their classiﬁers because of
the fact that tweets are too short to identify their context withmachine learning. Ruchansky et al. [5] propose to use relation between users who post tweets regarding the same event
as well as texts of tweets to identify fake news. Kshirsagar etal. [7] propose a classiﬁer for detecting crises like suicide, selfharm, abuse, or eating disorders by using a recurrent neuralnetwork. They also develop an annotation mechanism for explaining how social media posts are related to the crises. Their
model uses texts of tweets as input values of their classiﬁer.
Our model is based on their model.
Finally, we introduce our previous study [19], which develops a communication framework for disaster management.
The key idea behind the proposed framework is to utilize social
media for collecting information regarding disaster situations.
The framework delivers social media posts to right persons byusing a machine learning technique. While the motivation of
the previous study is to develop a communication frameworkthat utilizes social media, the present study focuses on rescue
requests in social media.

76International Workshop on Informatics ( IWIN 2022 )case,” International Journal of Geo-Information, vol. 8,
Feb. 2019.
[11] Wikipedia, “Typhoon faxai (the 15th typhoon of 2019 injapan).” https://en.wikipedia.org/wiki/Typhoon_Faxai_(2019).
[12] Twitter, Inc., “Twitter api documentation—search
tweets.” https://developer.twitter.com/en/docs/tweets/
search/api-reference/get-search-tweets.
[13] Twitter,
Inc.
(@TwitterLifeline),“Twitterpost.”
https://twitter.com/TwitterLifeline/status/
1016519147738419201, July 2018.
[14] “MeCab: Yet another part-of-speech and morphologicalanalyzer.” https://taku910.github.io/mecab/.
[15] J. Pennington, R. Socher, and C. D. Manning, “Glove:
Global vectors for word representation,” in Proceedingsof Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543, Oct. 2014.
[16] S. Hochreiter and J. Schmidhuber, “Long short-term

memory,” Neural Computation, vol. 9, pp. 1735–1780,
Nov. 1997.
[17] D. Britz, A. Goldie, M.-T. Luong, and Q. Le, “Massive
exploration of neural machine translation architectures,”arXiv:1703.03906, Mar. 2017.
[18] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy,“Hierarchical attention networks for document classiﬁcation,” in Proceedings of Conference of the North
American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pp. 1480–
1489, June 2016.
[19] M. Jahanian, T. Hasegawa, Y. Kawabe, Y. Koizumi,
A. Magdy, M. Nishigaki, T. Ohki, and K. K. Ramakrishnan, “Direct: Disaster response coordination withtrusted volunteers,” in Proceedings of International Conference on Information and Communication Technologies for Disaster Management, Dec. 2019.

77

International Workshop on Informatics ( IWIN 2022 )How can SNS Data Contribute to Disaster Damage Assessment?
Kei Hiroi† , Akihito Kohiga‡ , and Yoichi Shinoda‡†

Disaster Prevention Research Institute, Kyoto University, Japan‡
Japan Advanced Institute of Science and Technology, Japanhiroi@dimsis.dpri.kyoto-u.ac.jp

Abstract - Reports of damage posted on SNS by residents ofdisaster-striken area at the time of a disaster are expected to beof a great use. They may be a valuable source of informationin areas where it is diﬃcult to install, operate, and maintainobservation device, or where devices are just missing. However,it has not yet been determined how they can be used eﬀectively
for damage assessment. Therefore, a study on a complementary
use of SNS data for ﬂood analysis using data assimilation to
improve damage assessment is in urgent demand. In this paper,
we report the evaluation results of data assimilation assumingthat SNS data can be collected stably, and discuss about the
impact of error values.

due to the water overﬂowing from small rivers and waterways
spreading throughout the city in a complicated manner and dueto the water from the rain that cannot be completely drained.
Therefore, it is necessary to calculate the high temporal-spatial
ﬂood level estimation that ﬂuctuates according to the rainfall
situation to know the risk with a high temporal resolution for
guiding evacuation behavior. Our research goal is to detect
ﬂooding as time-series data with only a limited number of observation devices.
This paper investigate whether SNS data can be used to assess
ﬂooding. SNS data is said to be eﬀective to ﬁgure out ﬂood levels
even in places where it is diﬃcult to install, operate, and manage
observation devices. Although there have been many studies on
ﬂood damage detection using SNS, their eﬀectiveness has notbeen clariﬁed, and the amount and content of data collected are
not ﬁxed depending on the case of ﬂood damage. This paper
validates the SNS data using the following procedure based on
our state-space model(SSM), which is our previous research.
The purpose of SNS Data Validation is to investigate whether
SNS data can contribute to the accuracy of ﬂood assessment andunder what conditions SNS data can improve the accuracy.
Then we also suggest a system for an appropriate SNS use casethat could improve the accuracy of ﬂooding assessment further
by adding SNS data as well as observation data, and calculate
ﬂooding conditions for the entire aﬀected area. In order torealize this system, it necessary to validate the eﬀectiveness of
the SNS data.
We generate quantiﬁed SNS data from ﬂood simulations inthis paper. To generate the SNS data, we use time-series datacollected from observation devices at multiple locations. The
results of ﬂood analysis simulation were assimilated using thetime-series data to simulate the SNS locations, the timing of
postings, and numerical ﬂood levels . Then, we append errors
to the simulated SNS data. Afterward, we regenerate a ﬂood
level on the observation location based on the simulated SNS
data with errors and examine the accuracy of the errors on thedata assimilation accuracy.

Keywords: ﬂood estimation, state-space model, temporalspatial analysis, data assimilation

1 Introduction
There are concerns that the risk of ﬂoods on a global scale
will intensify. The IPCC’s Fifth Assessment Report stated that
global warming is gradually progressing, and it is likely that thefrequency and intensity of rainfall will change accordingly [1].
There are already many areas where the frequency and intensityof heavy rain and ﬂooding are increasing worldwide [2] [3]. As
part of measures against ﬂood damage, observing rainfall, river,and ﬂooding, and grasping the changing situation of rainfall andriver and its inﬂuence enables people to judge what action theyshould take and take eﬀective steps to prevent or mitigate damage. Many previous studies have attempted to estimate ﬂood
risks using the vulnerability of the area. For example, in [4], theﬂood risk in the city was estimated with a detailed spatial resolution of approximately 2 meters. [5] [6] conducted research toestimate index-based ﬂood risk using a theoretical hydraulic engineering model. Furthermore, a Chinese case in [7] examinedrecognizing risks with the situation of 1997’s Red River ﬂood.
Studies are actively conducted to correctly analyze risks by presenting risks to people in aﬀected areas and raising awarenessof individual ﬂood risks to lead to mitigation behavior [8] [9].
However, these previous studies are not temporal estimation
methods but rather only static estimation approaches to calculate
the maximum water level. Such a static estimation result can besaid to be a risk estimation in which the risk value might change
due to ﬂuctuations in rainfall. Considering evacuation behavior, dynamic risk estimation is required because ﬂood situations
change very rapidly with the ﬂooding phenomenon over streets

2 Related Works
2.1 Flood Monitoring
Traditional river sensors [10], [11] have succeeded in detecting disaster signs of large-scale rivers, which have both theadvantage of stable monitoring and the disadvantage of installa-

79International Workshop on Informatics ( IWIN 2022 )tion limitation (i.e., very big equipment, high installation cost of
several million dollars, and complicated pre-conﬁguration). Improvements in installation limitations suggest the possibility of alarge number of sensor installations and reliable detection to improve monitoring sensors with higher resolution. Approaching
ﬂood prediction, hydrological techniques [12] or Artiﬁcial Neural Networks [13], [14] are proposed as high prediction methods.
Predicting rising river levels has resulted in highly precise riverinﬂow in the view of large-scale river analysis. However, theseprevious methods cannot predict the ﬂooding of smaller rivers
and waterways. This is because complex water ﬂow prediction
requires analyzing complicated relationships among a plurality of conﬂuent rivers and factoring in the impact of rainfall
dynamics.

They report that their system shows potential disaster locations
with an average accuracy of 0.93. Teodorescu [21] designed
a method for the analysis of SNS for the purpose of forecasting needs and measures for relief and mitigation. His method
analyzes SNS related time-series with the aim of establishingcorrelations between characteristics of the disaster and the SNS
response.

2.4 Issues and Approaches
In order to comprehend safe evacuation routes, it is important
to ﬁgure out the situation with regard to roads in urban areas.
Currently, ﬂood damage assessment is based on two methods:
numerical simulation (e.g., ﬂood analysis) and monitoring using
low-resolution ground observation data (precipitation and riverwater levels).
Numerical simulations are based on diﬀerential equations for
ﬂood ﬂow in urban areas for a given amount of precipitation, andthe maximum ﬂood level in a detailed space (e.g., a 10-m grid)is calculated. Based on the calculated results, hazardous areas in
the event of heavy rainfall are published. However, the analysis
uses an ideal model that assumes ﬁxed parameters such as theamount of precipitation, its runoﬀ coeﬃcient, and the outﬂow
conditions of drainage channels. Therefore, in urban areas withcomplex rainfall distribution and land use, the analysis results
and actual ﬂood levels will diﬀer. As the result, ﬂooding of roads
occurs prior to the announcement of warnings and evacuation
information, leading to damage.
On the other hand, monitoring establishes thresholds for dangerous water levels at speciﬁc locations where there is concern
about road underpasses and river breaches. This is a situation
monitoring method to detect the occurrence of ﬂooding basedon observation data. This method is easy to assess the actual
damage but has the problem of limited observation points.
SNS data is expected to solve these monitoring limitations.
As indicated in the previous section, the importance of SNS
in ﬂood damage detection has long been known and has been
applied in many ﬂood damage cases. However, there is a fundamental problem with water damage detection using SNS. That
is, SNS data is not necessarily posted in every case. While it
may work eﬀectively in ﬂoods with a high number of postings,
it is highly likely that it will not be as accurate as reported inﬂoods with a low number of postings. In particular, it may
be diﬃcult to post while ensuring safety in heavily damaged
areas, and communication problems may prevent posting. We
convinced that these problems are obstacles to the eﬀective useof SNS for ﬂood damage detection. Therefore, in this paper,
we investigate how much SNS data on the number of postings,
their contents, and the timing of postings would be eﬀective fordamage assessment.
We intend to develop a system for improving the accuracy of
ﬂood level estimation through data assimilation using heterogeneous data as a prospect for the investigation in this paper.
We have previously proposed a method for estimating the expansion process of ﬂooding by applying data assimilation using

2.2 Risk Estimation with Higher Spatial
Resolution
Various studies have already attempted to generate information on which place is dangerous. In case studies such as[15] [16], research aimed at presenting the risk on a map. Sinnakaudan et al. [15] developed an ArcView GIS extension as aneﬃcient and interactive spatial decision support tool for ﬂood
risk analysis. Their extension has the capability to analyze thecomputed water surface proﬁles and produce a related ﬂood map
for the Pari River in ArcView GIS. As another GIS-based ﬂood
risk assessment, Lyu et al. [16] studied the Guangzhou metro
system’s vulnerability. Their results showed the vulnerabilityof several metro stations using the ﬂood event that occurred in
Guangzhou on May 10, 2016.
Some studies have already proposed modeling methods thatstrictly collect data as input data [4] [17] [18]. Ernst et al. [4]presented a microscale ﬂood risk analysis procedure as a 2-meter
grid, relying on detailed 2D inundation modeling and on a highresolution topographic and land-use database. However, detailed risk estimation requires detailed data measurement, suchas laser altimeter data, and it is not realistic to measure such datain all areas.

2.3 Flood Ditection through Social Networking
Service
One other way to learn about ﬂooded areas is through social
networking services (SNS). Kim et al. [19] stated that social
networking is the fourth most popular information source foraccessing emergency information. Then they applied social
network analysis to convert emergency social network data into
knowledge for the 2016 ﬂood in Louisiana. Their objective
was to support emergency agencies develop their social media
operation strategies for a disaster mitigation plan. This reference
explores patterns of interaction between online users and disaster
responses.
Suﬁ et al. [20] designed a disaster monitoring system on social media feeds related to disasters through AI- and NLP-based
sentiment analysis. Their system has a mean accuracy of 0.05.

80International Workshop on Informatics ( IWIN 2022 )observation time-series data as heterogeneous data to simulate
ﬂood analysis. Our estimation method showed a signiﬁcant improvement, with an error of less than 9 cm. We are planning to
add SNS data to this estimation method to further improve its
accuracy and to develop a system to present the ﬂood disastersituation of the entire aﬀected area. However, although there
have been many studies on ﬂood damage detection using SNS,the amount and content of the data collected are not well deﬁned
for each ﬂood damage case. The eﬀectiveness of using SNS for
ﬂood level estimation is not yet clariﬁed. Therefore, with theaim of developing the system to accurately estimate and present
the ﬂooding situation of the entire aﬀected area, this paper investigates the relationship between SNS data and the accuracyof ﬂooding estimation through simulations.

(a) SSM using Observation Data

3 Proposal of Data Assimilation Method for
Improving Accuracy of Flood Estimation(b)SSMusingSNSData

Figure 1: SNS
Data
Validationfor
Data
Assimilation(st1,...,i :observation location, mdk :SNS data posting
location)

3.1 SNS Data Validation for Data Assimilation
The basic idea of SNS Data Validation for Data Assimilationin this paper is illustrated in Figure 1(a), state-space model: SSM
using SNS Data. The purpose of SNS Data Validation is to investigate whether SNS data can contribute to precise estimationof ﬂood assessment and under what conditions SNS data can improve the accuracy. Since we are unable to determine the error
rate contained in the SNS data collected at the time of ﬂooding,we generate altinative quantiﬁed SNS data from simulations. To
generate the SNS data, we use time-series data collected fromobservation devices at multiple locations. The results of ﬂood
analysis simulation are assimilated with the time-series data tosimulate the SNS locations, the timing of postings, and numerical ﬂood levels (Figure 1(b)( i )). For this data assimilation, weused a spatial-temporal state-space model proposed in a previous our study [22](Figure 1(a) SSM using Observation Data).
Note that, a spatial-temporal state-space model in this paper, isapplied without using the waterway and sewer data used in theprevious study [22], due to these data are generally limited inavailability.
Then, we append errors to the simulated SNS data (Figure 1(b)(ii)). This assumes the errors in location, time, andwater level value that are present in the textual data of the actual
SNS data. Afterward, we regenerate a ﬂood level on the observation location based on the simulated SNS data with errors
and examine the accuracy of the errors on the data assimilation accuracy (Figure 1(b)(iii)). This regeneration uses a statespace model that applies the state-space model of the previous
study [22] toward the spatial direction. Here, if there is a small
diﬀerence between the time-series data and the data assimilation results on observation location, it can be expected that the
SNS data is applicable to ﬂood assessment by data assimilation.
Contrary to this, if the data assimilation accuracy is low despite
the small error appended to the simulated SNS data, then there
are problems in the use of SNS data.

3.1.1 Process( i ) Simulated SNS data for Flood Analysis
Simulation
The process ﬂow is shown in Figure 2. In process( i ), theresults of the ﬂood analysis simulation are assimilated with timeseries data collected from observation locations to simulate thelocations, timing of posting, and ﬂood water level values. Thissection outlines the spatial-temporal state-space model used inour data assimilation. The basic ﬂood analysis is based on theconventional simulation using a surface ﬂooding model. Thismethod calculates the amount of runoﬀ at each grid location
by expressing the ﬂooding ﬂow as a continuous equation andmotion equations.
A continuous equation is deﬁned as follows.
∂h ∂M
∂N
+
+
=0
∂t
∂x
∂y

(1)

Motion equations are
∂M
∂U M
∂V M
∂H
1
+
+
+ gh
+ τx (b) = 0
∂t
∂x
∂y
∂x
ρ

(2)

∂N
∂U N
∂V N
∂H
1
+
+
+ gh
+ τy (b) = 0
∂t
∂x
∂y
∂y
ρ

(3)

Each parameter is deﬁned as t: time, H: water level, h:ﬂood level, U : ﬂow velocity (X direction), V : ﬂow velocity (Ydirection), g: gravity acceleration, ρ: water density, M : ﬂux (Xdirection), N : ﬂux (Y direction) (M = uh, N = vh).
Here, shear force in x direction τx (b) and shear force in ydirection τy (b) are deﬁned as follows.
√ρgn2 U U 2 + V 2
τx (b) =
1
h3

81

(4)

International Workshop on Informatics ( IWIN 2022 )   
 

(i)rt 
 

(i)       xt                    (i)                        

τy (b) =√U2 + V 2(5)1
h3
The roughness coeﬃcient n (The resistance value of riverwater to touch obstacles) can be expressed as follows, taking
into account the inﬂuence of the building.
n2 = n20 + 0.020 ×

4
θ
× h3
100 − θ(6)

（n:bottom roughness coeﬃcient, no :composition equivalent roughness coeﬃcient,θ:building occupancy rate)
Equation (1)-(3) calculates ﬂood level h for each grid, accounting for the runoﬀ from the inside of the sewer line to theground surface and the ﬂooding to the ground surface fromrainwater. For the above equations, the inﬂow into each grid
represents the ﬂux into each grid from adjacent grids and theeﬀect of buildings on the inﬂow in each grid.
We deﬁne D as the two-dimensional space corresponding to
the region of interest and divide D into m grids of d meters
each. Let si ∈ D denote the location coordinates of each grid(si is denoted by i). Using equation (1)-(3), we calculate ht (i)for ﬂood level of each grid at time t.
Then, using the state-space model, we estimate the ﬂood level(i)of grid sk from the observations yt collected at the observation location at time t. Grid sk (k = 1, 2, 3...., m) is the location
indicated by the SNS data. The state-space model is represented
by two types of observation equations; the ﬂood analysis simulation result ht (i) at grid si , and the diﬀerence between theﬂood analysis simulation and the observed value at the observation location. This state-space model is deﬁned by the equations(7)(8)(9).
(i)

yt

(i)

(i)

(i)

(i)

= St rt + Gt xt + et

(i)

(i)

(8)

= xt−1 + ut

(9)
(i)

3.1.2 Process(ii) Appending Errors to Simulated SNS Data

Figure 2: SNS Data Process Flow (st1,...,i :observation location,
mdk :SNS data posting location)

ρgn2 V(i)

The rt denotes the state at time t and vt denotes noise.
(i) (i)(i)
The term Gt xt represents the total inﬂow/outﬂow, and xt
is the diﬀerence between the ﬂood analysis simulation results(i)and the observed values. The ut denotes the noise at time t.
(i)
Gt is the adjacency matrix indicating the spatial component.

 
       (i)= rt−1 + vt

(7)

82

Now, in process(ii), we append the error component to thesimulated SNS data. Actual SNS data shows a variety of representations of water levels. For example, “It’s ﬂooded up to myknees," “The car is ﬂooded," or pictures are posted with comments such as “It’s raining so hard." This paper considers SNS
data that expresses water levels in words. The ﬂood level ht (k)indicated by the SNS data is assumed to contain an error component e. The representation type of the SNS data is considered
as a factor that causes errors due to the quantiﬁcation of the SNS
data (Table 1). We assume that the type of data representation
occurs for each of the posted location, time, and water level values. For water level values, SNS data can be expressed in theform of measurement, comparison with an object, or descriptionof the situation. In the case of measurement, it is considered tobe measured at a guess, which results in a diﬀerence from theactual water level.
When comparing with objects, water levels are explained
based on objects such as knee height or up to the ankles, but
the sizes of these objects vary from each user, so even if quantiﬁed, they diﬀer from the actual water level. Although this is only
an assumption, the average length below the knee for Japanesepeople is 46.7 cm in males and 42.9 cm in females, a diﬀerenceof about 4 cm even in the average value. In the description of
the situation, the data explains mostly describes the aspect of
ﬂooding, with little mention of water levels, in consequence, it
is expected that quantiﬁcation itself is often diﬃcult.
For information representation of location, the following information can be considered: GPS, address, road/river, landmark, and city/town name. When GPS is given to SNS data,
the exact location at the time of posting can be reﬂected in thequantiﬁed SNS data. However, if the location indicated by theposted message diﬀers from the location at the time of posting,there is an error from the posted location. The same error occurs
for other types of data representations. In some cases, addressesof ﬂooded areas are posted for rescue in ﬂooding situations.
Although there may be an error of a few meters, the location
information would be approximately correct. If a road/river is
described as a location, it is considered diﬃcult to determine
the exact location from the text content itself. In the case oflandmarks, the data may indicate the location in front of thelandmark, whereas it is also possible that the data indicates theInternational Workshop on Informatics ( IWIN 2022 )the actual water level h(k  ) is shown as the eﬀect of the error
component eΔv,Δl,Δt (k) + ζΔl ,Δt (k) on the data assimilation
method. This paper validates the SNS data eﬀectiveness bycomparing ﬂood level h (i) regenerated from the simulated SNS
data at location k with the actual observed water level h(i).

Table 1: SNS Data Representation Types

Data
Level

Location

Time

Types
measurementcomparison
descriptionGPS
addressroad/riverlandmarkcity/town nametimestamp
comparison
range

Example30cm
knee heightlooks like river
34.9104, 135.8002
1-1 Gokasho,Uji-city,
Route 24
In fromt of Kyoto Station
Uji city2022/7/8/21:00
just now
about 21:00

3.2

Evaluation and Discussion

3.2.1

Evaluation Purpose and Flood Case

This evaluation investigates the eﬀect of the data representationthat text-based SNSs have on the quantiﬁcation of ﬂood levels.
SNS data reporting water levels during ﬂoods often include
measurement of water levels or are expressed in comparison toa body of water. In addition to water levels, location and time
also contain errors in quantiﬁcation, depending on the type ofrepresentation. We investigate the impact of SNS data with theseerrors on statistical ﬂood analysis, such as data assimilation.
Based on the results, we will discuss what form of SNS data
would contribute to ﬂood assessments.
Our evaluation is based on SNS data simulated by a statespace model with observed data of ﬂood. We append errors to
the simulated SNS data and observe the eﬀects of the errors.
Then, we determine errors that could occur due to the quantiﬁcation of the SNS data, as shown in Figure 3. This paper uses
ﬂood observations collected in Tsushima City, Aichi Prefecture,
Japan, from October 22 to 23, 2017, due to rainfall caused by
Typhoon No. 23. A rainfall of 32 mm/h was observed at 23:00 at
the nearest precipitation gauge (Aisai Observatory, Aichi Prefecture) to our target area. Using this rainfall data as input values,
we calculated a ﬂood analysis simulation with the ﬂood analysis simulation NILIM 2.0. With the results of the simulated
ﬂood analysis, we apply the ﬂood estimation method using thestate-space model. For generating simulated SNS data, the statespatial model with observation data uses water level observation
data at waterways every 5 minutes collected from pressure-type
sensors installed at four locations in the target area.
Detailed information about the observation locations is presented as follows. Observation locations 1 and 2, and 3 and 4
are on the same waterway. The distance between observation
locations 1 and 2 is approximately 500 meters, and the distance
between locations 3 and 4 is approximately 600 meters. Observation location 4 is connected to the sewer. Two waterways
are approximately 500 meters apart. There are no ﬂoodgates
between the observation locations. The diﬀerence in elevationin this area is within 0.30 meters, and the elevation values (Elevation model by Geospatial Information Authority of Japan)
are equal at all four observation locations. The heights from thebottom of the waterway to the road are 1.01, 1.14, 0.72, and 1.28
meters, and the usual water levels are 0.06, 0.07, 0.16, and 0.31
meters. On the day of the ﬂood, the installed sensor devices
showed, water overﬂowing the waterways and ﬂood levels of up
to 0.26, 0.25, 0.63, and 0.48 meters above the road.

location where the landmark is visible, in which case a large
error of about 100 meters or more would occur. For city/town
names, we consider a signiﬁcant error of several kilometers in
identifying the location, due to the wide range of areas indicatedby the data.
For the representation of time information, there are threeforms to consider timestamp, comparison, and date/time range.
The timestamp can show the exact date and time in the simulated
SNS data. As for comparison, it is considered to be a popularform of time, nevertheless representations such as "just now" arelikely to include an error of several tens of minutes, as the senseof time diﬀers among individuals. Additionally, it is assumed
that there are many cases describing a range of dates/times,
such as "around 21:00" or "this evening". While errors can be
expected to be small for numerical time representation, in thecase of "night" and other representations, errors are likely to bein the order of several hours. Furthermore, as with location,
there are cases in which the time of posting also diﬀers from thetime of ﬂooding, and this diﬀerence may result in a signiﬁcanterror in the time representation.
We deﬁne the error components at location sk , resulting
from the quantiﬁcation, as the error in the representation typeeΔv,Δl,Δt (k), the error relative to the posting location/timeζΔl (k), and the error from the time of posting ζΔt (k).
3.1.3 Process(iii) Flood Level Estimation and its Validation
Process(iii) regenerates the time-series data for the observation
location to investigate the error impact on the data assimilation
accuracy based on the simulated SNS data with error ht (k) +eΔv,Δl,Δt (k) + ζΔl ,Δt (k). Here, we use a state-space modelthat utilizes the model detailed in 3.1.1 in the spatial direction.
For equation (7)(8)(9), at time t, the simulated SNS data ht (k)+(i)
eΔv,Δl,Δt (k)+ζΔl ,Δt (k) is substituted into yt to estimate the 
ﬂood level h (k ) of the target location sk . Simulated SNS dataon a particular location is not continuous time-series data. Thus,
there is only one t in the simulated SNS data, and the state-spacemodel in process(iii) is applied only in the spatial direction.
The diﬀerence between the restored water level h (k  ) and83

International Workshop on Informatics ( IWIN 2022 )

 



           





(a) st1 Estimation Value(b) st2 Estimation Value(c) st3 Estimation Value(d) st4 Estimation Value"!  #!   $!
 

%!    Figure 3: Appending Errors to Simulated SNS Data

3.2.2 Evaluation Procedure

Figure 4: Preliminary result: SSM with Observation Data

The evaluation randomly extracts simulated SNS data according
to the Number of SNS data (8,24,48,80,120,435) from the Posted
area size (10,20,30,40,50, 100[meters]). Errors appending to thesimulated SNS data are parameterized to indicate ﬂuctuations.
The parameters that indicate the ﬂuctuation of each error are
0,10,20,30[cm]. We begin by using the number of SNS data
posted and area size where SNS data were posted as two common
parameters. Subsequently, we adopt the parameters to generate
the errors as water level errors. The error, resulting from theinformation representation of the water level value, could be theﬂuctuation of the value when the water level is estimated fromthe posted SNS data. Although this ﬂuctuation can have a rangeof diﬀerent values, this paper assumes a ﬁxed parameter for theerror in order to verify the eﬀect of the SNS data. We add thevalue indicated by the parameter: water level value error to thewater level in the simulated SNS data.
We repeat the above procedure ﬁve times and compute themean value of the estimation. Simulated SNSs appended withthe errors are applied to the state-space model to calculate theestimated water levels for the four observation sites with timeseries data.

Table 2: RSME: SSM with Observation Data

Location
st1
st2
st3
st4

mean
0.19
0.20
0.38
0.28

minimum
0.12
0.15
0.28
0.18

maximun
0.24
0.23
0.43
0.33

accurate using the data from st3 , with an average RMSE of 0.19
meters. The estimated water level for st2 in Figure 8 (b) was also
the most accurate, with an average RMSE of 0.20 meters, using
data from st3 . Both estimation for st3 in Figure 8 (c) and st4
in and Figure 8 (d) using data from other observation locations
were less accurate, with average RMSEs of 0.38 meters and0.28 meters. The results of either estimation resulted in a large
diﬀerence from the water level at the ﬂood peak. These results
are due to the failure of the "SSM using Observation Data"
to follow the rising and falling water levels. In our previous
study [22], the maximum error was 9 cm because we included
data of waterways and sewers. However, this paper does not
use those data, in order to apply our system in areas where itis diﬃcult to collect the data. For all observation locations,
we found that estimation using data from distant observation
locations resulted in large errors.

Preliminary result: SSM with Observation Data For comparing the accuracy, this section shows the estimated ﬂood level(i)
ltT ,k using the state-space model with observation data (Figure 1 SSM using Observation Data). Of the four observation
locations, we apply the observation data from three locations(from time t = 0 to t = 50) to "SSM using Observation Data"to estimate the ﬂood water level at the remaining one location(the water level at st1 is estimated using the water level timeseries data at st2 , st3 , and st4 ). The estimation results are shownin Figures 8. The black lines indicate the actual observed ﬂood
water levels, and the magenta, green, blue, and orange dots indicate the estimated water levels using the state-space model. Root
Mean Squared Error: RMSE between the mean of the estimation
results and the actual observed values is shown in Table 2.
In Figure 8 (a), the estimated water level at st1 was the mostResult This section shows the results of applying the statespace model with error values appended to the simulated SNS
data as Water Level Value Error in Table 3. When no error values
were added, the mean values of st1 and st2 showed the smallest
RSME. The larger error value resulted in a larger RSME. On theother hand, st3 and st4 showed a large RSME in mean value.
Figure 13 compares the time-series data of the actual observations with the mean value of the estimated values. For st1
and st2 , as the water level changes, the water level estimatedfrom the SNS data also changes and shows little diﬀerence from84

International Workshop on Informatics ( IWIN 2022 )Table 3: RSME: Result(3)：Water Level Value Error

3.2.3 Discussion

Value Error [meter]mean
st1
minimunmaximunmean
st2minimunmaximunmean
st3
minimunmaximunmean
minimunst4
maximun

We found that as long as the error is small and the ratio of fakedata is small, the estimation accuracy does not reduce signiﬁcantly. All evaluations showed signiﬁcant improvements compared to the state-space model with observations. This is a resultof the fact that the observation locations are about 500 meters
away from the estimated locations, whereas the simulated SNS
data are within 100 meters, allowing for more accurate estimation. Unlike time-series data, SNS data is sparse in the time
direction, although it is eﬀective for estimation if collected at
locations that are nearby to the estimated location.
In this evaluation, we also treated the number of SNS dataand posted area size as parameters. These two parameters did
not signiﬁcantly aﬀect the estimation results. Hence, we foundthat even if the number of SNS data is small, locations near theestimation location can be estimated with suﬃcient accuracy.
For st1 and st2 , the smaller the parameters, the smaller theerror. These results indicate that time lag and location gap areallowable within the range of values of this evaluation and thata small number of errors in water level values and fake data
prevents a large error. For st3 and st4 , the errors are large even
without adding the water level error, requiring investigation as
to the cause. Since this paper covers only four observation
locations, we assume that this is due to the eﬀect of low waterlevels observed at the other locations. Process in 3.1.1 calculated
a low accuracy estimation of the observed location. As a result,the simulated SNS data based on the estimation is also calculated
lower than the actual water level. Therefore, we need to improve
the state-space model itself. One idea is to apply observations
and SNS data together to the state-space model and implementa Kalman ﬁlter for locations where time-series data is available.
The purpose of our research is to investigate the conditions
under which SNS can contribute to ﬂood estimation: how SNS
data, if available, can improve the accuracy of ﬂood analysis. In
order to achieve the system in Figure ?? Phase 1 validated theeﬀectiveness of SNS data in improving the accuracy of ﬂooding
assessments as SNS Data Validation. The results of the assessment using simulated SNS data showed that a certain degree of
error was allowable and that the accuracy was better than thatof estimation using observation data collected over a longer distance. We conclude that further processing improvements areneeded, such as removing larger errors by estimation using a
combination of observed and SNS data.

0.00
0.04
0.00
0.13
0.04
0.00
0.26
0.21
0.00
0.30
0.13
0.00
0.22

0.10
0.05
0.00
0.17
0.03
0.00
0.15
0.17
0.03
0.28
0.09
0.00
0.18

0.20
0.10
0.01
0.27
0.07
0.00
0.25
0.13
0.00
0.25
0.06
0.00
0.37

0.30
0.15
0.06
0.30
0.12
0.03
0.39
0.08
0.00
0.22
0.05
0.00
0.52

(a) st1 Estimation Value(b) st2 Estimation Value(c) st3 Estimation Value(d) st4 Estimation Value

Figure 5: Result(3)：Water Level Value Error (Mean Value)

the actual observation. In addition, larger error values tend toprovide larger estimates of the results. Meanwhile, ST3 and ST4
result in a diﬀerence of about 0.20 meters between the estimatedresult (error value: 0) and the actual water level at the peak of
the ﬂooding. We consider that the estimated values were closer
to the actual observed value when the error value was increased
since there was such a large diﬀerence at the error value: 0. One
possible reason for this is that the values of st3 and st4 were
estimated lower due to the use of data from other observation
locations when generating the simulated SNS data.
Table 3 shows that the minimum value is almost 0.00 meters, even when large values are added as errors. However, themaximum value results in an extremely large value. In st1 and
st2 , the mean values of RSME do not change signiﬁcantly after
adding the error value, contrary to this the error value: 0.30
meters in st4 shows an RSME of 0.52 meters, a large error.
Accordingly, this would require a data collection method andanalysis process to reduce the error in values, and a process tovalidate the reliability when large water levels are estimated.

4 Conclusion
This study investigated whether SNS data can be used to assess
ﬂooding. Because we believe that SNS data can be eﬀective toﬁgure out ﬂood levels even in places where it is diﬃcult to install,
operate, and manage observation devices. Although there havebeen many studies on ﬂood damage detection using SNS, theireﬀectiveness has not been clariﬁed, and the amount and content
of data collected are not ﬁxed depending on the case of ﬂood
damage.

85

International Workshop on Informatics ( IWIN 2022 )Resolution Geographic Data, Natural Hazards, Springer,
Vol.55, No.2, pp.181–209, 2010.
[5] Saudi, A. S. M., Ridzuan, I. S. D., Balakrishnan, A., Azid,A., Shukor, D. M. A., Rizman, Z. I., New Flood Risk Index
in Tropical Area Generated by using SPC Technique, Journal of Fundamental and Applied Sciences, Vol.9, No.4S,
pp.828–850, 2017.
[6] Silva, S. F., Martinho, M., Capitão, R., Reis, T., Fortes,
C. J., Ferreira, J. C., An Index-based Method for Coastalﬂood Risk Assessment in Low-lying Areas (Costa de Caparica, Portugal), Ocean & Coastal Management, Elsevier,Vol.144, pp.90–104, 2017.
[7] Burn, D. H., Perceptions of Flood Risk: A Case Studyof the Red River Flood of 1997, Water Resources Research, Wiley Online Library, Vol.35, No.11, pp.3451–
3458, 1999.
[8] Hall, J. W., Meadowcroft, I. C., Sayers, P. B., Bramley,
M. E., Integrated Flood Risk Management in England andWales, Natural Hazards Review, American Society of Civil
Engineers, Vol.4, No.3, pp.126–135, 2003.
[9] Miceli, R., Sotgiu, I., Settanni, M., Disaster Preparedness
and Perception of Flood Risk: A Study in an Alpine Valley
in Italy, Journal of Environmental Psychology, Elsevier,Vol.28, No.2, pp.164–173, 2008.
[10] Pulvirentia, L., Chinib, M., Pierdiccaa, N., Guerrieroc, L.,
Ferrazzolic, P., Flood Monitoring using Multi-temporal
COSMO-SkyMed Data: Image Segmentation and Signature Interpretation, Remote Sensing of Environment,
Vol.115, No.4, pp.990-1002, 2011.
[11] Basha,E. A., Ravela, S., Rus, D., Model-based Monitoring
for Early Warning Flood Detection, In Proceedings of the
6th ACM conference on Embedded network sensor systems(SenSys), pp.295-308, 2008.
[12] Elshorbagy, A., Corzo, G., Srinivasulu, S., Solomatine, D.,Experimental Investigation of the Predictive Capabilities
of Data Driven Modeling Techniques in Hydrology―part2: Application, Hydrology and Earth System Sciences,
Vol.14, pp.1943-1961, 2010.
[13] Raﬁeeinasab, A., Norouzi, A., Kim, S., Habibi, H., Nazari,B., Seo, D., Lee, H., Cosgrove, B., Cui, Z., Toward Highresolution Flash Flood Prediction in Large Urban Areas– Analysis of Sensitivity to Spatiotemporal Resolution ofRainfall Input and Hydrologic Modeling, Journal of Hydrology, Vol.531, part 2, pp.370-388, 2015.
[14] Ruslan,F. A., Samad, A. M., Zain, Z. M., Adnan, R., Flood
Prediction using NARX Neural Network and EKF Prediction Technique: A Comparative Study, In Proceeding of theIEEE 3rd International Conference on System Engineeringand Technology (ICSET), pp.203-208, 2013.
[15] Sinnakaudan, S. K., Ghani, A. A., Ahmad, M. S. S., Zakaria, N. A., Flood Risk Mapping for Pari River Incorporating Sediment Transport, Environmental Modelling &Software, Elsevier, Vol.18, No.2, pp.119–130, 2003.
[16] Lyu, H. M., Sun, W. J., Shen, S. L., Arulrajah, A., Flood

This paper validates the SNS data using the following procedure based on our state-space model. The purpose of SNS Data
Validation is to investigate whether SNS data can contribute to
the accuracy of ﬂood assessment and under what conditions SNS
data can improve the accuracy. Since we were unable to determine the error rate contained in the SNS data collected at the timeof ﬂooding, we generate quantiﬁed SNS data from simulationsin this paper. To generate the SNS data, we used time-series
data collected from observation devices at multiple locations.
The results of ﬂood analysis simulation were assimilated using
the time-series data to simulate numerical ﬂood levels. Then,
we appended errors to the simulated SNS data . Afterward, we
regenerated a ﬂood level on the observation location based on
the simulated SNS data with errors and examine the accuracy ofthe errors on the data assimilation accuracy.
Estimation results show that if water levels contained errors,
we found that the errors were small and that if the ratio of fakedata was small, the estimation accuracy would not be significantly reduced. All evaluations showed signiﬁcant improvements compared to the state-space model with observations.
Unlike time-series data, SNS data is sparse in the time direction,although it is eﬀective for estimation if collected at locations
that are nearby to the estimated location. However, some of thesensors have large errors in the process of calculating simulated
SNS data. This error can be derived from the fact that waterway
and sewer data were not used in the state-space model. Since
waterway and sewer data are diﬃcult to collect, we plan to resolve this issue statistically using a Kalman ﬁlter. After a further
improvement in accuracy, we will develop and implement Phase
2: Flood Assessment & Promotion Requirement and Phase 3:
Flood Assessment.

Acknowledgement
This work was supported by JST, PRESTO Grant Number
JPMJPR2036, Japan.

REFERENCES[1] Intergovernmental Panel on Climate Change Fifth Assessment Report (AR5), Retrieved April 17, 2018, fromhttps://www.ipcc.ch/report/ar5/, 2018.
[2] Milly, P. Christopher D., Wetherald, Richard T., Dunne, K.
A., Delworth, Thomas L, Increasing Risk of Great Floods
in a Changing Climate, Nature, Nature Publishing Group,
Vol.415, No.6871, page514, 2002.
[3] Hirabayashi, Y., Mahendran, R., Koirala, S., andKonoshima, L., Yamazaki, D., Watanabe, S., Kim, H.,
and Kanae, S., Global Flood Risk under Climate Change,
Nature Climate Change, Nature Publishing Group, Vol.3,
No.9, page816, 2013.
[4] Ernst, J., Dewals, B. J., Detrembleur, S., Archambeau, P.,
Erpicum, S., Pirotton, M., Micro-scale Flood Risk Analysis based on Detailed 2D Hydraulic Modelling and High86

International Workshop on Informatics ( IWIN 2022 )Risk Assessment in Metro Systems of Mega-cities using a
GIS-based Modeling Approach, Science of the Total Environment, Elsevier, Vol.626, pp.1012–1025, 2018.
[17] Hunter, N. M., Bates, P. D., Horritt, M. S., Wilson, M. D.,
Simple Spatially-distributed Models for Predicting Flood
Inundation: A Review, Geomorphology, Elsevier, Vol.90,
No.3-4, pp.208–225, 2007.
[18] Horritt, M. S., Bates, P. D., Evaluation of 1D and 2D
Numerical Models for Predicting River Flood Inundation,
Journal of Hydrology, Elsevier, Vol.268, No.1-4, pp.87–
99, 2002.
[19] Kim, J., Hastak, M., Social Network Analysis: Characteristics of Online Social Networks after a Disaster, International Journal of Information Management, Vol.38, No.1,
pp.86–96, 2018.
[20] Suﬁ, F. K., Khalil, I., Automated Disaster Monitoring From
Social Media Posts Using AI-Based Location Intelligenceand Sentiment Analysis, IEEE Transactions on Computational Social Systems, pp.1–11, 2022.
[21] Teodorescu, H. N., Emergency-Related, Social Network
Time Series: Description and Analysis, Time Series Analysis and Forecasting, Springer International Publishing,pp.205–215, 2016.
[22] Hiroi, K., Murakami, D., Kurata, K., Tashiro, T., Shinoda, Y., A Proposal of Data Assimilation Approach for
Flood Level Estimation and Evaluation with Urban Flood
Disasters, Journal of Information Processing: Consumer
Devices and Systems, Vol.10, No.3, pp.55–64, 2020 (InJapanese).

87International Workshop on Informatics ( IWIN 2022 )$FWXDOCRQGLWLRQRIDLVDVWHUPUHYHQWLRQRDGLRBURDGFDVWLQJLQ$WVXJL&LW\
DQGLWV $UWLILFLDO,QWHOOLJHQFH%DVHGDLVFULPLQDWLRQRIDLIILFXOW\LQLLVWHQLQJTakumi Hashimoto*, Takahiro Miura**, Hideo Kasuga*,
Tetsuo Tanaka*, Yoshimichi Ogawa* and Mari Ueda**

Kanagawa Institute of Technology, Japan
National Institute of Advanced Industrial Science and Technology, Japans2285004@cco.kanagawa-it.ac.jp**$EVWUDFW - Atsugi City and Kanagawa Institute ofTechnology have been discussing how to ensure thatdisaster prevention radio broadcasts can be heard even intimes of disaster, in light of the current nationwide
inaudibility of disaster prevention radio broadcasts.
Therefore, we conducted acoustic analysis and listening
experiments to investigate the actual situation of inaudibilityof disaster prevention radio broadcasts in Atsugi City, andattempted to determine whether Artificial Intelligence can
automatically discriminate inaudibility. In this paper, we
conduct a listening experiment using recorded data of test
disaster prevention radio broadcasts and analyze theinaudibility of sound sources for five types of noise andweather conditions. In addition, Convolutional Neural
Network is implemented to discriminate the inaudibility oftest disaster prevention radio broadcasts by supervised
learning image classification using 800 spectrogram imagesof the recorded data. As a result, Atsugi City's disasterprevention radio broadcasts are considered unlikely tofunction during disasters caused by typhoons or torrential
rains. In addition, it was found that it is difficult todetermine the difficulty of hearing disaster prevention radio
broadcasts by artificial intelligence in this annotation.

AI, with the aim of clarifying the actual condition of
inaudibility of disaster prevention radio broadcasts in AtsugiCity.

 +($5,1* 6859(< )25 7+( 2*,12
$5($ 6XPPDU\Based on discussions with the Crisis Management
Division of Atsugi City and the Regional Collaboration
Center, we decided on a series of procedures for evaluating
the hearing in the Ogino district of Atsugi City, referring to
the first edition of "ASJ Technical Standards for Ensuring
Performance of Outdoor Loudspeaker Systems in
Emergency Situations such as Disasters"[2]. The specificflow was as follows: 1. decision to broadcast a test broadcast
for the Ogino area (Atsugi City), 2. decision of the survey
date, 3. preparation and distribution of flyers describing thesurvey to residents in the Ogino area, 4. evaluation of
audibility by residents in the Ogino area (Web), 5. analysis
and sharing of the results (Kanagawa Institute ofTechnology). The listening survey was conducted targeting
7,500 people in the Ogino area. On the day of the survey,
residents were asked to listen to the test broadcasts outdoors
and evaluate the difficulty of hearing the test broadcasts inthe form of a questionnaire using a dedicated Web site. The
dates and times of the test broadcasts were November 18(Thursday) and November 21 (Sunday), 2021, at 11:00 a.m.

.H\ZRUGV: disaster prevention radio broadcasting, Artificial
Intelligence, Convolutional Neural Network, noise, weather

 ,1752'8&7,21
In recent years, disaster prevention information has
become indispensable in Japan, where natural disasters haveoccurred in many parts of the country, and Atsugi City and
Kanagawa Institute of Technology's Research Center for
Regional Cooperation and Disaster Care have been
discussing how information should be provided in times ofdisaster. In Atsugi City, information on evacuation actionsin the event of natural disasters is provided by means ofdisaster prevention radio broadcasts, e-mail newsletters, andradio broadcasts[1]. However, in view of the current
situation in which disaster prevention radio broadcasts are
not heard nationwide, it was decided to discuss thepossibility of ensuring that disaster prevention radio
broadcasts can be heard even when people are outdoors
during a disaster.
In this study, we report on acoustic measurement andlistening experiments of test disaster prevention radio
broadcasts, and automatic identification of inaudibility using

Table 1: Heraring survey

89

7RWDOIRUGD\V

Number ofcases

Proportion

All audible

195

42.3%

A little inaudible

125

27.1%

Almost inaudible

95

20.6%

Not at all inaudible

46

10.0%

7RWDO

461

100.0%

International Workshop on Informatics ( IWIN 2022 )Table 2: Heraring survey (Including location data)

7RWDOIRUGD\V 7RELR .DPLRJLQR 1DNDRJLQR 6KLPRRJLQR 0LKDUXQR 0DWVXNDJHGDL 2WKHU 7RWDO
All audible

41

36

6

13

25

3

2

126

A little inaudible

31

12

4

3

2

14

1

67

Almost inaudible

10

7

3

42

8

0

5

75

Not at all inaudible

8

1

0

18

0

0

4

31

Total

90

56

13

76

35

17

12

299

 5HVXOWV

lightly. The data evaluated as "B" were difficult to hearbecause it was raining so hard that an umbrella was

Tables 1 and 2 shows the total results of the two-day
hearing evaluation in Ogino area. The evaluation was madeon the following four levels: 1. All audible, 2. A little
inaudible, 3. Almost inaudible, and 4. Not at all inaudible.
The results of this survey showed that the largest number of
respondents were able to understand the content of thedisaster prevention administrative broadcasts; however, theresults of this survey showed that there were some points
where many respondents were unable to understand oralmost unable to understand the content of the broadcasts.

necessary, and the sound of splashing water when a car
drove through a puddle masked the sound of the soundsource.

+HDULQJHYDOXDWLRQLQEDGZHDWKHU
Subjective evaluation of listening difficulty was
conducted on 11 students (11 males, mean age 21.5 years)
using 20 sound source data of test disaster prevention radio
broadcasts, including those recorded in the previous study[3]. The headphones used were Beoplay H9i (Bang &Olufsen). A four-point scale was used to evaluate listening
difficulty: 1. not difficult to listen, 2. somewhat difficult tolisten, 3. quite difficult to listen, and 4. very difficult tolisten [4]. Then, ordinal logistic regression analysis wasused to examine the effects of five factors (weather (sunnyor rainy), presence of passing cars or bicycles, presence ofinsects, presence of birdsong, and presence of wind noise)on the difficulty of listening. Two types of analyses wereconducted: one in which only the main effect was analyzed,
and the other in which the main effect and up to first-orderinteractions were analyzed.

 +($5,1*2)',6$67(535(9(17,21
5$',2%52$'&$676)257(67,1*
 $FRXVWLF 0HDVXUHPHQW DQG 6XEMHFWLYH
(YDOXDWLRQIn accordance with the test method[2] described on
page 18 on the first edition of the "ASJ TechnicalStandard for Ensuring the Performance of Outdoor
Loudspeaker Systems in Emergency Situations," fixedpoint observations of the sound source emitted from thetest disaster prevention radio broadcast were conducted at
one of the Ogino slave stations. Observations were madefor 10 days each at a distance of about 100 to 250 m
concentrically from the sound source (Fig. 1). At the timeof writing this paper, about 200 days of data wererecorded by a PCM recorder (Roland R-07, sampling
frequency: 96 kHz, quantization bit: 16 bit). Although theliterature [2] stipulates that "situations in which weather
has a significant influence on listening tests, such as rain,snow, and strong winds, should be avoided as much as
possible", the same method was used regardless of
weather conditions in order to compare listening
evaluation results in sunny and bad weather conditions.
Fig. 2 shows a spectrogram of the recorded soundsources in rainy weather. The subjective audibility of theobservers was evaluated at two levels, A > B. The data
evaluated as "A" showed wind noise, but the soundsource was not so difficult to hear because it was raining

Fig.1 Measurement points and conditions

90

International Workshop on Informatics ( IWIN 2022 )rain noise, and thus the disaster prevention radio broadcastsof Atsugi City are unlikely to function as a means ofcommunicating information.

 $8720$7,& ,'(17,),&$7,21
,1$8',%,/,7<%<&11

We implemented Convolutional Neural Network, which isused in the field of image recognition such as image
classification and object detection, and created a model thatclassifies sound sources into two classes, Class A. "easy-tohear" and Class B. "hard-to-hear", using 800 spectrogram
images of test disaster prevention radio broadcast data. Then,we examined whether the accuracy of the model can be used
to evaluate the inaudibility of sound sources in the same
way as humans do.

Fig.2 Spectrogram of administrative radio system:HDWKHU:HDWKHU&DUVELF\FOHV,QVHFWV&DUVELF\FOHV



%LUGV
:LQG

/HDUQLQJ3URFHGXUH

,QVHFWV:HDWKHU%LUGV&DUVELF\FOHV%LUGV

The pre-recorded sound sources were rated in four levels(1. not difficult to listen, 2. somewhat difficult to listen, 3.
very difficult to listen, and 4. very difficult to listen) as
supervised learning, and the correct answers were labeled as
1 is easy to listen and 2 to 4 are difficult to listen, thus
discriminating them as a two-class classification problem.
The data was classified as a two-class classification problem.
In addition, the labeling process is ambiguous in that thenoise is not always present during the one-minute soundrecording, and it is not clear whether to judge the entire
recording as difficult to listen to or only the part with thenoise as difficult to listen to. Therefore, we divided theentire one-minute recording data into five-second segments
to allow for more accurate labeling and more data. Note that
all image data were resized to 143*143 squares.

%LUGV,QVHFWV%LUGV:HDWKHU:LQG

:LQG



%LUGV:LQG








/RJ2GGVUDWLR



2)











/RJ2GGVUDWLR

Fig.3 Log odds ratios to listening difficulty on five factors
derived by considering only main effects (left) and main
effects with two-way interactions (right)
Fig. 3 shows the effect of each factor on the difficulty oflistening (+: p<0.10, *: p<0.05, **: p<0.01, ***: p<0.001).
When analyzing the effect of only the main effect as shownin the left figure of Fig. 3, no significant main effect was
confirmed for differences in weather (p>0.10). On the other
hand, when the effects up to the first-order interaction werechecked as shown in the right figure of Fig. 3, a significant
main effect was confirmed for the difference in weather(p<0.01), and significant interactions were also observed forthe difference in weather and the presence of birdsong andthe difference in weather and the presence of wind noise(p<0.01).
The reason for this result is that, although sound sources
should be difficult to hear in rainy weather, when birdsong
and wind sound are heard, the effect of rain sound isrelatively small due to low precipitation. Therefore, it ispossible that the difficulty of hearing the sound was reduced
even when it was raining. Conversely, when precipitation isextremely high, birdsong and wind sounds are considered tobe less audible.
Based on the above, In the case of earthquake disasters,
the audibility will depend on the amount of traffic at thelistening point when the weather is clear, and in the case ofrain, it will depend on the amount of precipitation, but there
is a high possibility that the sound will be inaudible. In
addition, in the event of a disaster caused by heavy rain or atyphoon, the sound source will be almost inaudible due to5HVXOWV
Fig.4 shows that the loss values are large and thepercentage of correct responses is significantly low, around50%. Furthermore, the graph is stagnant from the second
epoch, suggesting that unlearning is occurring. From Fig. 5,
the classification model created was used to classify 40 test
data (20 spectrogram images of Class A and 20 spectrogram
images of Class B) that are not training data, resulting
accuracy of 0.53, precision of 0.53, recall of 0.53, FMeasure of 0.53, and this learning The results showed thatthe model was not able to discriminate the inaudibility ofdisaster prevention radio broadcasts in the same way as
humans. This result indicates that this learning model could
not discriminate inaudibility of disaster prevention radio
broadcasts in the same way as humans.

91

International Workshop on Informatics ( IWIN 2022 )if it were cracked loudly because it is too close. The failureof automatic identification of inaudibility by the
Convolutional Neural Network may well be due to theinadequacy of the model, but it is also possible that thecorrect label given by subjective evaluation may not always
be correct, since the inaudibility of the same recorded data
varies from person to person. However, it is also possiblethat the model is inadequate. In other words, the annotation
was not accurate. Therefore, it is necessary to define
whether or not it is difficult to listen to each data by multiple
people and to assign the correct label to each data.

 &21&/87,21
From the results, it is clear that Atsugi City's disasterprevention radio broadcasts do not function well as a meansof information transmission during disasters caused byrainfall such as typhoons and torrential rains. In addition, it
is considered that it is difficult to evaluate the difficulty oflistening to disaster prevention radio broadcasts using AI inthis annotation. In this listening survey, we received several
comments from residents that the way they hear thebroadcasts changes depending on the wind direction. In the
future, we will investigate how wind speed and direction
affect the way of hearing disaster prevention administrative
broadcasts. We plan to discuss with Atsugi City the future ofdisaster prevention administrative broadcasts and theintroduction of new information transmission methods using
ICT, etc., based on the results of this survey and the fact that
we were able to roughly identify points in the Ogino area
where it is difficult to hear.

Fig.4 Learning curve with 800 data

5()(5(1&(6[1] Disaster Information Transfer / Atsugi City,
https://www.city.atsugi.kanagawa.jp/soshiki/kikikanrik
a/7/8/2310.html [Reference date April 25th, 2022][2] ASJ Technical Standard for Ensuring Performance of
Outdoor Loudspeaker Systems in Disasters and Other
Emergencies,1stEdition,
https://asj-msscom.acoustics.jp/?page_id=277[3] Terashima et al., Acoustical Science and Technology,(Spring), 2020.
[4] Sato et al., Acoustic Journal, 63(5):275-280, 2007.

Fig.5 Confusion Matrix

 ',6&866,21
From the listening survey conducted in the Ogino area,
the points where the respondents answered that they could
hardly hear or could not hear at all were often located at theedge of residential areas, where the average noise level was
high, the distance to the loudspeaker was long, and there
were tall buildings or obstacles in the vicinity. These areconsidered to have a strong influence on the audibility of thedisaster prevention administrative broadcasts. However,there are also some stations that are located too close to thedisaster prevention radio loudspeaker, but are evaluated tobe difficult to listen to. This is thought to be due to the factthat the broadcast itself is audible, but the sound is heard as92

International Workshop on Informatics ( IWIN 2022 )9HULILFDWLRQRIWKHBHVWTiPHWRSHHCKHUU\BORVVRPVEVWLPDWLRQMHWKRG
Using Time SHULHVPUHGLFWLRQMHWKRG
Junya Sato*, Yusuke Takamori*, Masahiro Fujimoto*, Tomonari Horikawa**, Masaki Endo***,Shigeyoshi Ohno***, Masaharu Hirota****, and Hiroshi Ishikawa******

Electronic Information Course, Polytechnic University, Japan{b19308, b19309, b19315}@uitec.ac.jp**

Nippon COMSYS Corporation
horikawa-tomonari@comsys.co.jp
***
****

Division of Core, Polytechnic University, Japan{endou, ohno}@uitec.ac.jp

Department of Information Science and Engineering, Okayama University of Science, Japanhirota@ous.ac.jp*****

Graduate School of Systems Design, Tokyo Metropolitan University, Japan
ishikawa-hiroshi@tmu.ac.jp$EVWUDFW – With the recent spread of social networking

events and hobbies. Posting text, photos, and otherinformation related to Twitter is called "tweeting." Among
those tweets are some with location information added to asetting, called "geotagged tweets." Geotagged tweets canreflect real-world situations because they can share what
happened, where and when. Therefore, they are expected toserve as a social sensor for tourists to estimate and obtain
local tourism information in real time. The use of SNS will
enable low-cost, seasonal plant seasonality estimation.
Currently, biological regular observations are conducted by
the Japan Meteorological Agency based on the Biological
Seasonal Observation Guidelines. However, seasonal
biological observations that are fixed-point observations
using sample trees incur costs because of manual
observations. The estimation of seasonality using SNS is
expected to be useful as a new observation method when
biological regulation observation becomes difficult forbudgetary reasons. Currently, information related togeotagged tweets on Twitter is used to estimate the best timeto see cherry blossoms. Endo et al. [2] proposed a method thatuses simple moving averages to estimate the best time to see
cherry blossoms. This method, which collects geotagged
tweets that include specific key words, estimates the best timeto see species by obtaining a simple moving average of thenumber of tweets per date. In other words, it is possible toestimate the best time to view cherry blossoms in prefecturesand localities where a certain number of geotagged tweets are
displayed. Using this approach, they confirm the real-world
seasonality of tweets related to the names of organisms ineach region and demonstrate the potential for presenting
tourism information in real time. Takahashi et al. [3]proposed a method using weighted moving averages toimprove the estimation accuracy of the method presented byEndo et al. Compared to the method using a simple moving
average, this method can improve the results of estimating

services (SNSs), information from many tourists has come tobe shared and collected in real time. While proposing amethod for estimating the best time to view cherry blossoms
using Twitter, we are advancing research that is expected tohelp people obtain tourist information. For estimation of thebest time to view cherry blossoms, a low-cost moving
average method using geotagged tweets was proposed froman earlier study. By combining geotagged tweets and theseason-estimation conditions, a person can estimate the best
time to see cherry blossoms in any prefecture in Japan. We
also proposed a time-series prediction method using machinelearning to estimate, for a certain future period, the best time
to view cherry blossoms. This method uses past tweet
information as training data to predict the number of tweets
about cherry blossoms within a certain period. Moreover, themethod estimates the best time to view cherry blossoms basedon the results. Described herein are the results of estimating
the best times to view cherry blossoms for 2022, withconfirmation of the method’s usefulness.1.H\ZRUGV: Cherry blossoms; machine learning; SNS; text
mining; tourism

 ,1752'8&7,21
In today's information-oriented world, many people
obtain information from the web. In recent years, with the
spread of social networking services (SNSs), a great deal of
widely diverse information is posted and viewed by many
SNS users. It is not uncommon for tourists use SNSs to gather
tourist information when choosing places they would like tovisit. Twitter [1] is an example of a social networking service
through which tourism information is shared. Twitter isgenerally used to post and view information about personal
This work was supported by JSPS KAKENHI Grant
Number JP19K20418, JP20K12081, JP22K13776 and the
Okawa Foundation Research Grant.

95

International Workshop on Informatics ( IWIN 2022 )cherry blossom viewing season for each prefecture while
maintaining low costs. Horikawa et al. [4][5] applied theestimation method of Takahashi et al. and proposed a method
for estimating the best time to see cherry blossoms in the
future using time-series forecasts based on machine learning.
Time-series forecasting is the process of predicting data
trends over a future period based on past time-series data.
Because cherry blossoms are seasonal plants, time-series
forecasting is useful to predict the number of tweets related
to cherry blossoms for a certain future period. The judgmentof when the cherry blossoms are at their best is made byanalyzing the relative trends in the number of tweets.
Therefore, we have developed a highly accurate estimation of
the best time to view cherry blossoms using the time-series
predictions of tweets. As described herein, we improve theestimation method presented by Horikawa et al. and newly
add tweets for 2022 to estimate the best time to view cherry
blossoms. In an earlier study, the accuracy of the estimatedcherry blossom viewing time was less than 50% for all thecities tested, thereby rendering accurate estimation
impossible. We attributed this finding to the small amount ofdata used for machine learning. An earlier study used data of
2015–2021. The present study used data through 2022. The
accuracy was improved by removing unnecessary data used
for machine learning and by improving the data processing
methods.
The structure of this paper is the following: Section 2
presents a description of the research related to this paper;
Section 3 presents the proposed method; Section 4 explains
the experimentally obtained results and their evaluation; andSection 5 summarizes the conclusions of this study.

new model, the number of sites was decreased to 19.
Takamori et al. [9] conducted a forecast of the numberof new positive cases based on data published as open data by
the Tokyo Metropolitan Government. Using time-series
prediction by machine learning and using time-series data
including the number of newly positive cases of novel
coronavirus and relevant variables as a dataset, they indicate
the possibility of using this method to predict the number of
newly positive cases in the Tokyo Metropolitan Government.
As described above, some studies have examined
extraction of tourist information using SNS, studies of theestimation of cherry blossom viewing time, and studies using
time series prediction by machine learning. For the present
study, we demonstrate the usefulness of a method for
estimating the best time to see cherry blossoms using machinelearning time-series prediction for a certain period in the
future.

 352326('0(7+2'
This chapter presents descriptions of the data and the method
used to estimate the timing of cherry blossom viewing.

 'DWDSUHSDUDWLRQFor this study, we used the Twitter Streaming API tocollect geotagged tweets from Japan that include location
information [10]. Then, using the simple reverse geocoding
service [11] of the Ministry of Agriculture, Forestry and
Fisheries based on latitude and longitude information, thecollected geotagged tweets were assigned to the prefecturefrom which they originated. The timing of cherry blossom
viewing was estimated by predicting and analyzing thetransition of tweets including specific key words from tweets
in each prefecture. The target regions for the experiment wereTokyo, Kyoto, and Shizuoka, all of which have large
populations and which are famous for cherry blossoms. The
key word was "cherry blossom" in kanji, hiragana, or
katakana; the analysis period was February 1, 2015 –February 28, 2022.

 5(/$7('5(6($5&+
Obara et al. [6] use regional associative words andpattern matching from Twitter text to extract tourist
information and to elucidate where users live. They haveconstructed a system that displays the distribution of tweeted
locations and estimated user residences by graphing the ratioof acquired regional associative words and the number ofusers who tweeted tourist information. The method of
extracting tourist information using pattern matchingachieved a conformance rate of 80.2%.
Silaa et al. [7] propose a new on-the-spot tourist opinion
extraction method from the internet for tourism information
analysis systems by building a classifier that automaticallydistinguishes and adds geotagged tweets. They present a
method to support lesser-known tourist attractions byautomatically collecting and analyzing geotagged tweets andby creating on-the-spot reviews.
Aono et al. [8] proposed a simple method to estimate theflowering date of Someiyoshino cherry trees. The method
incorporates data of the temperature during the spontaneous
dormancy period. The starting date is obtained from thelatitude of the location, distance from the coast, and averagetemperature during January–March. The model accuracy isimproved by estimating the flowering date at 23.8 days,
which is the optimal solution common to the entire country
for the integrated value. When using the earlier model, 35 of
216 sites had an RMSE of more than 3 days. When using the 7LPHVHULHVIRUHFDVWBased on tweet information collected in the past, a timeseries prediction of the number of tweets during the estimatedcherry blossom viewing period was performed. Because
cherry blossoms bloom mainly during March–April, theestimated period of cherry blossom viewing is the two monthsof March 1 – April 30, 2022. By performing time-series
forecasting of the number of tweets during this period, we canestimate the best seasons for viewing in two-monthincrements. This method uses Amazon Forecast [12], aservice that performs time-series forecasting using statisticaland machine learning algorithms from Amazon Web Services[13] (AWS). The training data for this method constitute alarge dataset of more than 300 days. In addition, the training
data are regarded as seasonal because tweets related to cherry
blossoms blooming around spring of each year are collected.
Therefore, the Amazon Forecast algorithm uses Prophet,which is suitable for seasonal training data, and DeepAR+,
which is suitable for large datasets of 300 days or more. The

96

International Workshop on Informatics ( IWIN 2022 )predicted value is the value that satisfies the 50% demandobtained from the weighted quantile loss. Weighted quantile
loss is a type of metric used during forecasting with Amazon
Forecast. The predicted number of tweets is rounded down to
the nearest whole number. If the number of tweets is negative,
it is treated as 0. The number of tweets during a certain period
predicted by machine learning is used to estimate the best
time to view the cherry blossoms. The training data used for
estimating the best time to view cherry blossoms are tweets
from February 1, 2015 through February 28, 2022. No tweet
information was available for 2020 as a result of system
limitations. Therefore, for prediction, we used tweet
information from February 1, 2015 through December 31,
2019, and from January 1, 2021 through February 28, 2022.
The time-series forecasting period for the number of tweets isthe two-month period from March 1 through April 30, 2022.
Prophet is a time series forecasting algorithm based on an
additive model for which the nonlinear trend fits year, week,
and daily seasonality. It is most effective for time series withstrong seasonal effects and several seasons of historical data
[14]. When using Prophet, data for all months of the year areused for forecasting.  Deep AR+ uses recurrent neuralnetworks (RNNs) for a supervised learning algorithm forpredicting 1D time series. It works best with large datasets[15]. DeepAR+ extracts and uses the period from February 1
to April 30, which is a characteristic period for the number oftweets about cherry blossoms. In addition, because thetraining data for time-series prediction in DeepAR+ use
continuous values, the dates of the extracted data areconverted so that they become continuous values. For an
earlier study, three months of data for each year were
processed to fit three months of actual dates when converted
to continuous values. Specifically, data for February throughApril of each year were processed and used as data for
November–January, August–October, and May–July; data
for February–April were 89 days or 90 days in leap years,
which are fewer days than in other seasons. Therefore, when
applying the data to another three-month period, the data for
May of the same year were used to compensate for a few daysof May data to process data for the forecast. In time-series
forecasting, cyclical fluctuations in past time-series data areused as a factor in forecasting. Therefore, the data processedin the earlier study might include unnecessary data other thanthose of February–April, which might therefore impair thedata periodicity. Therefore, for this study, to avoid spoilingthe periodicity of the time-series data, the data of February–
April of each year were converted to continuous values andwere used as training data, without supplementing the May
data. To give a specific example of processing, we extracted
data of February–April from 2015–2021 and February datafrom 2022, and transformed February 1 through April 30 in
2021 from November 4, 2021 to January 31, 2022, February
1 through April 30 in 2019 from August 7, 2021 to November
3, 2018, and February 1 to April 30, 2018 as May 10 toAugust 6, 2021, respectively. In an earlier study [5], higher
accuracy predictions were obtained when using Prophet.

blossoms by tweet transition uses the conventional method ofweighted moving averages. A weighted moving average is amoving average with each value weighted. In the method
used for this study, the median value is set to 1. The minimum
and maximum values are set respectively as ±0.5 from themedian value. To estimate the best viewing period for cherry
blossoms, we analyze the frequency of geotagged tweets
including the key word. The following three values will beanalyzed by date and will be used as the basis for estimating
the optimal viewing period.
1.
2.
3.

1-year simple moving average of the number of tweets
7-day weighted moving average of tweets
5-day weighted moving average of tweets

To assess the increase in the frequency of geotagged
tweets containing the key words, a 1-year simple moving
average of the number of tweets was obtained. Because
tweets including key words have words that are unrelated toactual cherry blossoms, one can ascertain the increase in
tweets related to actual cherry blossoms by comparing thenumber of tweets with the one-year simple moving averageof the number of tweets. To analyze the increasing trend of
the number of tweets, a 7-day weighted moving average anda 5-day weighted moving average are obtained. Because thenumber of tweets tends to increase on Saturdays and Sundays,a 7-day weighted moving average is obtained. The formula to
be used is shown below. xy denotes the number of tweets x ofy days prior.
‫ܪ‬௔௩௚଻ ൌ଴Ǥହ௫ళ ା଴Ǥ଺଻௫ల ା଴Ǥ଼ଷ௫ఱ ା௫ర ାଵǤଵ଻௫య ାଵǤଷଷ௫మ ାଵǤହ௫భ଻

(1)One can also find the weighted moving average of 5days, which is the average number of days between the cherry
blossom bloom and full bloom. The formula to be used ispresented below. xy is the number of tweets x of y days prior.
‫ܪ‬௔௩௚ହ ൌ଴Ǥହ௫ఱ ା଴Ǥ଻ହ௫ర ା௫య ାଵǤଶହ௫మ ାଵǤହ௫భହ

(2)Using these estimation criteria, we can ascertain thatcherry blossoms are at their best when the following two
conditions are met.
1.
2.

Number of tweets is greater than the 1-year simple
moving average
The 5-day weighted moving average of the number oftweets is greater than the 7-day weighted moving
average for three consecutive days

These conditions are used to estimate the best time
period to see each attraction.

 (;3(5,0(175(68/76
In this chapter, we present results of the estimation ofcherry blossom viewing time using this method. The peakcherry blossom peak prediction can be achieved by
combining tweet information with the best time-of-view
estimation conditions. The accuracy of the peak prediction is %HVWWLPHHVWLPDWLRQPHWKRGThe condition for estimating the best time to see cherry97International Workshop on Informatics ( IWIN 2022 )evaluated by comparing results of peak estimation with thecorrect prediction period, and by the values of the
reproduction rate and the goodness-of-fit rate. The correct
prediction period is defined as the period from the cherry
blossom blooming date to the full blooming date, as observedby the Japan Meteorological Agency [16]. The cherry
blossom date represents the first day when five to six or morecherry blossoms are in bloom on a specimen tree. A cherry
blossom full bloom date is the first day on which 80% or moreof the specimen trees are open. First, Figures 1–3 respectively
present results for Tokyo, Kyoto, and Shizuoka prefecturesfor 2022: estimates of the cherry blossom viewing season
based on the number of tweets obtained from the time-series

forecast.
Figures 4–6 respectively present estimated results of thebest time to see cherry blossoms in Tokyo, Kyoto, andShizuoka prefectures in 2022, based on the number of tweets
obtained using DeepAR+ as the algorithm for time-series
prediction.
Figures 7–9 respectively present results obtained fromestimating the cherry blossom viewing season in Tokyo,
Kyoto, and Shizuoka prefectures in 2022. Table 1 presents
the actual number of tweets and the predicted number oftweets. Table 2 shows the reproducibility and goodness-of-fitof the estimation of the cherry blossom season using theactual number of tweets and the predicted number of tweets.

Figure 1: Tokyo after forecasting in 2022 (Prophet).

Figure 4: Tokyo after forecasting in 2022 (DeepAR+).

Figure 2: Kyoto after forecasting in 2022 (Prophet).

Figure 5: Kyoto after forecasting in 2022 (DeepAR+).

Figure 3: Shizuoka after forecasting in 2022 (Prophet).

Figure 6: Shizuoka after forecasting in 2022 (DeepAR+).

98

International Workshop on Informatics ( IWIN 2022 )Figure 9: Shizuoka in 2022.

Figure 7: Tokyo in 2022.

Table 1: Total numbers of actual and expected tweets


$UHD

$FWXDO3URSKHW

1XPEHURIWZHHWV

7RN\R



'HHS$5

$FWXDO

3URSKHW.\RWR



'HHS$5$FWXDO

3URSKHW

6KL]XRND'HHS$5




Figure 8: Kyoto in 2022.
Table 2: Reproducibility and goodness-of-fit rates for actual tweets and predicted tweets
for the estimated viewing method



$UHD

5HFDOO

3UHFLVLRQ







'HHS$5



$FWXDO







'HHS$5



$FWXDO











$FWXDO3URSKHW3URSKHW3URSKHW

7RN\R.\RWR

6KL]XRND'HHS$5
A comparison of Figures 1–9 is illustrative. Comparisonof Figures 1–3 and 4–6 with Figures 7–9 confirms thepredictability of the trend in the number of tweets for bothalgorithms. Particularly, only in the trend of the number oftweets in Shizuoka prefecture can we confirm the existenceof a peak in early March, both in terms of actual and predicted
number of tweets. This finding is attributable to the fact that
Shizuoka prefecture is famous for its early blooming cherrytrees such as Kawazu-zakura (in Japanese Ἑ ὠ ᱜ ) and
Kakegawa-zakura (in Japanese ᥃ ᕝᱜ ). As an example,“Kawazu Cherry Blossoms are in full bloom! Kawazu-zakura,
the original cherry tree, known throughout Japan as an early
blooming cherry tree, is now in full bloom.” the tweet reads.
In fact, 108 of the 211 tweets posted during March 1–10
included the words "Kawazu-zakura or "Kawazu-gawazakura-namiki." Therefore, it is conceivable that the trend of
tweets related to Kawazu cherry blossoms was predicted.
Table 1 presents the results. The total number of tweets
predicted using Prophet tended to be higher than the actual
number of tweets in Tokyo and Kyoto prefectures, but lower

99

International Workshop on Informatics ( IWIN 2022 )than the actual number of tweets in Shizuoka prefecture. The
total tweets predicted by DeepAR+ tended to be fewer than
the actual tweets in Tokyo, Kyoto, and Shizuoka prefectures.
Table 2 also presents results. The time-series predictionof the number of tweets using Prophet showed higher
reproducibility and better goodness-of-fit than the method
using the actual number of tweets. The reproducibility ofTokyo improved from 62.5% to 100%; the fit rate improved
from 27.8% to 36.4%. Kyoto prefecture's reproducibility
improved from 57.1% to 100%; its compliance rate increased
from 22.2% to 29.2%. Shizuoka prefecture's reproducibility
improved from 80% to 90%; its compliance rate increased
from 47.1% to 52.9%.
For Tokyo, Kyoto, and Shizuoka prefectures, thereproducibility and fit rates were higher than those obtainedusing the timing estimation method with the actual number oftweets. Shizuoka prefecture in particular achieved aremarkably high reproducibility rate of 52.9%.
The time-series prediction of the number of tweets using
DeepAR+ demonstrated the possibility of predicting thenumber of tweets in Tokyo and Kyoto prefectures with higher
reproducibility and goodness-of-fit rates than the method
using the actual number of tweets, although it was notpossible to predict the number of tweets in Shizuoka
prefecture with high reproducibility and goodness-of-fit rates.
The reproduction and fit rates of time-series forecasting
method using DeepAR+ and the method based on actual
tweet counts were compared. The reproduction rate for Tokyo
improved from 62.5% to 87.5%; the fit rate improved from
27.8% to 58.3%. Kyoto prefecture's reproducibility improved
from 57.1% to 100%; its conformance rate increased from
22.2% to 35.0%. Shizuoka prefecture's reproducibility
decreased from 80% to 20%; its compliance rate declined
from 47.1% to 16.7%.

the time-series forecasting method using Prophet in Tokyoand Kyoto prefectures.
For this study, we demonstrated the possibility of using
a method for estimating cherry blossom season with timeseries prediction of the number of tweets to predict cherry
blossom season trends for a certain future period.
Additionally, we demonstrated the possibility of using both
Prophet and DeepAR+ as algorithms for time-series
forecasting to achieve higher accuracy. Particularly, the timeseries forecasting method using DeepAR+ enabled moreaccurate forecasts than Prophet by screening data used fortime-series forecasting. Additionally, results show that theuse of tweet data enables low-cost estimation of cherry
blossom seasonality for seasonal plants. In future studies, we
intend to increase the number of target areas and to achieve
more accurate forecasts using new cherry-blossom-related
variables such as temperature.

5()(5(1&(6[1] Twitter, https://twitter.com/[2] M. Endo, K. Mitomi, K. Saeki, Y. Ehara, M. Hirota, S.
Ohno, and H. Ishikawa, Study of information provided
by the best time to see estimation method ofphenological observations using tweets, Proceedings ofthe 12th Annual Conference on Japan Society for
Tourism Informatics, Shizuoka, Japan, pp.47-60 (2016).
[3] M. Takahashi, M. Endo, S. Ohno, M. Hirota, and H.
Ishikawa, Automatic detection of tourist spots and besttime estimation using social network services,
International Workshop on Informatics 2020, pp.65-72(2020).
[4] T. Horikawa, M. Takahashi, M. Endo, S. Ohno, M.
Hirota, and H. Ishikawa, Estimating the best time to see
cherry blossoms using SNS and time-series forecastingof tweet numbers using machine learning, International
Workshop on Informatics 2021, pp.37-44 (2021).
[5] T. Horikawa, M. Takahashi, M. Endo, S. Ohno, M.
Hirota, and H. Ishikawa, Estimating the Best Time toView Cherry Blossoms Using Time-Series Forecasting
Method, Multidisciplinary Digital Publishing Institute,
pp.418-431 (2022).
[6] M. Ohara, K. Morita, M. Fuketa, and J. AOE, Extractionof Tourist Information from Contents of Tweets andBuilding an Analysis System, The 29th Annual
Conference of the Japanese Society for Artificial
Intelligence, Vol.29, pp.1-3 (2015).
[7] V. Silaa, F. Masui, and M. Ptaszynski, Vol.12, No. 5,2321, pp.1-24 (2022).
[8] Y. Aono, and N. Murakami, A simplified method toestimate cherry blossom phenology considering
temperature during endodormancy process, Climate in
Biosphere, Vol.17, pp.25-33 (2017).
[9] Y. Takamori, R. Watanabe, D. Kato, M. Endo, and H.
Ishikawa, Numerical estimation of new COVID-19
positive cases using time series analysis by machinelearning, Proceedings of the 13th International
Conference on Management of Digital EcoSystems,
pp.153-159 (2021).
[10] Twitter Developers, https://dev.twitter.com/ (referred &21&/86,21
As reported herein, we conducted experiments to assessa time-series forecasting method for estimating the best time
to view cherry blossoms at a certain time in the future. The
experiment results indicated the usefulness of this time-series
forecasting method using machine learning.
When the number of tweets in the predicted time series
is used to estimate the best time to view cherry blossoms, therepeatability and goodness-of-fit rates are expected to be as
good as or better than those obtained by estimating the best
time to view cherry blossoms using actual tweets, except for
the time series in Shizuoka prefecture using DeepAR+.
Therefore, results confirmed the possibility of predicting thebest time to view cherry blossoms for a certain period of timein the future using an appropriate algorithm. Comparison of
the results obtained respectively using Prophet and DeepAR+
demonstrated that the accuracy of time-series prediction
using DeepAR+ was higher in Tokyo and Kyoto prefectures,
although the accuracy of time-series prediction using Prophet
was higher in Shizuoka prefecture. An earlier study showedthat the time-series forecasting method using DeepAR+ failed
to improve the reproducibility and goodness-of-fit rate
significantly compared to the method using actual tweets. By
contrast, this study showed higher reproducibility than that of100

International Workshop on Informatics ( IWIN 2022 )in June 2022)[11] Agricultural Research Institute, Simple Reverse
Geocoding Service, https://aginfo.cgk.affrc.go.jp/
rgeocode/index.html.ja (referred in June 2022)[12] Amazon Forecast, https://aws.amazon.com/jp/
forecast/ (referred in June 2022)[13] AWS, https://aws.amazon.com/jp/console/ (referred inJune 2022)[14] Prophet, https://docs.aws.amazon.com/ja_jp/forecast/latest/dg/aws-forecast-recipe-prophet.html (referred inJune 2022)[15]
DeepAR+,
https://docs.aws.amazon.com/ja_jp/
forecast/latest/dg/aws-forecast-recipe-deeparplus.html(referred in June 2022)[16] Japan Meteorological Agency https://www.data.jma.
go.jp/sakura/data/index.html (referred in June 2022)

101



International Workshop on Informatics ( IWIN 2022 )$6WXG\RI'LIIHUHQW9DOXHVEHWZHHQ,QGLYLGXDOVDQG&RPSDQLHV5HJDUGLQJ3HUVRQDO,QIRUPDWLRQ
Mayu Satake, Hideki Goromaru**

Faculty of Social Systems Science, Chiba Institute of Technology, Japan
goromaru.hideki@p.chibakoudai.jp$EVWUDFW - In recent years, with the spread of mobile
devices such as smartphones and tablets, there have been
more and more opportunities to easily post text, photos,
videos, and other information on social networking services(SNS) on the internet and add comments. In addition, SNSs
has made it possible to communicate not only withacquaintances but also with people who have never met
before. However, there have been risks that outsider who are
unhappy with the content of comments or postingprocedures may misuse information posted on SNSs to
engage in problematic behavior such as stalking or
harassment. To reduce these risks as much as possible, it has
been important to avoid posting information that arevulnerable to misuse. However, we do not know what kindsof information are vulnerable to misuse. Therefore, we first
conducted investigates focusing on the value of information,
assuming that information that are easily misused are
information of high value, such as personal information. As
a result, we have found that, except for information related
to attracting customers such as gender, occupation, andgrade, individuals estimate the value of information higher
than companies and organizations.

acquisition, and entertainment, while keeping the damage
caused by misuse-prone information that should not be
posted on SNS within acceptable limits, in other words, tostrike a balance between opportunities and threats.
However, it is unknown what kind of information iseasily misused. In particular, if there is a discrepancy
between the personal value perception of information andthe value perception of the world, it may not be possible toproperly select due to the discrepancy. For example,
information that is considered to be okay for individuals to
publish may cause great damage if it is published on SNSs.
Therefore, we focused on the value of information andconsidered information that is easily misused as informationof high value. Next, we considered taking that value as amonetary amount and quantifying it. Then, in order to grasp
the difference between individual and general value
perceptions of information, we investigate the amount ofmoney for information by questionnaire in the case ofindividual value perceptions, and by court records ofcompanies and organizations in the case of general value
perceptions.
In this paper, the value of information and data isquantified for the goal of reducing the damage caused byinformation and data related to individuals within thepermissible range while maintaining the convenience ofSNSs as much as possible. Chapter 2 is the definition of
information and data related to individuals, Chapter 3 is thecurrent state of data and information on SNSs, Chapter 4 isthe problems and issues of conventional research, andChapter 5 is the proposal of information related toindividuals and the value estimation model of data. Chapter
6 describes the results. Chapter 7 describes the discussion
and Chapter 8 describes the conclusions.

.H\ZRUGV: Personal Information, Personal Data, Sensitive
Information, Privacy Information

 ,1752'8&7,21
In recent years, with the spread of smartphones and tablet
terminals, people have more opportunities to easily posttexts, photos, and videos on SNSs on the Internet. It has
become possible to communicate not only withacquaintances but also with people we have never met. And
many SNSs allow users to set the public range so that users
can receive as little damage as possible from outsiders [1].
However, even so, depending on the content of comments
and the way they are posted, outsiders may engage inproblematic behavior that users do not want, such as
stalking or harassment, and may misuse the information
posted on SNSs [2]. In addition, even if SNS’s users pay
attention to security and privacy on a daily basis, they may
become distracted due to poor health or security fatigue andpost unintentional content [3]. Therefore, it is important toavoid disclosing information on SNSs that could be misused
as much as possible.
However, the act of not disclosing information out ofconcern for misuse is not an intended use of SNSs andreduces its usefulness. Therefore, it is necessary to maintainthe enjoyment of SNSs, such as communication withoutsiders,self-disclosure,socializing,
information

 '(),1,7,2162)3(5621$/'$7$
In this section, we organize the terms because there areseveral terms with the same or similar names. For example,"Personal Data" under Japanese law [4] and "Personal Data"in the European GDPR are different concepts despite having
the same name. “Sensitive information” and “Special carerequired personal information” are similar concepts but
different terms. Data and information are strictly different
concepts. However, data and information are sometimes
used in the same class. To avoid confusion in this paper,"data and information on individuals" is unified as "Inf" anddivided into Inf.1 through Inf.6, and the position of eachterm is defined in Figure 1 and Table 1. Figure 1 is based on
existing literature on Personal Data (under the EU’s GDPR),
personal information and privacy [5][6][7][8][9][10].

103International Workshop on Informatics ( IWIN 2022 ) $WWULEXWH,QIRUPDWLRQ ,QI DQG
,GHQWLILFDWLRQ,QIRUPDWLRQ ,QIa,QI ᵮᶃᶐᶑᶍᶌᵿᶊᴾᵢᵿᶒᵿᴾᵆᶓᶌᶂᶃᶐᴾᶒᶆᶃᴾᵣᵳḚᶑᴾᵥᵢᵮᵰᵇ‚※⁠⁘†‣‛

ᵮᶃᶐᶑᶍᶌᵿᶊᴾᵧᶌᶄᶍᶐᶋᵿᶒᶇᶍᶌ

Ishii [12] classifies information on SNSs into two
categories: identification information and attributeinformation as follows:(1) Attribute information (Inf.1)
x
Information that cannot be identified by one piece of
information (example: hobbies, occupation / grade,family structure, gender, current location, movement
history, age, birthplace, address to prefecture,presence / absence of lover's spouse, blood type,etc.).
(2) Identification information (Inf.2~Inf.5)xInformation that combines basic information(example: name, affiliated company, school, facephoto, address, etc.)
In the case of SNSs, the identification information is afactor to expand the friendship to "known friends", but it is
not a factor to expand to "friends only by SNSs". On thecontrary, attribute information is a factor that expands
friendships in both cases. In addition, Sato et al. [13]consider that Personal Information on the Internet can beclassified into four categories, identification information issimilar to contact information, and attribute information is ahobby. Therefore, when posting to an SNS that can beviewed by an unspecified number of people, it is often
posted for the purpose of communicating with anunspecified number of people. In that case, attributeinformation rather than identification information is easier todisclose self-information.

‚※⁠⁘†․‛
ᵮᶃᶐᶑᶍᶌᵿᶊᴾᵢᵿᶒᵿᴾᵆᶓᶌᶂᶃᶐᴾᵨᵿᶎᵿᶌᶃᶑᶃᴾᶊᵿᶕᵇ‚※⁠⁘†‥‛
ᵱᶃᶌᶑᶇᶒᶇᶔᶃᴾᵧᶌᶄᶍᶐᶋᵿᶒᶇᶍᶌ
Ҹ^ƉĞĐŝĂůĐĂƌĞͲƌĞƋƵŝƌĞĚƉĞƌƐŽŶĂůŝŶĨŽƌŵĂƚŝŽŶ
‚※⁠⁘†…‛‚※⁠⁘†‧‛^ƉĞĐŝĂůĐĂƌĞͲƌĞƋƵŝƌĞĚƉĞƌƐŽŶĂůĚĂƚĂ‚※⁠⁘† ‛

ᵮᶐᶇᶔᵿᶁᶗ
Figure 1 Personal Data and Privacy
Table 1 Features of Inf.
/ŶĨ/ŶĚŝǀŝĚƵĂůůǇŝĚĞŶƚŝĨŝĂďůĞ

^ĞĂƌĐŚĂďůĞ

ŶŽŶͲƉƵďůŝĐ

ϭ

Ϯ

䃺

ϯ

䃺

䃺

ϰ

䃺

䃺

ϱ

䃺

ϲ

䃺
䃺

 6HQVLWLYH,QIRUPDWLRQ ,QIDQG,QI 䃺When companies utilize information or data provided orcollected from individuals, Sensitive Information (e.g.,thought, physical characteristics: Inf.4 and Inf.5) cannot be
used without their consent [4]. However, in the case ofindividuals, the boundary definition of information to beprotected is ambiguous and depends on the individual's wayof thinking. Machida et al. [14] investigated information
leakage after SNSs postings from 137,877,745 postingsfrom June 12, 2012 to June 13, 2013, and found that 0.008%
of Inf.4 and Inf.5 existed. Although this detection value isvery low, it is necessary to detect even a very small numberof Inf.4 and Inf.5 to prevent leaks.

In this paper, we define it as follows:x“Personal Data” is not Personal Data under Japaneselaw, but Personal Data under the EU’s GDPR.
"Personal Information" includes Inf.1, Inf.2, Inf.3,
Inf.4, Inf.5 and Inf.6.
x“Personal Information” includes Inf.2, Inf.3, Inf.4
and Inf.5.
x
The terms “Sensitive information” and “Special
care-required personal information” are unified to"Sensitive information". ͆Sensitive information”
includes Inf.4 and Inf.5.
x"Special care-required personal data" includes Inf.6.

 6SHFLDOFDUHUHTXLUHGSHUVRQDOGDWD
,QI  '$7$$1',1)250$7,2121616

In the case of Issues of Special care-required personal
data (Inf.6), which requires careful handling, there are two
factors that cause it to be posted, although we are carefulwhen posting it to SNSs. The first point is "in the casewhich it is used for self-disclosure, entertainment, andsocializing with friends." The second point is "in the case ofhuman error".
The first point is that there are many cases where an SNS
is used for self-disclosure, entertainment, and socializing
with friends. Reasons for participating in the "online

It is easy to judge whether Inf.1 corresponds to Personal
Data. Personal Data that individuals should pay particularattention to when posting on SNSs are Inf.4, Inf.5 and Inf.6
that require careful handling. In the research of
Humphreys.L et al. [11], Simple information with personal
identification such as e-mail address was posted on SNSs
turns out that this is rare.

104

International Workshop on Informatics ( IWIN 2022 )community," which can be said to be the predecessor ofSNSs, are the four factors of self-disclosure, socializing,
information acquisition, and entertainment. Entertainment
value, social promotion, and group norms have been foundto influence Facebook1's intent to use [15] [16]. In addition,
the use of Facebook contributes to the maintenance andformation of social capital for university students [17].
The second point is the case of human error. For example,
location information such as GPS is attached to SNSs
postings, and the location information function can beturned on or off depending on the situation at the time of
posting. However, forgetting to turn it off due to humanerror can lead to leakage of current location and movement
history. In a study by Tehila et al. [18], 81% (2,383) of thesurveyed Facebook users posted family-related informationsuch as family name, date of birth, and photo, according to asurvey of Facebook users. Given these facts, the possibilityof information leakage remains, and caution should be
exercised when posting on SNSs.

 ,QIRUPDWLRQ9DOXH&DOFXODWLRQ0RGHOPonemon [20] proposes a formula to calculate the average
cost per case for the occurrence of personal information
leakage. Romanosky et al. [21] have created a costcalculation model formulated from incident information of
Advicen in the United States. However, it is a formula that
focuses on the corporate cost from the amount of leaked
information instead of each personal information, and it is
not possible to calculate the price of each one just by
knowing the total price.
There are also calculation models such as conjoint
analysis [22] and CVM [23]. However, since these methods
are analytical methods for investigating the needs of victims,
they are inappropriate methods for investigating the value ofobjective information. Therefore, in order to obtain thevalue of objective information, we considered to use theresults of the trial and the ideas of companies andorganizations as the basis. Then, it is necessary toinvestigate how the users of SNSs perceive the value of
personal information and to investigate the difference fromthe basic axis.

 (DVHWRVHDUFKDQGHDVHWRFKDQJH
Ishikawa et al. [19] show two points to consider when
Personal Data leaks on SNSs. The first point is "informationthat can be easy to search". The second point is "informationthat can be easy to change".
The first point, " information that can be easy to search," isinformation that makes it easy to connect the person who
leaked the information and the person on the SNSs (it turns
out that they are the same person) when the information
leaks. For example, an email address is information that canidentify an individual, is easy to link to another SNSs, and iseasy to access, so care must be taken when handling it. On
the other hand, if a hobby or profession is not a special
hobby or profession, it cannot be easily identified as the
person himself / herself, so there is no need to be careful inhandling it. It is important to be "difficult to search".
The second point, "information that can be easy tochange," refers to information whose contents can be easy tochange even if the information is leaked. By changing
contents, it is possible to make it impossible to easily
identify the person who leaked the information and theperson on the SNSs as the same person. For example, thedate of birth and family structure cannot be changed, but theemail address and mobile phone number can be changed. It
is important to be "easy to change".
"Difficulty to search" and "ease to change" are major
factors influencing the value of information. Conversely,"ease to search" and "difficulty to change" are major factorsthat increase the negative effects of information leaks. These
are different from the concepts of Inf.1 to Inf.6 of Personal
Data. In other words, it is unique concepts of information
leakage by SNS's users, which is different from information
leakage of companies.

 -20RGHOJNSA [24] aggregates information on incident articles
reported in newspapers and Internet news since 2002, anddocuments related to incidents released by organizations,and calculates the estimated amount of damage
compensation for each customer of each company. The JO
model (JNSA Damage Operation Model for Individual
Information Leak) is a method that can calculate the price ofeach item and is suitable for our purpose because it isdesigned for companies and organizations to understand thepotential risk of personal information. The calculation
formula is shown below.
Damage compensation amount= Leaked information value × Social responsibility degree
× Post-event response degree= (Basic information value  Sensitive Information degree
Personal identification degree)
 Social responsibility degree Post-event response degree= Basic information value [500]6HQVLWLYLW\ ,QIRUPDWLRQ GHJUHH [max (10max (x) -1 + 5max(y) -1)]
 3HUVRQDOLGHQWLILFDWLRQGHJUHH [6, 3, 1]
 6RFLDOUHVSRQVLELOLW\GHJUHH [2, 1]
 Post-event response degree [2, 1]
The main indicators are as follows: 6HQVLWLYLW\,QIRUPDWLRQGHJUHH
x The leaked personal information is calculated using
a table showing ³SV\FKRORJLFDO GLVWUHVV´ x (3values) and ³HFRQRPLFORVV´ y (3 values)x For example, name is x = y = 1, disease name is x =1 and y = 2, etc.

 35(9,286:25.
Here, we describe a conventional model that calculates
the amount of money from the value of information.
1

Facebook is a registered trademark of Facebook, inc.

105

International Workshop on Informatics ( IWIN 2022 ) &RQYHUVLRQRILQGLFDWRUVDFFRUGLQJWR
616¶VXVHUV 3HUVRQDOLGHQWLILFDWLRQGHJUHHThere are three values of judgment criteria.
x If it is possible to easily identify an individual with"name" and "address", the value is "6".
x If there is a possibility that an individual can be
identified by adding the cost with "name" or"address + phone number", the value is "3".
x If it is difficult to identify other than the above, thevalue is "1".
 6RFLDOUHVSRQVLELOLW\GHJUHHThere are two values of judgment criteria.
x In the case of business (medical, financial, credit,etc.) that handles personal information, the value is“2”.
x In the case of other general companies, associations,and organizations, the value is "1".
However, the JO model has multiple problems [25].
Regarding the basic information value of the JO model, theconstant of 500 yen is determined by the subjectivity ofexperts, and there is no basis for it. Also, regarding thedegree of social responsibility and the degree of accident
response, the JO model is an old model designed nearly 20
years ago. The law at the time of the judgment has manydifferences from the recent law. Also, regarding thecoefficient used in the JO model, the coefficient wasdetermined by the subjectivity of the experts at that time andhas no basis.
Therefore, Yamada et al. [25] proposed a newmathematical model (here, it is called Yamada-Model).
Table 2 shows the coefficients of the new indicators(Economic Rank of Sensitive Information degree, Spiritual
Rank of Sensitive Information degree, Ease of
Identification) in Yamada-Model. This Yamada-Modelmade it a more realistic model than the JO model.

In the case of JO-Model and Yamada-Model, thecoefficients were "psychological distress" and "economic
loss". However, personal information leakage on SNSs andcorporate personal information leakage are different.
Therefore, we create a model tailored to personal
information on SNSs.
In this paper, we replace “psychological distress” with“difficulty to change” information. "Difficult to change"
information refers to information that cannot be changed,
such as the date of birth. Once leaked, it will remain on theInternet forever, causing psychological distress. Due to thiscausal relationship, "difficulty to change" information
becomes one of the factors of "psychological distress".
In the same way, we replace “economic loss" with "easy tosearch" information. "Easy to search" information refers to
information that can be easy to identify some persons withsearch, such as an email address and phone number. Once
leaked, the negative effects will be widespread rapidly andthe information be exploited, causing economic loss
possibly. Due to this causal relationship, "easy to search"
information becomes one of the factors of “economic loss".

 ,QIRUPDWLRQDQG'DWD&ODVVLILFDWLRQ
Some of the coefficients in the JO-model are subjective orbased on outdated data. Therefore, we decided to conduct
this survey in order to create objective and up-to-datecoefficients while utilizing the Yamada- model. The survey
items included a total of 15 items of Sensitive Information:
hobbies, occupation and grade, family structure, gender,
current location and travel history, landline phone number,age and date of birth, self-portrait, e-mail address, cell
phone number, address, school and work name, name, andphone number. These items were picked up from thesurveyed Inf.1 to Inf.6 basing on Chapter 3. The reason forthis categorization is that if we simply extracted only those
items that are frequently posted on social networking sites,
most of them would be attribute information (Inf.1), which
would lead to bias.
Section 3.4 discussed "ease to search" and "ease to change".
These are some of the key concepts involved in information
leakage for SNS's users. We considered applying this to the
New-Model, which is an improvement of the YamadaModel, in combination with Section 5.1. The combinations
are shown in Table 3. In Table 3, the higher up and to theright, the higher the risk.
Table 3 was created based on the following judgment inorder to process qualitative information quantitatively. Table
3 is evaluated on a three-values scale with reference to the
Yamada-Model for both "Changeable information" and"Searchable information".
x "Changeable information" conducts a graded evaluation
based on the time it takes to make changes. Specifically,
Easy (changeable within 2 weeks), Normal (changeable
within 2 weeks to 1 year), Difficult (changeable within 1
year or more before change).

7DEOH&RPSDULVRQRIFRHIILFLHQWVEHWZHHQ-2PRGHODQG<DPDGD0RGHOIndicators

Comparison
JO-ModelPsychological distress
Yamada-Model
JO-ModelEconomic loss
Yamada -Model
JO-ModelPersonal
identification
Yamada -Model

Three-step coefficient
50
51
52
1
1.0129 1.0261
100 101102
11.1723 1.3743
13
6
1
1.5158 2.8291

Note: Psychological distress and Economic loss are included
in “   6HQVLWLYLW\ ,QIRUPDWLRQ GHJUHH.” Personal
identification is “  3HUVRQDOLGHQWLILFDWLRQGHJUHH.” 1(:02'(/From the Yamada-Model, which is an improved versionof the JO model, by changing its coefficients and equations,
we can estimate the value of the information that a company
considers. And these models are a method to determine theimpact on victims of personal information leaks based on "psychological distress" and "economic loss". However, it isbased on the leakage of personal information of companies,
and is different from the leakage of personal information byusers of SNSs. Therefore, we create a new model for
estimate information values on SNSs .

106

International Workshop on Informatics ( IWIN 2022 )x "Searchable information" conducts a graded evaluation
based on the susceptibility to secondary damage. Easy
(easy to change), Normal (changeable), Difficult(unchangeable or difficult)

7DEOH0DWUL[RI6HDUFKDEOHLQIRUPDWLRQDQG
&KDQJHDEOHLQIRUPDWLRQ

Searchable information

Easy

NormalDifficultEasy
շ Emailaddress, cellphonenumberմ Current
location /movement
historyձHobbyշ

Changeable information
NormalDifficultո Address,
չMedicalschool name /records
work place,
name, family /friend's nameյ Fixed phone ն Age, date ofnumberbirth, own photoղOccupation /
Grade

ո

ճ Familystructure, gender 3ORWPHWKRGDQG)RUPXOD&RHIILFLHQWVThis section discusses ձthrough չin Table 3.
ձ For example, hobby. Least risk.
ղ For example, occupation, grade.
x Occupation is difficult to specify except for special
occupations, and it is often difficult to change
within 2 weeks.
x Also, if only the grade is leaked, it will not be
identified, but it is difficult to change within 2
weeks.
ճ For example, family structure, gender.
x Family structure and gender are difficult to identify
the individual, but unlike hobbies and occupations,
it is impossible to change or it takes more than ayear to change.
մ For example, current location / movement history.
յ For example, fixed phone number.
x Fixed phone numbers are information that issusceptible to secondary damage if leaked. And inorder to change the number, it is not permitted tochange the number except for prevention of wrong
calls or prevention of nuisance calls. Therefore, it isinformation that takes time to change. However, in
2020, the prevalence rate was 68.1% and has been
declining in recent years [21]. In addition,according to a 2019 survey by the Ministry ofInternal Affairs and Communications, 5.1% of
those in their 20s and 30s, who use SNSs, have afixed phone[22]. Therefore, it is easy to bedamaged if leaked, but the possibility of being
damaged is low in the first place.
ն For example, age, date of birth, own photo.
x Age and date of birth are information that cannot bechanged, and are information that can be easily
identified. However, it is almost impossible toidentify a person with only age and date of birth.
x Photographs are information that cannot bechanged. It is also information that can easily

չidentify an individual. For example, a photo of aperson wearing a uniform may clearly state thename of the school or company. However, before
posting on social networking sites, we often
consider the account to which we are posting andlimit the number of followers who can see the post.
For example, email address, mobile phone number.
x Both e-mail addresses and cell phone numbers are
information that is prone to secondary damage, andboth are information whose ownership has
increased in recent years with the proliferation ofcell phones. However, both are information that canbe changed on the same day.
For example, address, school name / work place,
name, family / friend's namex Addresses, names of schools and companies, andone's own name are information that can be used to
identify one's own person, and are vulnerable tosecondary damage. However, none of thisinformation is impossible to change.
x It takes a minimum of 3 days from application toapproval for a change of address, and a week to 10
days for long-term applications.
x The names of schools and companies cannot be
generally changed because of the emotional impact
of changing schools or companies, but thisinformation is not impossible to change.
x Although it is necessary to prove that the person isfree to change his/her name, it is possible to change
the name because there have been cases in the past
where people who were slandered on the Internet
and suffered from depression or sleep disorders
changed their names. It is also possible to change
one's name within two weeks to one and a halfmonths after filing a petition to change one's namewith the court.
x The names of family members and friends are thesame as one's own name.
For example, medical records. Highest risk.
x Information such as medical records is Sensitive
Information with an extremely high degree ofprivacy, and is vulnerable to secondary damage, sothat when it is leaked, it causes significant
psychological and economic damage.

In addition, Indicators of Yamada-Model in Table 2 isapplied as it is to this paper. "Personal identification" hasthree values of judgment criteria.
x If it is possible to easily identify an individual with"name" and "address", the value is "2.8291".
x If there is a possibility that an individual can be
identified by adding the cost with "name" or "address+ phone number", the value is "1.5158".
x If it is difficult to identify other than the above, thevalue is "1".
Table 4 summarizes the contents of this report. The
formulas used in this paper are based on Table 4.
107International Workshop on Informatics ( IWIN 2022 )the main the name of trial and estimation results regarding
information leakage (Table 6).

7DEOH )RUPXOD&RHIILFLHQWV

Indicators
Changeable information
Searchable information
Personal identification

Comparison
New-Model
New-Model
New-Model

Three-values coefficient
1 1.0129
1.0261
1 1.1723
1.3743
1 1.5158
2.8291

7DEOH0DMRUWULDOVUHJDUGLQJLQIRUPDWLRQOHDNDJH
Judgment
example1. 1. NTT
phone bookcase
2. Uji City,
Kyoto
Prefecture
Resident's
card dataleakage case
3. Waseda
University
Ezawa
Democratic
Seat Lecture
List
Submission
Case
4. Osu City
Information
Disclosure
OrdinanceCase
5. Yahoo! BB
customer
information
leak case
6. Tokyo
BeautyCenter
Questionnaire
Response
Information
Leakage Case
7. JAL cabincrew
monitoring
file incident

Applying the values in Table 4 to Table 3 results in Table
5. Each value is the sum of "Changeable information" and"Searchable information". This is the factor for each
information type.

7DEOH&RHIILFLHQWVIRUHDFKLQIRUPDWLRQW\SH

Searchable information

Easy(1.3743)

Normal(1.1723)
Difficult
(1)

Easy
(1)
շ Emailaddress,
mobile phone
number
(2.3743)
մ Current
location /movement
history(1.1723)
ձHobby(1)

Changeable information
NormalDifficult㸦1.0129㸧㸦1.0261㸧ո Address,
չMedicalschool name /records
work place,(2.4004)
name, family /friend's name(2.3482)
յ Fixed phone
ն Age, date ofbirth, own photo
number
(2.1984)(2.1852)
ղOccupation /Grade(2.0129)

ճ Familystructure, gender(2.0261)

It is considered that various information leakage information
is included in the amount of compensation that comes out inthe judgment of the court. And each piece of information
has a weight. Also, considering that it is easier to identify
the person by leaking multiple types of information insteadof simply adding, the following formula is obtained.
x
Number of information types: nxCoefficient for each information type: Ix (1 ≤ x ≤ n)xValue of any information: Vx (1 ≤ x ≤ n)xPersonal identification: PIxAmount of compensation for the trial: Vall

8. Benesse
Corporation
Customer
Information
Leakage Case

Vx= VallIx/㹹(I1+I2+I3+͐+Ix+͐+In)PI㹻
Here, as a premise, even if there are multiple leaked
information, if they are of the same type, they are counted asone. Using this newly created formula, the amount of
information leaked from the company is calculated.

Leakage information andamountName: 12,124 yen
Fixed phone number: 11,098 yen
Address: 12,124 yen
Name: 765 yen
Gender: 650 yen
Address: 765 yen
Date of birth: 705 yen
Family composition: 650 yen

Compensation
amount100,000 yen

Name: 451 yen
Student registration number /university name: 451 yen
Address: 451 yen
Fixed phone number: 413 yen

5,000 yen

Name: 4,501 yen
Address: 4,501 yen
Date of birth: 4,145 yen
Sensitive information: 4,526 yen

50,000 yen

Address: 443 yen
Name: 443 yen
Mobile phone number: 441 yen
E-mail address: 441 yen
Name: 1,569 yen
Address: 1,569 yen
Age: 1,445 yen
E-mail address: 1,560 yen
Occupation: 1,323 yen
Mobile phone number: 1,560 yen
Sensitive information: 1,578 yen
Address: 491 yen
Name: 491 yen
Date of birth: 452 yen
Mobile phone number: 488 yen
Fixed phone number: 449 yen
Gender: 417 yen
Sensitive information: 494 yen
Name: 176 yen
Gender: 150 yen
Family structure (scheduleddelivery date): 150 yen
Parent's name: 176 yen
Address: 176 yen
E-mail address: 175 yen

5,000 yen

10,000 yen

30,000 yen

220,000 yen

3,300 yen

Since ձ and մ in Table 3 did not exist in the trial resultsin Table 5, the price of each Personal Data was estimatedfrom the judgments of 13 items instead of 15 items. Table 7
shows the average of these. From these results, the one withthe lowest amount is the name of a family member / friend,and the one with the highest amount is a fixed telephone
number of less than 4000 yen. For example, if the allowable
range is less than 1000 yen, the information from occupation/ grade to fixed telephone number is highly valuable, so it isbetter to avoid exposing it to SNSs as much as possible.

 5(68/7
The results of a survey on value recognition regarding
Personal Data of companies (organizations) and individuals
are described.

 9DOXHRI3HUVRQDO'DWDUHFRJQL]HGE\FRPSDQLHVUsing the coefficients for each information type in Table
5 and the formula in Section 5.3, the amount of money foreach information type leaked from the actual compensation
for the court reparations was estimated. The following are108

International Workshop on Informatics ( IWIN 2022 )x When the questionnaire was conducted: November
24th to December 21st

7DEOH9DOXHRI3HUVRQDO'DWDUHFRJQL]HGE\FRPSDQLHV RUJDQL]DWLRQV 
Target Personal Data

Names of family and friends
Family structure
sexSchool name / work place
My photoemail addressmobile number
Occupation / grade
Age / date of birthSensitive information
Full nameaddress
Fixed phone number

Average unit price
(yen)
176
400
405
451
452
746
830
1323
1678
2199
2465
2565
3987

Figure 2 shows the contents of the questionnaire.
Amounts are expressed directly in yen. The results areshown in Table 8. The mean and standard deviation areabnormally large due to the influence of some outliers.
Although there is no bias, this may be due to the fact that the
respondents do not have knowledge of market prices andmake judgments based only on their senses without taking
market values into account. Therefore, we determined thatthe mean value does not represent the current situation andverified the results using the median value, which is lesssusceptible to outliers. The median, like the currency, is aseries of "0s," but it is an unadjusted, unbiased value.

7DEOH9DOXHVRI3HUVRQDO'DWDUHFRJQL]HGE\LQGLYLGXDOV㸦
㸦<HQ㸧

 9DOXHRI3HUVRQDO'DWD5HFRJQL]HGE\,QGLYLGXDOVTargetPersonal
Data

We conducted an online questionnaire on the value of
personal data for 100 university students in their second to
fourth year at Chiba Institute of Technology.

GenderFamilyStructure
Occupation/ GradeAge and
date
of
birth
Photo orvideo
ofyourself
Schoolname/work place
Emailaddress
Names offamily andfriends
Cell phone
numberSensitive
Data
Full name

The Price of Personal Information
Suppose your personal information is leaked to a stranger
through your contribution. What would you estimate theprice of the leaked personal information to be? (If you havenever posted before, please use your imagination.) (If you
have never posted before, please answer in your imagination.
Interests
Occupation / Grade
Family Structure
GenderCurrent location and travel history
Age and date of birth
Photo or video of yourself
Email address
Cell phone number
Fixed phone number
Address
Full nameName of family member or friend

Fixed
phone
numberAddress

School name / work place

Sensitive Data

Sensitive data is data with a particularly high degree ofprivacy regarding physical characteristics, thoughts andbeliefs, political or religious views

Average valueStandard deviation

114011.7
1177083998594.3
9993660

100
1000

9000000000023080000

89548869339593500000

1000

1011282249

9999375514

2000

1790629

10227958

4500

9000000000023080000

89548869339593500000

5000

275621114357866

5000

9000000000023080000

89548869339593500000

10000

22415057

134446428

10000

9.99999999999999
1039
1.00000000000002
1021
4493180

1.00498548997919
1041
9.94987437106621021

10000

19653382

10000

100106980519

999938392002

50000

The results of the survey are as follows:x Gender has the lowest amount.
x The amount of money for the address is 50000 yen,which is the highest. In other words, they are afraidthat their addresses will be clarified.

)LJXUH&RQWHQWVRIWKHTXHVWLRQQDLUHThe target audience for the questionnaire is as follows:x Gender ratio: Male: 62, Female: 35, No response: 3x Age: 18 to 24 years old (3 people answered that they
were in their 20s)x Department: Chiba Institute of Technology

109

Median10000

International Workshop on Informatics ( IWIN 2022 ) 'LIIHUHQFHVLQYDOXHSHUFHSWLRQRI3HUVRQDO
'DWDEHWZHHQFRPSDQLHVDQGLQGLYLGXDOVThe direct factor in Group A is that individuals value
gender low and companies highly value occupations andgrades. In addition, these two pieces of information arehighly evaluated by companies because they recognize thatthey are elements that can be used to attract customers, suchas the Recommend function. On the contrary, for individuals,
it is "attribute information that cannot be identified by one
piece of information", and even if it is leaked, it will not bedirectly harassed. Therefore, companies recognize that they
are more valuable than individuals. It is necessary torecognize that the information in this group is more valuable
than the individual thinks.
Next, consider Group B. Leakage of “Photo or video ofyourself”, “School name / work place”, “Cell phone
number”, “Address” and “Names of family and friends”
may be abused by a malicious person. Companies also
recognize the important value from the amount of money,
but individuals recognize that it is even more important.
There are three possible reasons why individual value
recognition has become extremely high.
One is fears about the damage that can occur when
information is leaked. Second is vague fears that the person
actually burned up in the post and was personally identified
and learned the fact of slander in the news etc. Third,
privacy influences value recognition. Sensitive Information,“Photo or video of yourself”, etc. are information related to
privacy. This is less scary than the leak of “School name /work place”, “Cell phone number”, “Address” and “Namesof family and friends”, but there is a dislike for theunspecified number of people to know their privacy, somany SNSs It is a factor that boosts individual value
recognition of posted information. Compared to corporate
value recognition, individual value recognition was
excessive. This is different from our assumption, and it was
found that individuals have higher value recognition for
more information posted on SNSs than companies.

 Since the amount of money for companies is based on theopinions of lawyers and experts, and the amount of money
for individuals is based on subjective recognition, it isjudged that the influence of bias is less for companies andorganizations than for individuals. The magnification ofindividual value recognition is shown based on the company/ organization. The results are shown in the table below.
(Table 9) As a result, it was found that the value recognition ofindividuals is low for gender, occupation and grade, but thevalue recognition of other items is higher for individuals
than for companies / organizations. In other words, it can beseen that the value recognition is higher for individuals thanfor companies as a whole. In addition, the personal
information with the smallest difference in value recognition
between individuals and companies / organizations was age/ date of birth, and on the contrary, the difference in value
recognition of 10 times or more occurred in school names /workplaces and mobile phones. The numbers and addresses,
especially the names of family and friends, differed by morethan 50 times.

7DEOH0XOWLSOLHURILQGLYLGXDO3HUVRQDO'DWDYDOXHUHFRJQLWLRQEDVHGRQFRPSDQ\No.

Leakage information

1
2
3
4
5
6
7
8
9

Gender
Occupation / gradeAge and date of birth
Family Structure
Fixed phone number
Full nameSensitive Dataemail address
Photo or video ofyourself (photo)
School name / work
place
Cell phone number
Address
Names of family andfriends

10
11
12
13

Comp
any

Individual

405
1323
1678
400
3987
2465
2199
746

100
1000
2000
1000
10000
10000
10000
5000

Individual value
recognition ratio
when thecompany is set to
1
0.25
0.76
1.19
2.50
2.51
4.06
4.55
6.89

452

4500

9.96

451

5000

11.08

830
2565

10000
50000

12.05
19.49

176

10000

56.69

 &21&/86,21
In this paper, as a result of investigating the value of theinformation posted on SNSs posts, it was found that thevalue recognition of individuals is higher than that ofcompanies (organizations) for many information. This isbecause there is anxiety about damage after information
leakage, vague fear of actions that users do not want, andinvasion of privacy. In addition, it was found that
information related to attracting customers, such as gender,
has a higher recognition of the company's value than
individuals.
As future issues, since the tolerance is 10% in thisquestionnaire, we will increase the number of respondents,
reduce the tolerance, investigate whether there is any change
in the results, and the influence of the difference in value
recognition between companies and individuals. At the sametime, in order to reduce the number of outliers, we plan toreview the questions and content of the questionnaire.

 ',6&866,21
First of all, it can be seen that individuals have a higher
recognition of the value of Personal Data than companies.
The results can be divided into the following two groups.
A) Information that companies value more than
individuals
xGenderxOccupation / gradeB) Information that individuals value higher thancompaniesxAge and date of birthxNames of family and friends

110International Workshop on Informatics ( IWIN 2022 )students use facebook?, Computers in Human
Behavior, Vol.27 Issue4, pp.1337-1343 (2011).
[17] Nicole B.Ellison 㸪 Charles Steinfield and Cliff
Lampe, The Benefits of Facebook ͆ Friends ͇㸸Social Capital and College Students̓ Use of Online
Social Network Sites, Journal of Computer-MediatedCommunication, Journal of Computer-MediatedCommunication, Vol.12 Issue4, pp 㸬 1143-1168(2006).
[18] Tehila Minkus, Kelvin Liu and Keith W.Ross,
Children Seen But Not Heard: When Parents
Compromise Children's Online Privacy, Proceedingsof the 24th International Conference on World Wide
Web, pp.776-786 (2015).
[19] Tomohisa Ishikawa and Kouichi Sakurai, A Study ofCompensation in Personal Information Leakage,Computer Security Symposium 2014, pp.1185-1191(2014) (Japanese only).
[20] Calculating the Cost of a Data Breach in 2018, the
Age
of
AI
and
the
IoT,
https://securityintelligence.com/ponemon-cost-of-adata-breach-2018/[21] Sasha Romanosky, Lillian Ablon, Andreas Kuehn
and Therese Jones, Content analysis of cyber
insurance policies: how do carriers price cyber risk?,
Journal of Cybersecurity, Vol 5, Issue 1, pp.1-19(2019).
[22] J. Gregory Sidak and Jeremy O. Skog, Using
Conjoint Analysis To Apportion Patent Damages,
The Federal Circuit Bar Journal Vol. 25, No. 4,
pp.581-620 (2016).
[23] Hong Kwon, Eun-Ju Lee, Tae-Sung Kim and HyoJung Jun, Estimating Compensation for Personal
Information Infringement in Korea Using Contingent
Valuation Method, Journal of the Society for theProtection of Intelligence, Vol.22 No.2, pp.367-377(2012) (Korean only).
[24] Investigation Report on Information Security
Incidents,
https://www.jnsa.org/result/incident/data/2016inciden
t_survey_attachment_ver1.0.pdf[25] Michihiro Yamada, Hiroaki Kikuchi, Naoki
Matsuyama and Koji Inui, Proposal on a
Mathematical Model to Estimate Loss of Leakage of
Personal Data, Information Processing Society ofJapan, Vol.60 No.9, pp.1528-1537 (2019) (Japaneseonly).

5()(5(1&(6[1]

How to control your Twitter experience,https://help.twitter.com/en/safety-andsecurity/control-your-twitter-experience[2] E. Christofides, A. Muise and S. Desmarais, Risky
disclosures on Facebook㸸The effect of having a badexperience on online behavior, Journal of Adolescent
Research, Vol.27, pp.714-731(2012)㸬
[3] Brian Stanton, Mary F. Theofanos, Sandra Spickard
Prettyman and Susanne Furman, Security Fatigue,
IEEE Computer Society, pp.26-32 (2016)[4] Amended Act on the Protection of Personal
Information,
https://www.ppc.go.jp/files/pdf/APPI_english.pdf[5] Difference between “personal information” and“privacy”,
https://privacymark.jp/wakaru/kouza/theme1_03.html(Japanese only)[6] Japanese Industrial Standards, JIS Q 15001:2017
Personal Information Protection Management System– Requirements (Japanese only)[7] Whatispersonal
data?,
https://ec.europa.eu/info/law/law-topic/dataprotection/reform/what-personal-data_en[8] Part 1: Special Feature: The Data-driven Economy
andSocialTransformation,
https://www.soumu.go.jp/johotsusintokei/whitepaper/ja/h29/html/nc121100.html (Japanese only)[9] Promoting the Utilization of Personal Data andEnsuring
Consumer
Confidence,https://www.ppc.go.jp/files/pdf/report_office.pdf(Japanese only)[10] Report of the Study Group on the Use andDistributionofPersonal
Data,
https://www.soumu.go.jp/main_content/000231713.p
df (Japanese only)[11] Lee Humphreys, Phillipa Gill, and Balachander
Krishnamurthy, How much is too much? Privacy
issues on Twitte, Proc. ICA’10, pp.1-30 (2015).
[12] Kenichi Ishii, The “Strong-tied” SNS and “Weaktied” SNS: A Comparison Regarding the Disclosureof Personal Information and Personal Relationships,
Information Processing Society of Japan, Vol.29
No.3, pp.25-36 (2011) (Japanese only).
[13] Hirotsune Sato and Naoya Tabata, Creating an
Internet version of the Privacy Dimension Scale,Japan Society of Personality Psychology, Vol.21
No.3, pp.312-315 (2013) (Japanese only).
[14] Shimon Machida, Tomoko Kajiyama, Shigeru
Shimada and Isao Echizen, Settings of AccessControl by Detecting Sensitive Data Leaks in SNS,
Information Processing Society of Japan, Vol.55
No.9 2092–2103 (2014) (Japanese only).
[15] Kenichi Ishii, Uses and Gratifications of Online
Communities in Japan, Observatorio, Vol.2 No3,
pp.25-37 (2008).
[16] Cheung,Christy,M.K. 㸪 Chiu, Pui-Yee and Lee,
Matthew K.O, Online social networks: Why do

111

International Workshop on Informatics ( IWIN 2022 )Organization Training Formulation Method in Project Management- Building a Skills Matrix with Rubric Assessment
Akihiro HAYASHI† , Michi KOMURA, Midori ISHIHARA
Department of Information Design, Shizuoka Institute of Science and Technology, Japan
Osaka Metropolitan University, Japan
ATAS Laboratory, Japan{pixysbrain, mitikomm347, midorippa2021}@gmail.com
Abstract - Development projects in companies have thecritical task of employee development in addition to project
goals such as improving QCD. The members assigned to a
project are given goals, and they learn by experience throughon-the-job training (OJT) to acquire the skills necessary for
their respective organizations. Currently, CMMI and PMBOK, which are used as standards for project management,
refer to organizational training, but it is not clear how to proceed with OJT in development projects and how to evaluate
the results of such training. This study proposes a method toformalize OJT in project management by using rubric evaluation.
Keywords: Project Management, OJT, Rublic Evaluation

1 INTRODUCTION
The organization or projects that develops a system has thecritical task of training its employees. It is as important asproviding products and services to customers.
Employee training in a company mainly consists of grouptraining and on-the-job training (OJT).When new graduates
join a company, there is a training period of several weeks toseveral months. Training is provided mainly through group
education. However, since only textbook content is taught inclassroom training, specialized business knowledge is transfered through OJT through real work after being assigned development projects. OJT is deﬁned as an educational method
in which employees are given appropriate roles and responsibilities within a project, and are guided by their supervisors and senior staffs in the workplace to acquire knowledge,
skills, etc. through actual work.For example, in the case of asoftware development company, programming languages canbe taught to freshmen through group training. Many interactions with long-time customers are undocumented. Suchcontent is passed on through OJT.
However, previous studies have not established best practices in organizational training. The method of training mayvary widely depending on the nature of the company’s business and organization.In some cases, common content is provided to employees in accordance with training manuals, whilein other cases, as in the old apprenticeship system, employees

115

are told to watch and learn from their seniors.
When planning a development project, a deadline is set justin time for the estimated man-hours. There is usually notenough time or cost to meet these deadlines. Even if employees set goals and create training plans within the project,they cannot charge the customer for the time and cost of the
training, which is added to the project cost. The plan often
falls through.
Even when organizational training is conducted using classroom lectures or on-the-job training, the evaluation methods
are often not clear. They may be evaluated based solely on
the subjectivity of the observer, or from perspectives that thelearner cannot even imagine. Sometimes, personnel evaluations may be mixed.In order to avoid such a situation, it isconsidered necessary to provide clear indicators and measures
before undertaking organizational training, to conﬁrm eachother’s training goals, and to have a system to evaluate thelevel of achievement upon completion of the project.
In this study we propose a method that applies the conceptof rubric evaluation to organizational training in development
projects to make it efﬁciently and effectively. Rubrics are asystem of evaluation used for quality assurance in university
education.It uses rubrics consisting of measures of the success of learners’ performance and descriptions of the characteristics of each measure.The rubric is used as an indicator toevaluate the achievement of learners by presenting a scale ofqualities and abilities required by the organization to learners
in advance.
Similarly, the rubric is provided at the start of organizational training to activate communication between instructor
and learner by conﬁrming training objectives. Provide timely
feedback to avoid one-way teaching and learning, and assesslearner’s achievement in accordance with the agreed indicators at project completion.
As for previous studies in this ﬁeld, Ozawa et al[1] pointed
out that classes that emphasize learner autonomy are implemented rather than one-way lectures from teachers, such asproblem-based learning and learner-constructive classes. They
pointed out the importance of the learning environment inlearner-constructed classes. Hayashi and Terashima et al[2]clariﬁed the evaluation criteria between learners and evalu-

International Workshop on Informatics ( IWIN 2022 )ators as an issue in measuring educational outcomes.They
point out the adverse effects of emphasizing evaluation information from the observer and unilateral grading from theevaluator.Nagata et al[3] report a practical example of incorporating portfolio creation activities and evaluation in a teacher
training course, using a class in a different grade level.There
have been studies using rubrics for educational activities andlearner self-evaluation at universities and other institutions.Thereis no particular discussion of the application of rubrics to organizational training in system development projects to improve the process.
In this paper, Chapter 2 describes the current status and issues of organizational training in system development projects.
In Chapter 3, we explain the concept of rubric evaluation tosolve the issues clariﬁed in Chapter 2. Chapter 4 describes
future issues.

2 CURRENT SITUATION AND ISSUES OFORGANIZATIONAL TRAINING ON
DEVELOPMENT PROJECTS
2.1 Organizational Training at MMICMMI (Capability Maturity Model Integration) is currently
the most commonly used process management method in companies, and it describes the basic behaviors that should be encouraged in organizations and projects that develop systems.
22 processes are presented as best practices in ﬁve levels, andthe maturity levels are graded.One of the processes in CMMI
Maturity Level 3 is ”Organization Training”. It describes thetraining to be conducted in organizations and projects.
Organizational training in CMMI is described as PDCA
(Plan, Do, Check, Act) of process management like otherprocesses. By identifying the training needs of the organization and one-year organizational training is planed. Then, training is conducted in accordance sith the plan, and the degree of achievement of organizational training is evaluated at
progress meetings and project completion meetings. Corrective actions are taken if nessesary when you see a difference
between the plan and the actual results.
Organizational training includes group training and on-thejob training.Standardized content with textbooks, such as ITSS
(Skill Standard for IT Professionals), is suitable for grouptraining.The company deﬁnes the necessary skill sets, the internal training committee prepares training materials, and grouptraining is conducted through classroom lectures by in-house
instructors.
On-the-job training is best suited for specialized content
that has been passed down from generation to generation inan organization. OJT requires more experience and is not always easy to learn. For example, even if a new employee
understands the operation manual, there may be a difference
in work speed and product quality between a new employee

116

and a skilled employee. This is called a learning curve, andwork efﬁciency increases as more experience is gained. Thisis due to the acquisition of tacit knowledge through experience. Newer employees, who still have less experience, lackit. This difference occurs. There is also a ”sense” or ”feeling”
that comes with experience.In order to develop skilled employees to the level of skilled employees, the skills and techniques of skilled employees must be transferred to younger
employees as well.
However, most skills and technologies are tacit knowledgeof skilled employees. It is not easy to transfer tacit knowledge
to others.Since it is difﬁcult to transfer tacit knowledge, there
is no choice but to have young employees acquire tacit knowledge through experience, just as skilled employees do.Since
it is difﬁcult to hand down tacit knowledge to others, young
employees, like skilled employees, have no choice but to acquire tacit knowledge through experience.Since it is difﬁcult
to hand down tacit knowledge to others, it is necessary to haveyoung employees acquire tacit knowledge through experiencein the same way as skilled employees.
CMMI is the most reliable best practice in process improvement. It describes methodologies. It tells you what to
do to help your organization acquire the skills it needs, but
not how to do it. Even with CMMI in place, there is no established methodology for successful organizational training.

2.2 Requirements for Organizational Training
In this study, we refer to the players in organizational training as ”instructors”, those who teach, such as lecturers, senior
employees, skilled workers, and mentors; ”learners”, those
who receive training, such as new hires, young employees, inexperienced workers, and mid-career hires; and ”evaluators”,
those who judge the improvement of learners’ skills. Thissection summarizes the requirements for establishing an organizational training process by dividing the organizational
training into return on investment, instructors, learners, andevaluators.
2.2.1 Return on Investment(ROI)
Providing organizational training in a company means sacriﬁcing direct hours to make money in the company’s core business and providing training as an indirect operation. From theorganization’s point of view, providing organizational training is an investment. Since it is an investment, it is necessary
to measure the ROI to ensure that the investment has been
worthwhile.
For example, in cases where a company pays the examination fee to take a qualiﬁcation examination, the investment
has been recovered if the employee passes the examinationand obtains the qualiﬁcation. Group education conducted inclassrooms and on-the-job training (OJT) within a project are
similar investments. There is no clear means to measure theInternational Workshop on Informatics ( IWIN 2022 )much the training cost. Ensure that the technicians who complete the training are performing reasonably well on the next
project. For instructors, assess whether they are skilled enoughto fulﬁll the roles of instructor, educator, and mentor. If not,a train-the-trainer is needed. For the learner, the results of
the organizational training are assessed and registered in askills matrix to evaluate the effectiveness of the organizational
training.
An important aspect of the evaluator’s perspective is to distinguish between the role of the project and the evaluationof organizational training. Evaluators evaluate performanceagainst project roles. Project performance and organizational
training outcomes do not necessarily coincide. Both instructors and learners prioritize the role of the system development
project. And, organizational training evaluations and personnel evaluations should not be mixed.

effectiveness and determine that the investment has been recovered.
The concept of ROI for organizational training differs depending on the position of the players. Even if a learner tries
to acquire the expected skills, the evaluator may have a different point of view. Meaningful organizational training cannot
be expected unless fair, accurate, and honest evaluations areguaranteed.
2.2.2 Instructors
In university education, active learning and role-playing havebeen introduced instead of one-way mass-production education. In-house training is not limited to classroom lectures,
but also includes a variety of exercises. This is to make surethat the trainees have fully digested the content of the training.
Even in the case of OJT, the assigned roles and responsibilities in the project structure chart are presented to clarify
the skills to be acquired by the learner. Evaluate whether thelearner can perform the work to earn as a professional in thedevelopment project.
Some instructors, however, are more of the apprentice type,where the learner is expected to watch and learn from thework of the instructor and other skilled workers. In suchcases, the instructor knows the skill set to be acquired by thelearner. However, because it is tacit knowledge, it is difﬁcult
to transfer to the learner and does not lead to the expected results. Therefore, the instructor is required to clarify the content of the training in advance and to agree with the learner.

2.3 Issues to be solved
Leaders, learners, and evaluators are involved in organizational training from different perspectives. To evaluate organizational training, it is necessary to measure that the results
are commensurate with the investment.
The issue to be resolved is to establish a methodology for
efﬁcient and effective implementation of organizational training.

3 ORGANIZATION TRAININGFORMULATION METHOD USINGRUBRIC ASSESSMENT

2.2.3 Learners
In Chapter 3, we propose a method of organizational training formallation using rubric assessment to solve the issues
presented in Chapter 2.

Organizations provide training to learners in order to help
them acquire the skill sets they need. For example, project
manager skills, conﬁguration manager skills, advanced C programming skills, meeting minute taking skills, and so on.
From the learner’s perspective, the purpose of organizational
training may be to study in an area of interest, to study for acertiﬁcation, or for a career path to self-fulﬁllment.
The skills required by the organization do not always matchthe skills required by the learner. Providing one-way educational content may not be necessary for learners. Learners do
not always overtly promote their own skill sets, leading to adiscrepancy between instructor and learner awareness.
To efﬁciently and effectively conduct organizational training, learners self-assess and disclose their skill sets. Agreement on the skill sets and content to be taught and appropriate
feedback during learning are necessary.

3.1 What is Rubric Evaluation?
A rubric is a ”descriptive form of evaluation criteria thatconsists of a scale indicating the degree of success of a learner’s
performance and descriptive words describing the performance
characteristics found in each scale”[1].
Rubric assessment, widely used in international university
education, increases fairness, objectivity, and planning in thegrading of courses. It has been conﬁrmed to be effective
for regular evaluation through prior presentation of evaluation
criteria and feedback.

3.2 Suggested Rubric for Organizational
Training

2.2.4 Evaluators

This study aims to realize efﬁcient and effective organizational training by applying rubric evaluation, which is widelyused in universities, to development projects.

Evaluators need to evaluate organizational training one step
above instructors and learners; to assess ROI, examine how

117International Workshop on Informatics ( IWIN 2022 )Figure 1: Rublic for Organizational Training

Using the rubric, learners will have a clear guide for learning activities and self-assessment in organizational training.
This leads to the discovery and improvement of their own
learning issues. By clarifying achievement levels, rubrics canalso be used to evaluate management skills and communication skills that are difﬁcult to assess. By using the rubric to reﬂect on their own learning, learners maintain their motivation
for learning and sense of purpose. To make self-evaluationand self-improvement a routine, and to encourage students tolearn proactively.

scale.
The FLPN rating in CMMI is Fully (100-86 ％), Largely(85-51 ％), Partially (50-16 ％), and Not (15-0 ％). This is nota numerical value in a quantitative sense, but rather a somewhat sensory evaluation of the degree of skill when a perfectscore is set at 100 ％. Not only CMMI, but also ISO/IEC
15504 and other standards use a 4-point scale. ISO/IEC 15504
also uses a 4-point scale.

3.3 Organization Training Process Using
RubricIn this study, we propose a rubric for organizational training as shown in Figure 1, targeting organizations that are
executing system development. In Figure 1, the subjects ofthe evaluation indicators on the left vertical axis are the skill
items required by the organization. The major items are management, engineering, and communication. Management includes project management and process management, engineering includes development skills and business knowledge,
and communication includes progress meetings and stakeholders. The evaluation criteria on the upper horizontal axis
describe the evaluation perspective for each of the middle
items on the FLPN (Fully, Largely, Partially, Not Satistifed)

The organizational learning process based on rubric evaluation proposed in this study consists of three phases. Each
phase is described in detail in the following sections.
3.3.1 Phase 1
Phase 1 is to present the rubric rating scale in advance, so thatthe instructor and the learners can clearly deﬁne the rubric’s
points of view. The rubric is then shared with the learner. The
rubric shown in Figure 1 is somewhat abstract as an evaluation scale. However, because it is written in an abstract man-118

International Workshop on Informatics ( IWIN 2022 )Partially, and Not Satisﬁed stages. Learners can objectively
grasp the level of proﬁciency of their own skill set, and the
rubric serves as a checklist to guide them to the next step[4].
Instructor Feedback Instructors and learners share the rubrics
and their evaluation perspectives in advance and are awareof the purpose of organizational training. However, once thework begins, deadlines and deliverables must be met, and the
rubric perspective is disregarded.
Instructors provide feedback at appropriate times, such asat the end of the lifecycle, to let them know what the ﬁnal
evaluation is likely to be if they continue at the current pace.
Rubric Grading by Evaluator After the development project,
the evaluators, with the input of the instructor, will give arubric grade of the learner’s proﬁciency level. Free-text comments are also allowed, and comments are included to help
with future improvements.
3.3.3 Phase 3
Phase 3 is the phase in which the rubric grading results areregistered in the skills matrix and a pass/fail decision is madefor organizational training.

Figure 2: Organization Training Process Using Rubric
ner, it can be applied to the development projects that you arein charge of by interpreting and deciphering the rubric.

Skill Matrix Registration The FLPNs, which were rubricgraded by the evaluators in the second phase, are used as
quantitative values to measure employee growth. The FLPN
ratings are graphed as F=4 points, L=3 points, P=2 points, and
N=1 point, respectively. The system manages the progress ofeach employee’s skill development to the set goal. Figure 3
below shows an illustration of the system (this is only an illustration, not an actual measurement).
The graph on the left plots the skill development of the engineers every six months. The employees’ skills have progressed to the values that they set as their skill improvementtargets. The right ﬁgure is a radar chart for the middle itemsof the rubric presented in Figure 1. The rubric indicates the
skills that the organization requires of the learner, but not necessarily the skills that the learner wants to acquire. It is also
useful to identify the type of engineer.
For example, in an organization that adopts the staff engineer system, all employees who join the company as new
recruits are promoted along the same route until they reachthe rank of chief.
However, when moving up from the chief position, employees can choose either the management course or the staff
engineer course. One is the managerial course, in which theemployee progresses from chief to section manager. The otheris a course in which the employee receives the same salary asa manager but has no subordinates and is promoted as an engineer.

3.3.2 Phase 2
Phase 2 describes how to actually use the rubric within theassigned development project.
Self Assessment The learner self-assesses the six medium
items as objectively as possible according to the rubric provided, and evaluates them on the FLPN. This allows them tovisualize their own rubric.
In the FLPN evaluation, F and L are strengths, and P and N
are weaknesses. The self-assessment results show that the P
and N are the weak areas of one’s skill set, and that by focusing training on these areas, one can acquire a well-balanced
skill set.
Conduct Gruop Training and OJT Throughout the appropriate period of time, organizational training is conducted
through group training or on-the-job training, according to theorganizational training plan. Self-study time is not included
in organizational training.
The rubric shown in Figure 1 evaluates the degree of skill
acquisition required by the organization using the FLPN, andshows not only the ﬁnal form of the acquired skill set, but also
the intermediate stages of the skill set in the form of Lagely,

119

International Workshop on Informatics ( IWIN 2022 )Figure 3: Graph of skill development based on rubric (image)This is because a good engineer does not necessarily become a good manager.It is unwise to force an engineer who
is not suited for a managerial position to become a manager.
Even if a good engineer becomes a manager, if he or she becomes too busy with management duties, he or she will nothave time to engage in development work and will not be able
to take advantage of the high skills that are available in thecompany. From an organizational point of view, this is equivalent to losing effective resources. Such a type of engineer
may choose to progress from staff engineer to senior staff engineer. Adopting the rubric would be effective for this type of
job-type decision.

evaluating its application in actual work. This is a future task.

ACKNOWLEDGMENTS
In Conducting This Research, We Referred To The Activities Of ATAS (Arts Therapy Activity Study), a collaborative
research project on the spread of various arts therapy activities
in local communities. We would like to express our deepest
gratitude to ATAS.

REFERENCES[1] Ozawa S et al, Characteristics and Structures of Learning Environment Design in the Learner-Constructing
Course : Analysis of Planning the Joint Camp in DistantInter-Seminar Using a Bulletin Board System, Japan Society for Educational Technology, PP.143-154,2002[2] Hayashi T. et al, Possibilities for University Classroom Research Focusing on Evaluation Informationfrom Participating Observers, Nagasaki University Center for Continuing and Lifelong Education, Bulletin 5, ，PP.171-182, 2006s[3] Nagata T. et al, A Practical Study of a Course for
Prospective Teachers Involving Portfolio Development
in a CSCL Environment That Enhances Various Kindsof Social Interactions, Journal of Japan Society for Educational Technology, PP.218-224, 2002
[4] Shunichi Fukuyama, Procedures for implementing
checklists that provide guidance for continuous improvements in software processes, Journal of Information Processing, Vol.42,No.3, PP.529 − 541, 2001
[5] Nishioka K. Rublic, Rublic, Gendai Kyoiku Houhou
Jiten, Toshobunka．PP.293, 2004

Performance Evaluation Finally, the organization training
is a go/no go decision. Organizational training is an investment, and we want to ensure that the skills are developed tothe target level in a single organizational training session.
However, as Figure 1 shows, it will take nearly 10 years
for a new employee to develop his or her skills to the level ofa project manager.It is not possible to create a PM with only
one or two organizational training sessions.It is not possible
to create a PM with only one or two organizational training
sessions. After rubric grading by the evaluator and registration in the skills matrix, performance is evaluated.

4 FUTURE ISSUESThis study points out the issues of organizational training
and proposes a formalized method of organizational training
using rubric evaluation.
The author works as a university faculty member. He is
not in charge of development projects. The proposed methodin this study has not been applied to an actual development
project. The author will further reﬁne the research content,
propose it to companies, and measure its effectiveness by120

International Workshop on Informatics ( IWIN 2022 )3URSRVDOIRU&RQWLQXRXV,PSURYHPHQWRI(PSRZHUPHQW$UWV7KHUDS\ $0HWKRGRI6HOI$VVHVVPHQWE\$SSO\LQJ5XEULF
Michi Komura*, Midori Ishihara**, Akihiro Hayashi****Osaka Metropolitan University, Japan**
ATAS Laboratory, Japan***
Shizuoka Institute of Science and Technology, Japan{mitikomm347, midorippa2021, pixysbrain}@gmail.com$EVWUDFW - It is required to establish a method of objectively
grasp the capabilities of practitioners (individuals or groups)of Empowerment Arts Therapy that are based on arts
therapy or the therapeutic effects of art, which is conductedin local communities etc., to improve the quality of activities.
In this study, we propose a self-assessment method of
capability of practitioners by applying rubric evaluation
system, which can reveal their strengths and weaknesses andidentify areas to improve. This method will enable
practitioners to conduct continuous improvement for theircapabilities and the quality of activities.

In this study, the authors propose a rubric-based selfassessment method for continuous improvement of EAT.
Chapter 2 outlines previous studies on EAT and the issues
that need to be resolved. Chapter 3 describes thesignificance and expected effects of rubric-based selfassessment. Chapter 4 presents future issues and researchprospects

 %$&.*5281' $1' ,668(6 ,1 ($7678',(6 3UHYLRXV6WXGLHVRQ($7.H\ZRUGV: Empowerment Arts Therapy, Rubric assessment,

Social activity

In 2008, the Konan Institute of Human Sciences (KIHS)
at Konan University launched a joint research project,“Interdisciplinary Research for Establishing a Common
Platform for Study of Arts and Arts Therapy.” As a part ofthe project, a questionnaire and interview survey of arts
therapists were conducted. As a result, various aspects ofarts therapy in Japan and its social significance wereclarified. It was confirmed that arts therapy activities exist
not only as psychotherapy based on a therapeutic approach,but also as empowerment, mainly for the purpose ofmaintaining and improving health and QOL, and that they
are functioning effectively in local communities. At thesame time, the issues related to training of the practitionersof these activities emerged [1].
Based on the results of this research by KIHS, a joint
research project targeting various types of arts therapy
activities and their practitioners in local communities was
newly launched in 2012, which conducted a nationwide
questionnaire survey, interview survey (semi-structured
interviews), and on-site observation of activities. As a result,the status quo and characteristics of arts therapy in thecommunities were identified as well as the issues that need
to be addressed for its development. Furthermore, arts
therapy mainly intending support rooted in daily life wasdefined as EAT, and the requirements for EAT activities andpractitioners were developed in 2017 [1].
The surveys above revealed that the main issues related to
EAT are sustainability (management structure) and quality
assurance (quality of services, skills of the staff). 65% of the
respondents answered they earned "one million yen or less"through arts therapy activities and spend for arts therapy
activities for “10 hours or less” per week [2]. Considering
this reality, financial independence through EAT activities
alone is not feasible in the short term. Therefore, it is ,1752'8&7,21
Since 2012, the author has been engaged in a collaborative
study on the expansion of various kinds of arts therapy
activities in local communities (Arts Therapy Activity Study,
hereinafter referred to as “ATAS”).
Arts therapy is a type of psychotherapy that uses artistic
techniques such as painting, music, drama, dance, poetry etc.
In recent years, a variety of activities based on arts therapy
or similar activities to it have been increasing in local
communities, not only as psychotherapy or psychotherapy.
ATAS distinguishes these types of activities from arts
therapy as psychotherapy for therapeutic purposes. ATAS
has defined them as Empowerment Arts Therapy(hereinafter “EAT”) and has been working to understand its
status quo and issues, and to clarify the characteristics andcompositional requirements of its activities.
Empowerment arts therapy aims to support and empower
people with relatively higher level of mental health, rather
than to treat people with mental illness. It is also conductedin an area that is closer to daily life and encompasses awider range of people than clinical arts therapy which isbased on the relationship between therapists and clients.
These various EAT activities are thought to play a certain
role in improving people's mental health and Quality of Life
(hereafter “QOL”), and in supporting and empowering
people with issues in the community.
On the other hand, many of these EAT activities facedifficulties in securing funding and human resources, and insustaining their activities. Social recognition of the activities
and their practitioners is vague, and there are no establishedcriteria or guidelines for judging the skills of thepractitioners and the content of their services.

121

International Workshop on Informatics ( IWIN 2022 )realistic to put financial independence on hold and explore
ways to increase the sustainability of the activities.
The internal factors of the practitioners, such as lack ofskills and lack of awareness, are issues of higher priority, as
they are related to the interests of the users, safety assurance,social responsibility, and above all, the credibility of EAT.
Many practitioners are engaged in EAT activities based on
their good intentions and initiative. However, there is also arisk of self-righteousness. The practitioners must avoid
being unaware of possible risks or making mistakes that
could be detrimental to the users because EAT is an activity
committed to human psychology. Recognizing possible risks
and taking appropriate measures to prevent them areessential to improve the quality of EAT activities.
In ATAS, as a response to the identified issues on EAT, theEAT composition requirements were developed andpublished as criteria for self-assessment [1]. The reason for
adopting the self-evaluation method in ATAS is that, as
described below in the Chapter 3.1, since the fields,
purposes, methods, scale, and aspects of EAT activities arediverse, establishing a uniform and objective evaluation
method would require a large-scale and long-term researchand study. On the other hand, it was an urgent issue toprevent errors in the assessment in the implementation of
EAT activities. Considering the current situation of EAT, aself- evaluation method that allows the practitioners
themselves to work on proactively was adopted.

the quality of services by having practitioners conduct
regular self-assessment and clearly identify areas in need of
improvement, and to provide concrete directions for
improvement that should be aimed for.

 0(7+2'
)25
,03529(0(172)($7&217,18286

 :KDWLV($7"
EAT is an academic descriptive concept or category thatcomprehensively describes various types of arts therapies in
the community, as opposed to arts therapy which is based onclinical and therapeutic approaches. It is not a term intended
to be used as a name for individual and specific activities ora title of activists, so EAT practitioners do not call theiractivities “Empowerment Arts Therapy” or call themselves“Empowerment Arts Therapists” in their actual activities.
The concept of empowerment in social contexts haschanged over time, beginning with its use in the civil rights
and feminist movements in the U.S. in the 1960s and 1970s,
gradually expanding its meaning and applied fields. In thelate 1990s, it expanded into the fields such as medicine,
health care, welfare, and education. Today, it targets people
with difficulties and difficulties in their lives in general, andby working on their inherent and latent resources andabilities, it has come to refer to empowering them to becomeaware of their resources and abilities, and to accompany
them in thinking, choosing, and asserting themselves
independently [4].
The introduction of the empowerment concept makes it
possible to view arts therapy on two axes: “therapy (ortreatment)” and “empowerment”. In ATAS, arts therapy
based on clinical approach is distinguished as
Psychopathological Arts Therapy (hereinafter “PAT”) fromthe concept of EAT. The distinction between EAT and PAT
makes it possible to organize the relationship between the
two and to examine various arts therapy activities in termsof both EAT and PAT elements.
The following facts about EAT were obtained from thestudy so far [1].
(1) EAT is not a therapy in the narrow sense which means
treatment of patients with mental disorders but isintended for the empowerment of people withrelatively higher level of mental health. The main
targets of EAT are, for examples, those who seek toreduce stress at work or in daily life, those who seek todevelop their skills or improve their QOL, and those
who have some mental problems but are not at thestage where treatment is necessary.
(2) The fields of EAT activities are diverse, including
welfare and education, etc. ATAS has classified arts
therapy into five categories, each organized in relation
to psychopathological and empowering approaches asfollows (Fig.1).
(3) Many EAT activities emerge from within the practiceand are developed spontaneously and autonomously.
While referring to and applying theories andtechniques of psychotherapy as necessary, thepractitioners are not bound by existing systems and ,VVXHWREH6ROYHG
The self-evaluation method proposed in ATAS was that
EAT practitioners score the degree of achievement of eachcomponent, depicts it graphically on a radar chart, grasps
his/her own characteristics and position by the result, anddiscloses it on the Internet, etc. However, it has not been
formed as a practical tool for practitioners to use, and it has
not yet been implemented. The development of a practicalevaluation tool that assesses the capability of practitioners
and contributes to the improvement of their skills and thequality of their services has been the issue to be solved.
Therefore, the authors have scrutinized and revised theevaluation criteria and methods by ATAS and proposed theAssessment Sheet for EAT Activities (hereinafter“Assessment Sheet”) [3]. The sheet is based on thecomponent requirements/evaluation criteria clarified
through ATAS evaluation. It includes a checklist of 25 items
in 8 categories (App. i) and supposed to be utilized as a selfevaluation tool by EAT practitioners. The practitioners are
asked to quantify each check item on a 5-point scale,calculate the average value for each category, and enter the
average value on a radar chart (App. ii). This allows thepractitioners to objectively view their activities andthemselves, and to visualize policy, basic stance,
characteristics, tendencies, strengths and weaknesses of theirown activities.
However, although this evaluation sheet clarifies the levelof achievement for each item, it is difficult for thepractitioners (i.e., evaluators) to grasp what specificimprovements should be made for items with low scores,
and what status to aim for as the next step. It is necessary toestablish a method to improve the skills of practitioners and122

International Workshop on Informatics ( IWIN 2022 )methods but add their own unique or original elements
to their own practice in response to the needs of the
field.

strengths and weaknesses, and contribute to theimprovement of their capability.
The purpose of using a rubric for EAT activities is selfassessment by the activity leader and continuous
improvement based on this self-assessment.
The advantages of rubric assessment in EAT activities are
as follows
1. What is required for each item is clarified.
2. What to specifically do is clarified to improve items
that are marked on low levels.
3. Practitioners can share information on what to keep
in mind when conducting EAT activities.
This will enable practitioners to identify their own
strengths and weaknesses, develop and actively promote thestrengths, and formulate plans for improving the weaknesses.
By doing this on a regular basis, it is possible tocontinuously improve and raise the level of activities. It isalso possible to form a certain common understanding of
EAT activities, which are diverse and difficult to standardize.
The proposed EAT rubric assesses the seven aspects of“Knowledge,” “Skill,” “Psychological Safety,” “Selfexploration / Self-understanding,” “Cooperation /Networking,” “Sustainability,” and “Philosophy / Mission”
on a four-point scale (Fig. 2).
The rating scale is “Level 1 (Not Satisfied)”, “Level 2(Partially Satisfied)”, “Level 3 (Largely Satisfied)”, and“Level 4 (Fully Satisfied)”.
Levels 1 and 2 can be identified as areas in need of
improvement. Level 1 is what should have been achieved,
and items checked here indicate that there is a great deal ofroom and need for improvement. Level 2 is in the process ofimprovement. Level 3 is a level that has been generally met,
and the practitioners are expected to make concrete efforts
to achieve this level first. Level 4 is the highest level thatcan be desired and is set as an ideal best practice and serves
as a goal or guideline for each item.

)LJ$UHDVDQG3RVLWLRQVRI$UWV7KHUDS\LQ-DSDQ(4) There are few opportunities to receive appropriate
remuneration for their activities, and the availability ofmanagement resources such as funds and humanresources is unstable. This fact makes the sustainabilityof their activities an issue.
(5) Since there are no institutional standards of behavior
and evaluation that define professional requirements,
there could be some cases of unsafe or self-righteous
behavior by practitioners. This poses the risk of
causing unintended psychological disadvantages tousers of services or clients.

 &RQVWUXFWLRQRI5XEULF
As described in Section 2.2, the purpose of this study is to
identify more clearly areas for improvement in EAT
activities and to develop index to show specific directions
for improvement. In this study, we propose a rubric that willserve as a guideline to indicate the direction of improvement
to be aimed for.
A rubric is a document that specifies a task description(assignment), a scale of some sort (levels of achievement,
possibly in the form of grades), the dimension of theassignment (a breakdown of the skills/knowledge involvedin the assignment), and description of what constitutes eachlevel of performance (specific feedback) [5] as the basic
elements that indicate learning achievement. In educationalsettings, it is used by teachers to provide accurate andprompt feedback to students, and by teachers and students toshare learning goals and evaluation index. In addition toteachers’ evaluation, self-assessment by learners is expected
to enhance learners’ autonomy, clarify learning goals andcontents, and enable them to proactively grasp the results
and challenges of their studies [6].
These features of rubrics include the potential to contributeto continuous self-improvement in the field of social actions
or enterprises as well as grading in school. However there
has not been discussion on the possibility of improvement
through utilizing rubrics in these fields in Japan. A rubric isalso expected to be applied to self-assessment for EAT
practitioners with diverse backgrounds, enabling them toreflect on their skills and activities, objectively grasp their



3URFHVVRI,PSURYHPHQWIn this study, the authors propose a method for continuous
improvement of EAT using the rubric indicated in 3.2. The
method consists of self-assessment using the rubric,
identification of areas in need of improvement, formulationof improvement plans, implementation of improvement
plans, and self-inspection using the checklist (Fig.3)

 6HOI$VVHVVPHQWE\5XEULF
EAT practitioners (individual or by group) periodically
self-assess, using a rubric, which rating scale each of theiractivities fits into.

 ,GHQWLILFDWLRQRI$UHDVLQ1HHGRI
,PSURYHPHQW
Based on the results, the practitioners identify their own
Strengths and Weaknesses, and then identifies the areas inneed of improvement. Items rated 1 or 2 on the rubric scale
are areas in need of improvement. When there are multiple

123International Workshop on Informatics ( IWIN 2022 )/HYHO
&ƵůůǇ^ĂƚŝƐĨŝĞĚ

 .QRZOHGJH

 6NLOO

 3V\FKRORJLFDO6DIHW\

6HOIH[SORUDWLRQ
6HOIXQGHUVWDQGLQJ&RRSHUDWLRQ1HWZRUNLQJ

 6XVWDLQDELOLW\ 3KLORVRSK\0LVVLRQ+DYHVXIILFLHWNQRZOHGJHRIDUWVWKHUDS\UHTXLUHGLQ\RXUVSHFLDOL]HGDQGUHODWHG
DFWLYLW\DUHDV+DYHJRRGFRPSUHKHQVLRQWKURXJKRXWDUWVWKHUDS\DQGEHDEOHWRH[SODLQLWIRU
RWKHUVWRXQGHUVWDQG
$EOHWRSURYLGHVDIHDQGHIIHFWLYH($7
ZLWKRULJLQDOO\GHYLVHGPHWKRGVDVZHOO
DVPHWKRGVPDVWHUHG
$EOHWRFXVWRPL]HWKHPHWKRGVDQG
SURFHVVIOH[LEO\DFFRUGLQJWRDFOLHQW V
FRQGLWLRQ
3V\FKRORJLFDOVDIHW\LQWKHDFWLYLWLHVLV
VHFXUHG
8QH[SHFWHGWURXEOHLVUHVSRQGHGVZLIWO\
DQGSURSRHUO\$EOHWRSURYLGHSURSHUDOWHUQDWLYH
VROXWLRQVVXFKDVWRLQWURGXFHWKHFOLHQWWRRWKHUUHOHYDQWDJHQFLHVVSHFLDOLVWVLQ
FDVHWKDWWKHPDWWHULVEH\RQG\RXUVNLOO
+DYHVXIILFLHQWDQGSURIRXQG
XQGHUVWDQGLQJRI\RXUGHHSLQVLGHDQG
HVVHQWLDOSURSHQVLW\<RXURZQSV\FKRORJLFDOLVVXHVKDYH
EHHQRYHUFRPH
3HULRGLFDOO\WDNHWUDLQLQJDQDO\VLVRU
VXSHUYLVLRQ+DYHUHOLDEOHFRQQHFWLRQVWRFRQWDFWSURIHVVLRQDOSHRSOHLQWKHVDPHILHOGDV\RXDQGRWKHUUHODWHGILHOGV%HORQJWRDFDGHPLFVRFLHWLHVDQG
UHJXODUO\SDUWLFLSDWHRUPDNH
SUHVHQWDWLRQVLQFRQIHUHQFHV:RUNLQJVXVWDLQDEO\DQGFRQVWDQWO\DV
ZHOODVKDYHDSRVVLELOLW\RIGHYHORSRIDFWLYLWLHV
7KHSKLORVRSK\PLVVLRQLVILUPO\HVWDEOLVKHGDQGFOHDUO\YHUEDOL]HG$EOHWRFRPPXQLFDWHWKHSXUSRVHDQG
LGHDOVRI\RXUDFWLYLWLHVWRRWKHUV/HYHO
>ĂƌŐĞůǇ^ĂƚŝƐĨŝĞĚ/HYHO
WĂƌƚŝĂůůǇ^ĂƚŝƐĨŝĞĚ/HYHO
EŽƚ^ĂƚŝƐĨŝĞĚ

+DYHVXIILFLHQWNQRZOHGJHRIDUWV
WKHUDS\UHTXLUHGLQ\RXUVSHFLDOL]HGDFWLYLW\DUHD
$EOHWRH[SODLQLWIRURWKHUVWRH[SODLQ+DYHEDVLFOHYHORINQRZOHGJHRIDUWV
WKHUDS\.QRZOHGJHLVOLPLWHGRUIUDJPHQWDO
+DYHQRWVWXGLHGDUWVWKHUDS\V\VWHPDWLFDOO\$EOHWRSURYLGHVDIHDQGHIILFLHQW($7
ZLWKPHWKRGPDVWHUHG

$EOHWRSURYLGH($7EDVHGRQWKH
PHWKRGVPDVWHUHG

+DYHYHU\IHZRUDIHZH[SHULHQFHVRI($7SUDFWLFH

$ZDUHRIWKHQHHGDQGLPSRUWDQFHWR
5HDG\WRSRVVLEOHWURXEHVWRVRPH
VHFXUHSV\FKRORJLFDOVDIHW\IRUFOLHQWVLQ H[WHQW$EOHWRVHHLIWKHPDWWHULV
WKHDFWLYLWLHVDQGQHFHVVDU\PHDVXUHVEH\RQG\RXUVNLOODWOHDVWDQGWDNHVRPH
DUHWDNHQPHDVXUHV

1RPHDVXUHVIRUV\FKRORJLFDOVDIHW\LV
WDNHQLQWKHDFWLYLWLHV'RQRWWKLQNRIDQ\WURXEOHVRXWRI\RXU
FRQWUROWKDWZLOOSRVVLEO\KDSSHQ+DYHQRZD\WRGHDOZLWKWURXEOHV

+DYHVXIILFLHQWVHOIXQGHUVWDQGLQJRI\RXULQVLGHDQGHVVHQWLDOSURSHQVLW\<RXURZQSV\FKRORJLFDOLVVXHVDUH
UHVROYHG

6HOILQVLJKWDERXWKLPKHUVHOIKDVEHHQ
GRQHWRVRPHH[WHQW$ZDUHRIKLVKHUSV\FKRORJLFDOLVVXHV
DQGZRUNLQJRQWRILQGZD\VWRVROYH
WKHP

1HYHUKDGDQ\VHOILQVLJKWDERXW\RXUVHOI
<RXURZQSV\FKRORJLFDOLVVXHVDUH
XQWRXFKHGRUXQUHFRJQL]HG+DYHUHOLDEOHSHUVRQDOFRQQHFWLRQVRUIHOORZV*DWKHULQJLQIRUPDWLRQFRQVWDQWO\7U\LQJWRKDYHDUHOLDEOHSHUVRQDO
FRQQHFWLRQVRUIHOORZV$EOHWRJHWLQIRUPDWLRQZKHQ\RXQHHG

+DYHQRSHUVRQDOFRQQHFWLRQVRUIHOORZVWRDVNRUFRQVXOWLQFDVHRIWURXEOH
+DYHQRZD\IRUJDWKHULQJLQIRUPDWLRQDERXWDFWLYLWLHV:RUNLQJVXVWDLQDEO\UHJDUGOHVVRIZKHWKHUSDLGRUQRW:RUNLQJRQRUPDNLQJHIIRUWVWRUDLVH
WKHVXVWDLQDEO\RI\RXUDFWLYLWLHV1HLWKHUWKLQNRIWKHVXVWDLQDELOLW\RI
DFWLYLWLHVQRUWKHPDQDJHULDOVLGHRI\RXUDFWLYLWLHVIRUUDLVLQJVXVWDLQDELOLW\

7KHSKLORVRSK\PLVVLRQLVHVWDEOLVKHG
DQGDEOHWRFOHDUO\H[SODLQLWWRRWKHUVFOHDUO\+DYHWKHSKLORVRSK\PLVVLRQDQG
YHUEDOL]HGWRVRPHH[WHQW%DVLFSKLORVRSK\PLVVLRQLVQRW
HVWDEOLVKHG
2UKDYHQHYHUWKRXJKWRIWKHP)LJ ($75XEULF
items requiring improvement, they are prioritized andaddressed for improvement. Basically, items with a rating ofLevel 1 are judged to be more in need of improvement thanthose with a rating of Level 2, but the priority of
improvement need not necessarily be judged mechanically.
It is advisable to make a proactive decision on what toimprove first in the light of one's own activity policy and thepoints that are important.

encourage. Following the rubric leads the practitioners to atireless and steady accumulation of the required activities.

 6HOI,QVSHFWLRQE\&KHFNOLVW
After carrying out the planned improvement activities,
self-evaluation using the checklist on the Assessment Sheet(App. i) is performed and entered in the radar chart (App. ii).
By following the changes in the shapes of the radar chart,
the process of continuous improvement of EAT activities
can be visualized. If the chart shapes become more balanced,
it can be confirmed that the improvement is well underway.
Finally, when the radar chart reaches the highest level, thepractitioners are supposed to have achieved the level thisrubric proposed and can be considered to have fulfilled therequired level of EAT activities.

 'HYHORSPHQWRI,PSURYHPHQW3ODQFor the items identified as areas requiring improvement, alist of points to be improved is made, and an improvement
plan is formulated by analyzing and discussing what points
are insufficient and how they can be improved, determining
the steps to be taken by setting priorities, deciding who will
be in charge, the period of implementation, and so on. The
improvement plan should be as specific as possible.
However, it is essential to make a plan that is feasible
according to the actual situation of the activities, because it
would be a complete reversal if the burden of improvement
activities becomes so heavy that it interferes with theprimary activities.

 ,PSOHPHQWDWLRQRI,PSURYHPHQW3ODQThe practitioners engage in improvement activities
according to the developed plan. The rubric is not only aself-assessment tool, but also a guidance of activities,
presenting basic behaviors that the practitioners should)LJ 3URFHVVRI,PSURYHPHQW

124

International Workshop on Informatics ( IWIN 2022 ) )8785( ,668(6 $1' 5(6($5&+
35263(&7672:$5'9(5,),&$7,21

Effective Feedback, and Promote Student Learning,
Second Edition, Sterling, Virginia, Stylus Publishing,
LLC, Chapter 1, 2013.
[6] Terashima, K. & Hayashi, T., Development of
Problem-Based Learning to Promote Self-Evaluation
Using Rubrics, Kyoto University Studies in Higher
Education, Vol.12, Dec.2006 (in Japanese).

In this study, a method for continuous improvement of
EAT activities using a rubric was proposed. In the future,
we would like to verify the effectiveness of this method andidentify areas for improvement by having as many EAT
practitioners as possible utilize this method. We plan tocollect and analyze as much feedback as possible, by
contacting the informants who cooperated ATAS project in
the past, developing new informants and obtaining research
grants, etc. These will be the subjects of future work, alongwith the discussion on specific methods for verification
surveys.
As Terashima and Hayashi (2006) stated, problems
associated with improvement activities based on selfassessment include the possibility that the practitioners are
unfamiliar with rubric evaluation and find self-assessment
difficult, and that it is easier to find problems than good
points, which in turn may lead to a loss of confidence intheir own activities[6]. These points need to be given
sufficient consideration in the verification surveys to be
conducted in the future.
Both rubric and checklist proposed will continuously berevised as necessary. The objective of constructing the EAT
rubric is to empower the practitioners to improve theircapability through the assessment. The goal is to establish
an evaluation index to improve the quality, sustainability,
credibility, and social recognition of EAT which are difficult
to evaluate uniformly due to their wide variety, while
drawing out their good qualities such as diversity andcreative potential. We will continue to work on thedevelopment and improvement of the tools for this purpose,as well as dissemination and raising public awareness.

5()(5(1&(6[1] Kaneko, H. & Ishihara, M., Study on Empowerment Art
Therapy Today [Empowerment gata art therapy kenkyu
no genzai], The Crisis of the Mind and ClinicalKnowledge, Bulletin of Konan Institute of Human
Sciences, Vol.20, pp.21-41, Mar.2019 (in Japanese).
[2] Ishihara, M. & Kaneko, H., Art Therapy Activities in
the Communities as Empowerment - Its Endogenous
and Autonomous Nature Seen from the Nationwide
Survey [Empowerment toshiteno shisei no art therapy
katsudo – Zenkoku jittai chousa kara mieru sono
naihatsusei to jiritsusei], The Crisis of the Mind andClinical Knowledge, Bulletin of Konan Institute of
Human Sciences, Vol.16, pp.105-130 Feb.2015 (inJapanese).
[3] Komura, M. & Hayashi, A., A Method of Assessment
for Empowerment Arts Therapy, IEICE Technical
Report - Shingakugiho, Vol.121, No.372, SWIM202131, pp.1-7, Feb.2022 (in Japanese).
[4] Kaneko, H. & Ishihara, M., A View of “Art Therapy for
Empowerment”, Bulletin of Kobe University of
Welfare, Vol.19, No.1, pp.1-22, Dec.2018 (inJapanese).
[5] Stevens, D.D. & Levi, A.J., Introduction to Rubrics:
An Assessment Tool to Save Grading Time, Convey

125International Workshop on Informatics ( IWIN 2022 )$SSHQGL[$VVHVVPHQW6KHHWIRU(PSRZHUPHQW$UWV7KHUDS\$FWLYLW\(PSRZHUPHQW$UWV7KHUDS\㸦㸦 ($7㸧
㸧 $VVHVVPHQW6KHHW
Ϩ ࠉ 6.,//62)35$&7,7,21(56̏ɻ.QRZOHGJH6NLOOV
8QGHUVWDQGDQGH[SODLQWKHWKHUDSHXWLF
ᶅ HIIHFWVDQGIXQFWLRQVRIDUWRU DUWLVWLF
H[SUHVVLRQ
3RVVHVVVXIILFLHQWNQRZOHGJHDQGVNLOOVLQ
DUWVWKHUDS\DQGDUWWRDSSURSULDWHO\ᶆFRQGXFWDFWLYLWLHVDFFRUGLQJWRWKHSXUSRVHV
DQGDLPV.QRZOHGJHDEOHDERXWWHFKQLTXHVFKDUDFWHULVWLFVKLVWRU\HWFRIDUWVWKHUDS\ᶇRXWVLGHRI\RXURZQDUHDRIH[SHUWLVHDVZHOODV\RXURZQH[SHUWLVH
3RVVHVVNQRZOHGJHDQGVNLOOVLQ
SV\FKRWKHUDS\SV\FKRWKHUDS\DQG
ᶈSV\FKRORJLFDOFRXQVHOLQJDWWKHOHYHOUHTXLUHGLQ\RXUSUDFWLFDODFWLYLWLHV0DUNLQJE\\RXUVHOI $YHUDJHᶒϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

+DYHXQGHUVWDQGLQJRIWKHQDWXUHDQG
ᶋ SRSHQVLWLHVRI\RXURZQDQG\RXU
SV\FKRORJLFDOLVVXHVDUHUHVROYHGϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

̕ 0DQDJHPHQWᶗ(QJDJHGLQDFWLYLWLHVUHJXODUO\DQG
FRQWLQXRXVO\2UZRUNLQJWRZDUGWKLVJRDO5HFHLYLQJFRPSHQVDWLRQUHPXQHUDWLRQᶘ
FRPPHQVXUDWHZLWKWKHDFWLYLW\([SHQVHVUHODWHGWRDFWLYLWLHVDUHFRYHUHG
E\FRPSHQVDWLRQIRUDFWLYLWLHVDQGRUᾝJUDQWVHWFDQGGRQRWUHO\RQRXWRI
SRFNHWH[SHQVHV$FWLYLW\DQGLQFRPHH[SHQGLWXUHSODQVDUHᾨGUDZQ

ϩ ࠉ 35$&7,&(
̑ 3UDFWLFHRI($7ϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϫࠉ 9$/8(ϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ̖ 3KLORVRSK\0LVVLRQϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

̒ 5DSSRUWZLWK&OLHQWV:RUNLQJWRREWDLQLQIRUPDWLRQDERXWWKH
FOLHQW VVRFLDOHQYLURQPHQWDQGUHODWHG
ᶐLQVWLWXWLRQVRUJDQL]DWLRQVVHUYLFHVHWF
QHHGHGWRVXSSRUWFOLHQWV
4XHVWLRQQDLUHVLQWHUYLHZVHWFDUH
ᶑ FRQGXFWHGZLWKFOLHQWVUHJDUGLQJWKH
HIIHFWLYHQHVVRI($7ϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

+DYHLQIRUPDWLRQDQGQHWZRUNVWRFRQQHFW
FOLHQWVWRWKHDSSURSULDWHH[SHUWVRUᶖ SURIHVVLRQDORUJDQL]DWLRQVGHSHQGLQJRQ
WKHFOLHQW VFRQGLWLRQDQGWKHVLWXDWLRQRIWKH
FDVHϪࠉ 0$1$*(0(1767</(ϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

:RUNLQJWRXQGHUVWDQGWKHFOLHQWVIURPᶏPXOWLSOHSHUVSHFWLYHVϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

ᶊ &RQGXFWVHOIDVVHVVPHQW

$EOHWRDFFXUDWHO\VHWREMHFWLYHVDQGJRDOVDFFRUGLQJWRWKHFOLHQW VVLWXDWLRQDQGᶍQHHGVDQGWRRUJDQL]HDQGSURFHHGZLWKZRUNDQGVHVVLRQV$SSURSULDWHSV\FKRORJLFDOGLVWDQFHLVᶎ PDLQWDLQHGZLWKWKHFOLHQWDQGDVWDEOHDQG
WUXVWLQJUHODWLRQVKLSLVHVWDEOLVKHGϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

+DYHDVXSHUYLVRURUPHQWRUWRFRQVXOWDV
QHHGHGᶕϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

DQGRUUHODWHGDUHDVRIH[SHUWLVH

7KHSXUSRVHDLPDQGWDUJHW FOLHQW RI\RXUDFWLYLW\DUHFOHDUϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

̔ 3URIHVVLRQDO1HWZRUN:RUNLQJWRLPSURYHNQRZOHGJHDQGVNLOOVᶌ0DLQWDLQFRQILGHQWLDOLW\UHJDUGLQJWKH
SULYDF\RIFOLHQWV

0DNHVXUHWKDWSDUWLFLSDQWVVKRXOGPDLQWDLQFRQILGHQWLDOLW\UHJDUGLQJWKHFRQWHQWDQG
ᶓSULYDF\RIZKDWWKH\VD\GXULQJJURXSZRUN
DQGVHVVLRQV
7KHQHFHVVDU\LQGLYLGXDOIROORZXSLVᶔ SURYLGHGDFFRUGLQJWRWKHSV\FKRORJLFDOVLWXDWLRQRIWKHFOLHQW̐ 6HOIH[SORUDWLRQ6HOIXQGHUVWDQGLQJ㻌㻌 ᶉ UHTXLUHGIRULWVDFWLYLWLHVLQ\RXUVSHFLDOL]HG̓ɻ
3V\FKRORJLFDO6DIHW\

7KHSKLORVRSK\PLVVLRQDQGWKHLGHDOVRIᾩ WKHDFWLYLWL\DUHFOHDUO\GHILQHGDQGDEOHWR
H[SODLQWKHPWRRWKHUVϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

7KHFKDUDFWHULVWLFVRI\RXU($7DFWLYLWLHV
DUHFOHDUDQGDEOHWRH[SODLQWKHPWRRWKHUVϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

0DLQWDLQDQDWWLWXGHWRSXUVXHWKHHVVHQFH
ᾫRIH[SUHVVLRQϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱᾪϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱϭ࣭Ϯ࣭ϯ࣭ϰ࣭ϱ

L

&KHFNOLVWLL

5DGDU&KDUW

126

International Workshop on Informatics ( IWIN 2022 )A Study of Progress Management Method using TA-Bot
for a Software Development Class
Kocichi Ono*, Ryozo Kiyohara‡ , and Yoshiaki Terashima** Graduate School of Science and Engineering, Soka University, Japan‡
Faculty of Informatics, Kanagawa Institute of Technology, Japane21m5311@soka-u.jp
Abstract - In this study, we discuss a method that uses TABot to manage student progress in software development classes.
TA-Bot calculates the severity of developmental delay to monitor student progress and provides technical advice based on its
experience of the class to resolve the delay. Moreover, experimentally determining the optimal timing of technical advice andthe technical advice as the system threshold for each student
requires student participation and repeated experiment. However, owing to the diﬃculty in creating the intended development
situation and forming teams of students, the experiment is not
reproducible. Therefore, we repeat experiment using simulation
and adjust the system thresholods for each student. For thispurpose, we deﬁne the student parameters required for faithfully
reproducing the development situation of students to intentionally design diﬀerent situations for each student.

experiment, parameters reproducing students’ development situation, and analyzing the evaluation result. Finally, Section5
concludes the manuscript.

2 Background
Recently, online classes are widely disseminated. Students
participation can take the class asynchronous each place. But,
they are diﬃcult to communicate with students participation,
unlike face-to-face ones. Thus, monitoring state of students’learning is a burden on a TA because it is diﬃcult for the TA tomonitor them. In addition, the more students are, the greater aburden on TA.
Education Network for Practical Information Technologies(enPiT) [?] is an education project to develop the advanced information technology human resources based on practical education in cooperation between universities and industry promoted
by Ministry of Education, Culture, Sports, Science and Technology (MEXT) of Japan. [?] enPiT has four ﬁelds; AiBiC,
Security, Emb and BizSysD, and software development is increasingly important. Now, enPiT is implemented at Japanese
universities, we presume that it will be implemnted at high schooland junior high school.
Based on these two things, the study of software development
was conducted so far.[?][?][?] So, we focus on TA monitoring students’ progress management for a software developmentclass.

Keywords: Software Development, progress management,
TA-Bot1 Introduction
In this study, the Teaching Assistant Bot (TA-Bot) is used to
manage student progress in a software development class. Conventionally, a TA provides technical advice depending on thedevelopmental situation by monitoring student progress. This
process is automated using TA-Bot. The class considered in thisstudy is conducted online, where students and teacher share information using social media. TA gets students’ progress reportand gives technical advice for students experiencing developmental delay. To perform these tasks eﬀectively, the TA-Bot
must be provided with deﬁnitions of optimal advice and timing. Thus, the developmental delay is quantiﬁed into levels andthe TA-Bot is provided with technical advice from a database.
Moreover, student participation is required to properly evaluate
the method. However, the intended development situation cannot be created owing to the disparity in the programing skill
levels of the students. Thus, the experiment is not reproducible.
To overcome this problem, the experiment is performed on asimulation of the development situation, which can be repeated
to conﬁrm each student the eﬀectiveness of the technical advice
and determine the optimal timing each student for it.
This paper is organized as follows: Section2 outlines the background of the study. Section3 explains software developmentclass and our proposed method for this study. Section4 explains

3 Software Development Class and Proposed
Method
3.1 Software Development Class
The target of this study is a software development class thatis conducted online. Figure1 illustrates the structure of the classcomprising approximately 100 students. Each student createdtheir developmental plan individually before the commencementof the class and submitted it to the TA. To help the TA evaluate
their development progress, the students specify their progress
rate and development phase (design, coding, or testing). Finally,each student develops the software asynchronously according totheir convenience.

127International Workshop on Informatics ( IWIN 2022 )Moreover, although the TA-Bot selects technical advice froma database, it does not understand how the advice aﬀects students. In other words, TA-Bot cannot select appropriate technical advice. Therefore, we need to deﬁne indicators that qualify
technical advice as appropriate. Furthermore, to provide students with the latest technical advice, we need to formulate
updated advice for the database.
3.3.1 System Conﬁguration
Figure3 shows the conﬁguration of the proposed system. The
social media for communication between students and the instructors that we use is Slack[?]. TA-Bot has a plan for each
student and acquires technical advice database. The TA, andteacher use Slack to share information with the students, who
submit weekly progress reports and specify their progress rate,
which indicates developmental progress, and developmental situation in the reports. TA-Bot has two system thresholds for each
student; TA-Bot stores the provided information in the database,calculates the severity of developmental delay by comparing the
progress rates in the progress report and the plan, and ﬁnally selects technical advice based on the severity. This type of advice
is called Static Advice. Furthermore, we store technical advice
provided in past classes and randomly add new technical advice.
TA-Bot identiﬁes keywords related to student development byanalyzing the comments on the social media, and searches forthe keyword on the web. Finally, TA-Bot stores new technicaladvice that is acquired from the web in the database. This typeof advice is called is Dynamic Advice.

Figure 1: Software Development Class

3.2 Management Problem
Based on the description above, we concluded that the burden
on a TA is large. It has to simultaneously monitor the progressof several students, which diﬀer signiﬁcantly corresponding totheir programing skills. In addition, conﬁrming the progress of
all the students is diﬃcult. Consequently, the TA cannot provide appropriately timed technical advice and the developmental
delay worsens.

Figure 2: Problem monitoring students’ progress

3.3 Progress Management Method
We propose a progress management method using TA-Bot
to resolve the aforementioned problems faced by the TA. We
automate the processes of monitoring student progress and providing technical advice, thereby lightening the workload of the
TA. Accordingly, we deﬁne two components for TA-Bot.

Figure 3: System Conﬁguration

1. Monitoring student progress

3.3.2 Severity

2. Providing appropriate technical advice to each student

Severity quantiﬁes developmental delay into separate levels. In
other words, the diﬀerence between the progress rate in the planand that in the progress report reveals developmental delay, but
not its severity. Figure4 shows how the developmental delay
and its severity is quantiﬁed. to ensure that the TA-Bot provides
more eﬀective technical advice. The severity is calculated bymultiplying the diﬀerence between the progress rates in the planand progress report by the number of elapsed days. Generally,

TA-Bot provides technical advice to students experiencing developmental delay. For this purpose, TA-Bot needs to be awareof the diﬀerence between the planned and actual progress. However, TA-Bot cannot measure the severity of the developmental
delay from this diﬀerence. Therefore, it can neither provide
appropriate technical advice nor eliminate developmental delay.

128

International Workshop on Informatics ( IWIN 2022 )Figure 4: Quantifying the severity of developmental delay
the severity is higher closer to the development completion date.
In this context, Tnp is the planned progress rate. Tna is the
progress rate in the progress report. E is the number of days
elapsed. D is total number of days. The formula to calculate
severity is given by (1).
T np − T na E
×
T np
D

Figure 5: Static Advice(1)

3.3.3 Advice
As explained earlier, two types of technical advice, i.e., Staticand Dynamic Advice, exist based on the source from which the
TA-Bot acquires technical advice.
Static Advice is the technical advice acquired from the database.
Figure5 shows how static technical advice is selected. TA-Bot
selects the technical advice depending on the calculated severityof developmental delay. For this purpose, we deﬁne thresholds
to help the TA-Bot in deciding what type of technical advice toprovide by splitting the severity into three levels, i.e., Level1,
Level2 and Level3. By adjusting the threshold for each student, the optimal timing for providing technical advice can be
adjusted. Table1 shows the three types of technical advice, i.e.,
providing keywords related to software development, directing
students to a website that contains a relevant sample of software,and providing a programming syntax. For Level1 severity, TABot provides a keyword that is prepared from its experience ofthe class. For Level2 severity, TA-Bot directs the student to a
website that is ﬁnds by scraping based on the keyword. Finally,
for Level3 severity, TA-Bot provides the programming syntax tofacilitate comprehension and encourages the students to develop
the software further. In addition, the database of the TA-Bot is
continually updated by the TA, which ensures that the students
receive optimal technical advice.
Table 1: Technical Advice
Severity Level
1
2
3

Dynamic Advice is the technical advice acquired from websites that contain information on software development. TA-Bot
analyzes student comments and identiﬁes keywords related tothe speciﬁc problem they are facing in the process of softwaredevelopment. Figure6 shows how to dynamic technical advice
is selected. TA-Bot divides each word in the comments usingmorphological analysis, identiﬁes keywords related to softwaredevelopment, and ﬁnds the relevant website by scraping basedon the keyword. Finally, it stores the new technical advice inthe database and makes it available to the class the next time advice is required. Thus, problems faced by students in softwaredevelopment can be ﬂexibly solved using Dynamic Advice.

Figure 6: Dynamic Advice
However, the eﬀectiveness of the proposed TA-Bot cannot
be easily evaluated because of two reasons: the scale of therequired experiment is large and ensuring student participation
is diﬃcult. Thus, data can be collected only from a few students,
which is insuﬃcient for evaluation. Moreover, the intended
development situation is not available for evaluation. Therefore,
the experiment is not reproducible.

4 Simulation

Technical Advice
KeywordWeb Information
SyntaxTo optimize two System Thresholds for each student; Diciding
Technical Advice Threshold determing providing the technicaladvice timing and Selecting Technical Advice deciding which
technical advice providing to students, we need to repeat experiment. Accordingly, we ask the students participating in the ex-

129International Workshop on Informatics ( IWIN 2022 )considered in this study is to implement a telephone switchboard using code, within a period of 6 months with the students
submitting weekly progress reports. During the developmental
period, TA-Bot uses the database to provide technical advice
based on the progress report. Moreover, TA-Bot acquires experience in providing technical advice by learning from the advice
in the database that was previously provided by the TA. Finally,it stores new information from the experiment to use as Static
Advice. We analyze the result to output technical advice history.

periment to solve programming problems, and TA-Bot provides
technical advice to each student depending on their developmental situation. Using this process, we repeat the experiment and,
thereby, adjust the timing and the eﬀectiveness of the technicaladvice.
However, because the real experiment is not reproducible, theevaluation is performed using simulation, for which we intentionally create developmental situations for each student. Consequently, we can repeat the evaluation for cases in which theprovided technical advice is ineﬀective, determine the cause,and, thereby, improve the system.
Figure7 shows the simulation model. For the simulation,
we need to deﬁne the input model and output log. We use
the progress report of the students as the input model. TA-Bot
calculates the severity depending on each of the progress reports
and provides technical advice accordingly. We record the dayon which the technical advice is provided, the technical advice
provided, and the severity as the output model.

4.2 Classifying students’ programing skills
In software development, the status and timing of developmental delay and depend on the level of programming skills of
the students. In other words, the cause of the delay is diﬀerent foreach student. Thus, diﬀerent technical advice must be provided
to diﬀerent students. For this purpose, we classify programing
skills into three levels.
1. Beginner
2. Intermediate
3. Advanced
The beginner level indicates that the student has ﬁnished basic
programming training. However, they cannot write code without
the aid of textbooks and web information.
The intermediate level indicates that the student can develop
basic code without the aid of textbooks or web information.
However, they need to collect information to perform applied
programming.
The advanced level indicates that the student can provide
technical advice to the Beginner and Intermediate level programmers. In addition, they can develop most software independently. Based on these three deﬁnitions, we can reproduce
developmental delay situations, for which TA-Bot can provides
technical advice depending on the skill level of the student.

Figure 7: Simulation Model
Moreover, we need to deﬁne the developmental situation ofeach student when creating their progress reports. For this purpose, we deﬁne two student parameters, i.e., the level of programming skills of the students based on the diﬀerence in theirprogramming proﬁciency and the development phase by dividing the development period into several stages.
We create the developmental situation by combining the twoparameters and obtain the output log through simulation basedon the developmental situation. Subsequently, we can conﬁrm
whether the technical advice is eﬀective and how the severity
varied by analyzing the output log and improve the system basedon the result. In contrast to a real-world experiment, we canrepeat the simulation to deﬁne new parameters and reconsider
existing ones. Therefore, we can re-evaluate the system for thesame situation after adjusting the timing and type of the technicaladvice.

4.3 Classifying development period
Furthermore, developmental progress is diﬀerent for each student. In addition, the developmental situation and severity ofthe developmental delay depend on when the delay occurs. To
reﬂect this, we divide the development period into three periods.
1. Phase1
2. Phase2
3. Phase3

4.1 Simulation Method
The evaluation method assumes that the developmental progressof every student is diﬀerent. The objective of the class that is

130

Figure8 shows how the development period is divided. The
phase Early denotes the period during which students create
their plan and understand the program speciﬁcations. The
Middle phase denotes the period during which the students
write code. The Late phase denotes the period of the test phaseInternational Workshop on Informatics ( IWIN 2022 )and extensions, if required. In addition, we divide degree of thedelay into two types, viz., large and small. When the degree ofthe delay is large, the delay is considered to be extremely severe.
However, when the degree of the delay is small, we don’t judge
that the delay is not severe. By dividing the development period
in the aforementioned manner, we can reproduce eight diﬀerent
patterns and, consequently, several development situations. In
addition, we can evaluate eﬀectiveness of the advice provided ineach period.

Figure 9: Experiment

4.6 Analyzing the evaluation result
We analyze the result of the evaluation using the severity
graph, comment and output log.In other words, we analyze thetiming providing technical advice is early or late by observing
the severity, we adjust threshold deciding technicaln advice. The
simulation outputs the technical advice, the day on which it wasprovided, and the severity of delay for each student. If the timing
is early, we raise the threshold and make the timing slowly. If
the timing is late, we lower the threshold and make the timing
early. We orgnize two points how to judge the timing is early or
slow.

Figure 8: Classifying development period

4.4 Reproduction of development

1. The advice that student hasn’t faced on task yet.

A total of 24 cases of developmental situation can be reproduced by multiplying the three levels of programing skills withthe eight types of development period. We create the progress
reports based on these cases and evaluate the system using simulation. Therefore, we can reproduce typical cases.

2. The advice that student has already solved.
We observe the severity after providing technical advice tojudge these points. If the severity decrease after providing technical advice, we conﬁrm the timing is appropriate. If the severity
increase after providing technical advice, we conﬁrm the timing
is not appropriate. For conﬁrming that, we analyze a commentin providing technical advice. The comment is written about
students facing developmental tasks, so we are able to adjust
the timing TA-Bot providing technical advice by investigating
students’ developmental situation based on the commnet. Therefore, we adjust the threshold deciding technical advie and technical advice by analyzing students facing developmental tasks.
Figure10 shows a sample of output log. Based on the timingof the technical advice, we observed that the severity declined
after the TA-bot provided the students with the URL of the website containing written development information. However, in
t2, the severity rose after TA-Bot provided students with the
keyword "REGEXP". In t3, the severity rose even further after
TA-Bot provided students with the keyword "extern". In t4, theseverity declined after TA-Bot provided the students with theprogramming syntax. Therefore, we can conﬁrm that Student
A found the syntax to be eﬀective advice. Therefore, by analyzing the result logﬁle of the simulated experiment, we canconﬁrm whether the technical advice provided for each student,
depending on their development situation, was eﬀective.

4.5 Experiment
Figure9 shows a sample of the simulation. We deﬁne the parameters for each student, and included three students, studentA,studentB and studentC in each sample. We set the programing
skill of studentA as Beginner, studentB as Intermediate, andstudentC as Advanced. In the actual process of software development, each student has a diﬀerent plan. Moreover, thetechnical advice provided by TA-Bot is not necessarily eﬀective
for all the students. Thus, we also set the eﬀectiveness of theadvice for each student. In the evaluation, the advice was considered as eﬀective for A and B and ineﬀective for C. Furthermore,
TA-Bot calculated the severity of delay for each student basedon the progress report created based on the aforementioned parameters. Using the severity graph, we conﬁrmed that the valuechange is diﬀerent for each student. In other words, we were able
to identify diﬀerences in the development situations by deﬁning
the parameters. Therefore, we can adjust the thresholds deciding
the type of technical advice.

131

International Workshop on Informatics ( IWIN 2022 )[4] T. Zimmermann, J. Coolen, J. Gross, P. -M. Pedrot and G.
Gilbert, "Advantages of maintaining a multi-task projectspeciﬁc bot: an experience report," in IEEE Software, doi:
10.1109/MS.2022.3179773.
[5] I. Beschastnikh, M. F. Lungu and Y. Zhuang, "Accelerating Software Engineering Research Adoption with Analysis Bots," 2017 IEEE/ACM 39th International Conference
on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER), 2017, pp. 35-38, doi:
10.1109/ICSE-NIER.2017.17.
[6] S. Saiki et al., "A Study of Practical Education Programon AI, Big Data, and Cloud Computing through Development of Automatic Ordering System," 2018 IEEE International Conference on Big Data, Cloud Computing,
Data Science & Engineering (BCD), 2018, pp. 31-36, doi:
10.1109/BCD2018.2018.00013.

Figure 10: Analyzing Logﬁle5 Conclusion
In this study, we proposed a progress management method
using TA-Bot for a software development class. To monitoring
student progress, TA-Bot quantiﬁed the severity of softwaredevelopmental delay and decided the timing of technical advice
based on it. Consequently, we were able to change the thresholds
deciding the type of technical advice.
For the experiment, we intentionally created students development situations. We deﬁned two parameters reﬂecting development situations, viz., programming skill levels and softwaredevelopment period. These two parameters were employed tofaithfully reproduce development situations. The total numberof cases that we could reproduce was 24. In the process ofthe simulated experiment, we were enabled to adjust the timing
and eﬀectiveness of the technical advice. This adjustment was
enabled by the repeatability of the simulation.
In this study, the system was evaluated using simulations.
However, the advice data is insuﬃcient for properly evaluating
the eﬀectiveness of the technical advice. Thus, new technicaladvice must be added. Furthermore, we intentionally createddevelopment situations and repeated the evaluation. Thus, we
could optimize the timing of the technical advice provided to individual students. However, actual development situations could
be more complex. Consequently, the proposed method may not
be able to eﬀectively optimize the timing of the technical advice. Therefore, in future work, we will evaluate the system not
only using simulation, but also using real development situations
with the help of student participation. For this purpose, we will
extend the system and modify it.

REFERENCES[1] Miyuki Yamamoto, Koichi Fukuoka, Ryozo Kiyohara,
Yoshiaki Terashima. "Progress Management Method forSoftware Development Project-based Learning Using Automated Teaching Assistants" 2019 Twelfth International
Conference on Mobile Computing and Ubiquitous Network(ICMU). IEEE, pp. 52–55 (2019)[2] https://www.enpit.jp/index.html[3] Slack ,https://slack.com/intl/ja-jp/132International Workshop on Informatics ( IWIN 2022 )An Experiment of an Office Worker Passing by
the Surrogate Robot of a Remote Worker Using VR Video
Kosuke Sasaki† , Zijie Yuan† and Tomoo Inoue‡†

Graduate School of Library, Information and Media Studies, University of Tsukuba, Japan‡
Faculty of Library, Information and Media Science, University of Tsukuba, Japan{ksasaki, inoue}@slis.tsukuba.ac.jp
results suggest that if the workers can see their partner throughthe HMD, they move to avoid collision predicting the directionof the partner by looking at their upper body.

Abstract - This study focuses on a situation in which a local
worker and a surrogate telepresence robot of a remote worker
pass by in an office corridor or passageway. A method of giving the remote worker’s image onto the telepresence robot using
VR technology has been proposed. In this paper, we examined
if the method was useful for workers through their subjective
evaluation. The result of a questionnaire survey and an interview indicated that the workers using this system could predict
where the remote worker was moving to by looking at his/her
upper body and could avoid collision.

2 Related Works
Passing by a human and a robot in office environment, whichis the situation this study focuses on, can be regarded as remotecooperative work between a local worker and a remote worker.
We first describe remote cooperative work in environment using
fixed displays.

Keywords: Cooperative work, Positional relationship, Nonverbal cue, Telepresence robot

2.1 Remote Cooperative Work Using Fixed
Displays

1 Introduction

Unlike face-to-face environments, visual nonverbal information such as gaze information or body movement may lack inremote environment. Since nonverbal information is importantfor smooth communication, studies have been conducted to cooperate between remote locations while communicating visual
nonverbal information to each other.
Making workers feel as if they are in the same space eventhough they are in remote environment is one method of supporting remote cooperative work. For example, some systems
use a common workspace or background for feeling that being in the same space[2], [3] or a system that overlays the remote partner on a landscape behind the display[4]. However,the use of such fixed displays places significant restrictions on
the worker’s physical movement, making them unsuitable forcooperative work that involves physical movement. Therefore,
some systems use AR or VR technologies to remove the restrictions.

Telework has increased in recent years, and there are more
and more situations in which robots and people work cooperatively. In particular, this study focuses on work environment
using mobile telepresence robots in which a remote worker operates the robot and works with a local worker. When a telepresence robot and a human are in the same space, it is important
for them to be aware of each other’s position. Considering a situation where a worker and a robot pass each other in an officecorridor or passageway, it is necessary to know how the partner
will move not to bump into each other.
Inoue and Yuan have proposed a method to give a human image to a telepresence robot using VR technology to address thisproblem[1]. However, the experiment environment was not so
realistic in that the initial distance between a human and a robot
was too close. Also, the paper only reported an initial evaluation and did not focus on the interaction between the worker
and the robot. In this paper, we evaluate this method based on
subjective evaluations obtained from participants.
In the experiment conducted in this study, workspaces at two
remote locations were prepared. In each workspace, a mobile robot equipped with an RGB camera was placed. Eachrobot moves synchronizing the position of its respective remoteworker. Each camera attached to the robot gave the partner’s image to an HMD which respective workers wore. In this environment, workers were paired up and performed a passing-by taskthat simulated passing each other in a company corridor. The

2.2 Remote Cooperative Work Using AR or VR
Technologies
Systems using AR or VR technologies with HMDs have also
been proposed to solve the problem of limited body movement.
For example, Fuchs et al. and You et al. have proposed systems that support remote cooperative work by immersing theuser in the same virtual space with a remote partner[5], [6].

133International Workshop on Informatics ( IWIN 2022 )(a)Using avatars also seems to work for smooth remote collaborative work. In fact, a number of studies have reported theeffectiveness[7]–[10]. AR and VR technologies can reduce therestrictions on worker movement, however, the worker does nothave a physical body and cannot interact with the real space invirtual spaces where many studies use in their system. Furthermore, using avatars can miss non-verbal information of workers
before they are made into avatars.

2.3 Remote Cooperative Work Using
Telepresence Robots(b)
Robot B

Worker B

3.0m

3.0m

Worker A

Robot A

Omnidirectional
cameraController to
get the location
Drive unit of thisrobot

Figure 1: (a) Environment of the experiment / (b) Appearanceof the telepresence robot

Telepresence robots have the possibility to solve the above issues, that is, reduce the limitation of body movement and enable
workers to interact with real space or objects without missing
nonverbal information. MeBot[11] developed by Adalgeirsson
et al. and iRIS[12] developed by Kawanobe et al. can transmit
gaze information and head movement of a partner by moving
the robot’s head and the display attached to the head. Such technology may be effective for cooperative work in remote work.
For instance, in a remote meeting, these technologies made iteasier to identify the conversation partner and reduced time required to answer questions.
Another study has suggested a system that immerses the remote worker in the robot which can display the remote worker’s
face to local workers and perform the same arm movement of
the remote worker[13], [14]. Such systems lead remote workers
to collaborate with local workers.
A mobile robot can present remote workers’ position in space
to local workers, enabling real-space interaction and cooperative work between remote locations without restrictions on movement within the space. Third Eye[15] and Yang et al. studies[16] used a one-directional but immersive mobile robot for
remote cooperative work. However, the remote worker’s movement are limited to those that the robots can move, which may
miss some nonverbal information.
Based on these studies, we decided to use VR technology andgive the mobile telepresence robot the appearance of the partner
as is. This makes it possible to convey visual information to the
local worker.

Both workers wear HMDs, and they could see images fromthe robot’s camera placed in front of the partner. In this way,
an environment was created in which the local worker and thetelepresence robot that surrogates the remote worker pass byeach other.
TM
In this study, VIVE by HTC Corporation1 was used for the
HMD worn by the worker and for the controller and tracking
camera to acquire the coordinates of the robot and worker. The
R
iRobot⃝Create 2 by iRobot2 was also used for the driving partof the robot. A height-adjustable tripod for a camera was placedTMon the robot, and a VIVE controller was attached to acquire
the robot’s coordinates. Additionally, a RICHO’s3 360-degreeTMcamera, Theta V , was mounted at the top of the tripod to capture the partner’s appearance.

4 Experiment
4.1 Passing-by Task
In this study, we conducted an experiment of passing-by task
in which a telepresence robot and a remote worker passed byeach other (approved by the Ethics Review Committee of Faculty of Library, Information and Media Science, University ofTsukuba (Notification number: 20-15)). In this experiment, theworker and robot initially stood facing each other at a distanceof 3.0 meters as shown in Figure 1(a). On the experimenter’scue, they started walking, the worker aiming at the robot’s initial position and the robot aiming at the worker’s initial position.

3 Environment of the System

4.2 Experiment Conditions

This study used a system based on the previous study[1]. Figure 1(a) shows the environment of the system, which a worker
and a telepresence robot pass by each other. Two workspaces
were prepared and a pair of a worker and a telepresence robot
was arranged in each workspace. Telepresence robots equippedwith a 360-degree camera (as shown in Figure 1(b)) were placedin front of each worker in different workspaces. The robots
moved synchronizing the respective worker’s positions in each
workspace.

We compared the following two conditions; the bidirectional
condition and the unidirectional condition. The bidirectional
condition was that workers were immersed in robots in each
1 VIVE - VR Headsets, Games, and Metaverse Life, https://www.
vive.com/ (Visited on Dec 10, 2022)
2 Coding Robots, Learning Library & STEM Outreach | iRobot Education,
https://edu.irobot.com/ (Visited on Dec 10, 2022)
3 Ricoh Global | EMPOWERING DIGITAL WORKPLACES, https://www.ricoh.com/ (Visited on Dec 10, 2022)

134International Workshop on Informatics ( IWIN 2022 )View of worker A

Worker B↓

4.4 Procedure
The experiment was conducted in a within-participant design.
Each pair participate both of the conditions with counterbalanced. In each condition, participants practiced the passingby task twice to familiarize themselves with the task using thetelepresence robot. Also, the experimenter explained the task
until both participants understood.
After that, the real task is performed once. After the realtask, the sequence was repeated under different conditions. The
unidirectional condition was asymmetry as opposed to the bidirectional condition. Therefore, after completing the real task
once in the unidirectional condition, workers changed roles andperformed the practice and real tasks again. Therefore, the realtask has performed a total of six times (one time for each pair)
for all participants in the bidirectional condition and 12 times(two times for each pair) in the unidirectional condition.
For participants’ safety, one experimenter was assigned toeach workplace to prevent workers from colliding with the robot
and catching their feet on the cords connecting the robot and theprocessing PC during the task.
After the real task in each condition, a questionnaire survey
was conducted. We set up 10 questions for the questionnaire,asking not only about the robot’s usability but also about thepresence of the partner and the sense of being in the same roomwith the robot. Participants responded to the evaluation items on
a 7-point scale from ”strongly disagree” to ”strongly agree.” Wealso conducted an unstructured interview after the completionof the task in both conditions, and participants were asked torespond freely to questions about their impressions of the task
and the system.

Robot A↓↑
Robot B↑
Worker A

View of worker B

Figure 2: The bidirectional condition
View of worker A

Worker B↓↑
Robot B↑
Worker A

View of worker B

Figure 3: The unidirectional condition
other’s remote workspace. Workers wore HMDs at both remotelocations, and each HMD displays the video from the camera
mounted on the robot at the remote location as shown in Figure
2. On the other hand, the unidirectional condition was a condition in which only one worker was immersed in a robot in the
remote workspace. Only one worker was immersed in the remote robot and only took the role of the controller of the robot.
Another worker also wore an HMD, but it displayed the seethrough image in front of him/her as it is. Also, a robot in frontof the worker who was a controller of the remote robot was notplaced as shown in Figure 3).
In both conditions, workers were not allowed to talk to eachother in order to examine the visual information transfer between workers’ actions. Therefore, during the experiment, workers moved based only on visual information.

5 Result and Discussion
Ratings were treated as a score from 1 to 7, and a Wilcoxon
signed-rank test was conducted for each question item. The results are shown in Table1. In particular, the scores following
four questions for the bidirectional condition were significantly
higher than those for the unidirectional condition.: Q4 “I could
clearly know the motion of the partner.” (p < .01), Q7 “I foundthe movement and direction of the partner.” (p < .01), Q8 “I
could predict where the partner was moving to.” (p < .01), andQ9 “Partner’s movement looked realistic.” (p < .05).
In the interview, we obtained comments such as “I could predict the robot’s path when looking at a human (P3: Bidirectional
condition),” “I could clearly see its direction of movement whenlooking at a human (P5: Bidirectional condition),” “I could not
see the robot’s direction of movement (P2: Unidirectional condition),” “I could not see the robot’s direction of movement(P7: Unidirectional condition),” and “I was confused about the
robot’s direction of movement (P8: Unidirectional condition).”
These results of the questionnaire and the interview indicatethat workers predicted the robot’s direction of movement, i.e.

4.3 Participants
Six pairs consisting of 12 graduate students (all male) participated in the experiment. Of the 12 participants, five used an
HMD for the first time in this experiment. Only one of the six
pairs had never met each other before.

135International Workshop on Informatics ( IWIN 2022 )Table 1: The results of the questionnaire. “Bi” and “Uni” means
the median of rating in the “Bidirectional condition” or “Unidirectional Condition” respectively. p-values are the results ofWilcoxon signed-rank test.
(∗∗ p < .01, ∗ p < .05, † p < .10)
Q1

Q2
Q3

Q4
Q5
Q6

Q7
Q8
Q9
Q10

Items
After using the system, I
felt uncomfortable similar to
motion sickness
The video was clear andpretty.
I could feel like the remoteperson was in the same roomwith me.
I could clearly know the motion of the partner.
I felt like I passed the otherperson in the same room.
The field of view changed
naturally according to mymovement.
I found the movement anddirection of the partner.
I could predict where thepartner was moving to.
Partner’s movement looked
realistic.
The experience in the virtualenvironment was consistent
with experiences lived inreal life.

6 Conclusion
In this study, we focused on a situation in which a telepresence robot and a local worker pass by in an office corridor or
passageway. An VR system that displays video of a remoteworker at the position of a surrogate telepresence robot to the
local worker to avoid collision has been studied[1]. However,its evaluation was very preliminary and was not very realistic.
In this paper, we experimented with a passing-by task and conducted a questionnaire survey and an interview that asked about
workers’ feelings during the task. The results suggested that
presenting the partner’s upper body was used to predict the partner’s movement, and thanks to this, the worker can avoid collision with the surrogate telepresence robot.

Bi
5

Uni
5

p-value
.79

5

4.5

.08†

6.5

6

.13

6

5

.008∗∗

5.5

5.5

.69

The authors would like to thank Dr. Yasuhito Noguchi for hiscooperation in conducting the experiment.

5.5

5.5

1.00

REFERENCESACKNOWLEDGEMENT[1] T. Inoue and Z. Yuan, “Remote video figure achieves
smooth cooperative movement in a bidirectional telepresence robot environment,” in Collaboration and Technology, A. Rodrigues, B. Fonseca, and N. Preguiça, Eds.
Cham: Springer International Publishing, 2018, pp. 91–
104.
[2] J. C. Tang and S. Minneman, “Videowhiteboard: Video
shadows to support remote collaboration,” in Proceedingsof the SIGCHI Conference on Human Factors in
Computing Systems, ser. CHI 91. New York, NY,
USA: Association for Computing Machinery, 1991, pp.
315–322. [Online]. Available: https://doi.org/10.1145/
108844.108932[3] L. Handberg, C. Gullströ, J. Kort, and J. Nyström,“Spatial and social connectedness in web-based work
collaboration,” in Proceedings of the 19th ACM Conference on Computer Supported Cooperative Workand Social Computing Companion, ser. CSCW ’16
Companion. New York, NY, USA: Association for Computing Machinery, 2016, pp. 45–48. [Online]. Available:
https://doi.org/10.1145/2818052.2874321[4] T. Inoue, M. Nawahdah, and Y. Noguchi, “User’s communication behavior in a pseudo same-room videoconferencing system bhs,” International Journal of Informatics
Society (IJIS), vol. 6, pp. 39–47, 11 2014.
[5] H. Fuchs, A. State, and J. Bazin, “Immersive 3d telepresence,” Computer, vol. 47, no. 7, pp. 46–52, July 2014.
[6] B.-J. You, J. R. Kwon, S.-H. Nam, J.-J. Lee, K.K. Lee, and K. Yeom, “Coexistent space: Toward
seamless integration of real, virtual, and remote worlds
for 4d+ interpersonal interaction and collaboration,” in

∗∗

6

4.5

.003

6

4

.001∗∗

6

4.5

.02∗

5.5

5

.16

the partner’s direction of movement, when the partner’s image
was displayed on their HMDs.
The importance of partners’ image in collaborative work has
been proven in previous studies, too.
In a study of a collaborative bicycle repair task in face-to-face
and in remote found the visual cues in the partner’s image could
make the grounding in their conversation and the task[17].
Another study of a remote collaborative task of assembling a
toy was conducted in a specially designed space where the local worker and a table were surrounded by eight fixed displays
showing a remote worker. It found the local worker could predict the remote worker’s movement by seeing the upper body ofthe partner and could prepare for the next action[18].
In contrast to these studies on predefined collaborative workin a fixed place, our study indicates the usefulness of showing
the partner’s video to pass by the robot smoothly by predicting
its moving direction.

136

International Workshop on Informatics ( IWIN 2022 )SIGGRAPH Asia 2014 Autonomous Virtual Humans andSocial Robot for Telepresence, ser. SA ’14. New York,NY, USA: Association for Computing Machinery, 2014.
[Online]. Available: https://doi.org/10.1145/2668956.
2668957[7] G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “Mixed
reality collaboration through sharing a live panorama,”
in SIGGRAPH Asia 2017 Mobile Graphics & Interactive
Applications, ser. SA ’17. New York, NY, USA:
Association for Computing Machinery, 2017. [Online].
Available: https://doi.org/10.1145/3132787.3139203[8] T. Piumsomboon, A. Day, B. Ens, Y. Lee, G. Lee,and M. Billinghurst, “Exploring enhancements forremote mixed reality collaboration,” in SIGGRAPH
Asia 2017 Mobile Graphics & Interactive Applications,ser. SA ’17. New York, NY, USA: Association
for Computing Machinery, 2017. [Online]. Available:
https://doi.org/10.1145/3132787.3139200[9] T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens,
R. W. Lindeman, B. H. Thomas, and M. Billinghurst,“Mini-me:
An adaptive avatar for mixed reality
remote collaboration,” in Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems,
ser. CHI ’18. New York, NY, USA: Association
for Computing Machinery, 2018. [Online]. Available:
https://doi.org/10.1145/3173574.3173620[10] T. Teo, A. F. Hayati, G. A. Lee, M. Billinghurst,
and M. Adcock, “A technique for mixed reality remotecollaboration using 360 panoramas in 3d reconstructedscenes,” in 25th ACM Symposium on Virtual Reality
Software and Technology, ser. VRST ’19. New York,NY, USA: Association for Computing Machinery, 2019.
[Online]. Available: https://doi.org/10.1145/3359996.
3364238[11] S. O. Adalgeirsson and C. Breazeal, “Mebot: A robotic
platform for socially embodied telepresence,” in 2010 5th
ACM/IEEE International Conference on Human-Robot
Interaction (HRI), March 2010, pp. 15–22.
[12] H. Kawanobe, Y. Aosaki, H. Kuzuoka, and Y. Suzuki,“iris: A remote surrogate for mutual reference,” in2013 8th ACM/IEEE International Conference on HumanRobot Interaction (HRI), March 2013, pp. 403–403.
[13] S. Tachi, “Telexistence: Enabling humans to be virtually
ubiquitous,” IEEE Computer Graphics and Applications,vol. 36, no. 1, pp. 8–14, Jan 2016.
[14] S. Tachi, K. Watanabe, K. Takeshita, K. Minamizawa,T. Yoshida, and K. Sato, “Mutual telexistence surrogate
system: Telesar4 - telexistence in real environments using
autostereoscopic immersive display -,” in 2011 IEEE/RSJ
International Conference on Intelligent Robots and Systems, Sep. 2011, pp. 157–162.
[15] A. P. Tarun, N. M. Baig, J. S.-K. Chang, R. Tanvir, S. Shi-

hipar, and A. Mazalek, “Third eye: Exploring the affordances of third-person view in telepresence robots,” in Social Robotics, M. A. Salichs, S. S. Ge, E. I. Barakova, J.-J.
Cabibihan, A. R. Wagner, Á. Castro-González, and H. He,
Eds. Cham: Springer International Publishing, 2019, pp.
707–716.
[16] L. Yang, B. Jones, C. Neustaedter, and S. Singhal,“Shopping over distance through a telepresence robot,”
Proc. ACM Hum.-Comput. Interact., vol. 2, no. CSCW,
Nov. 2018. [Online]. Available: https://doi.org/10.1145/
3274460[17] S. R. Fussell, R. E. Kraut, and J. Siegel, “Coordinationof communication: Effects of shared visual context on
collaborative work,” in Proceedings of the 2000 ACM
Conference on Computer Supported Cooperative Work,ser. CSCW 00. New York, NY, USA: Association
for Computing Machinery, 2000, pp. 21–30. [Online].
Available: https://doi.org/10.1145/358916.358947[18] N. Yamashita, H. Kuzuoka, K. Hirata, S. Aoyagi,
Y. Shirai, K. Kaji, and Y. Harada, “Effects of showing
user’s upper body in video-mediated collaboration,” IPSJ
Journal, vol. 51, no. 4, pp. 1152–1162, apr 2010. [Online].
Available: https://ci.nii.ac.jp/naid/110007970719/en/

137

International Workshop on Informatics ( IWIN 2022 )141

International Workshop on Informatics ( IWIN 2022 )142

International Workshop on Informatics ( IWIN 2022 )143

International Workshop on Informatics ( IWIN 2022 )144

International Workshop on Informatics ( IWIN 2022 )145

International Workshop on Informatics ( IWIN 2022 )146

International Workshop on Informatics ( IWIN 2022 )147

International Workshop on Informatics ( IWIN 2022 )148

International Workshop on Informatics ( IWIN 2022 )149

International Workshop on Informatics ( IWIN 2022 )150

International Workshop on Informatics ( IWIN 2022 )151

International Workshop on Informatics ( IWIN 2022 )152

International Workshop on Informatics ( IWIN 2022 )153

International Workshop on Informatics ( IWIN 2022 )154

International Workshop on Informatics ( IWIN 2022 )155

International Workshop on Informatics ( IWIN 2022 )156

International Workshop on Informatics ( IWIN 2022 )157

International Workshop on Informatics ( IWIN 2022 )158

International Workshop on Informatics ( IWIN 2022 )159

International Workshop on Informatics ( IWIN 2022 )160

International Workshop on Informatics ( IWIN 2022 )161

International Workshop on Informatics ( IWIN 2022 )162

International Workshop on Informatics ( IWIN 2022 )'HYHORSPHQWRIDQ$YDWDU5HPRWH&RPPXQLFDWLRQ6\VWHPIRU$/63DWLHQWV 
8VLQJ9LGHR&RQIHUHQFLQJ7RROV
Nagisa Kokubu *, Riku Sugimoto*, Shouma Hamada*,
Takayo Namba **, and Keiichi Abe**
Kanagawa Institute of Technology, Atsugi City, Kanagawa, Japan**
Kawasaki City College of Nursing, Kawasaki City, Kanagawa, Japan
abe@he.kanagawa-it.ac.jp
Abstract -Amyotrophic lateral sclerosis (ALS) patients withsevere symptoms have difficulty in walking, going out, andtalking with people. The authors believe that there is a needfor a system that allows severe ALS patients to maintaincontact with society and communicate with many people
while staying at home. A representative previous study forsolving this problem is an avatar robot. This avatar robot isplaced in a place where the patient cannot go, and thepatient can communicate through the avatar robot remotely
by gestures. However, the avatar robot needs to be set up at
the place where there is the person whom the patient isgoing to talk with, and the avatar robot needs to be removed
after the conversation is over. Therefore, we propose asystem that enables ALS patients with severe symptoms
who have difficulty in going out to talk with many people
via an avatar character easily and remotely forhimself/herself. We have developed and evaluated a
prototype of our proposed system.

effectiveness of the proposed system, we conducted aquestionnaire evaluation with an ALS patient.

When ALS patients become severely ill, it becomesdifficult for them to move their bodies and to speak.
Conversation aids are an existing technology to help
severely ill ALS patients communicate their intentions. The
first representative product of portable conversational aids isa keyboard input type conversational aid [3]. This keyboardinput type conversation aid uses the keyboard to create
sentences, which are then spoken by a voice synthesizer. It
also has functions for registering frequently used words in
categories and communicating using images. However, thiskeyboard input type conversation aid is difficult to use forseverely ill ALS patients who have quadriplegia and havedifficulty moving their bodies because the device requires
the user to input data by hand. On the other hand, there is agaze-input conversation aid [4]. Gaze input typeconversation aids are devices that use a gaze input instead ofa keyboard to create sentences and have them spoken byspeech synthesis. Gaze-input conversation aids use a special
gaze-input interface which is expensive and increases thesystem introduction cost. Common to these two products is
the issue of not being able to converse with many people atonce because they are face-to-face conversational types. As
a previous study to solve these problems, there is an avatar
robot developed by Oly Research Institute. This robot is
installed in a place where ALS patients cannot go, and it cantalk with a person whom the patient wants to talk withremotely by gestures. However, the robot needs to be set up
at the place where there is the person whom the ALS patientwants to talk to, and the robot needs to be removed after theconversation is over, which requires the help of others
besides the ALS patient. Another avatar technology otherthan robots is called Metaverse [5]. The avatars used in the
metaverse are computer graphics, and the user can choose
various characters of his/her choice. With this technology,
people who have difficulty communicating in the real world
can freely enjoy conversations and other activities in a
virtual space through the avatar of their choice. In thismetaverse, there is no need to set up an avatar like an avatar
robot in advance at the place where you want to have aconversation, and no need to remove the avatar after theconversation is over. However, the interface used in the
metaverse is currently a technology for able-bodied people,
and it is difficult for people with physical disabilities, suchas ALS patients, to use it because they cannot speak.

.H\ZRUGV: Remote Communication, ALS, online conferencing tool.

5(/$5(':25.6,1752'8&7,21

Amyotrophic lateral sclerosis (ALS) is a disease in which
the nerves that control voluntary movements (motor
neurons) are damaged, resulting in the loss of muscles
throughout the body and those necessary for conversation.
In severe cases, the patient becomes bedridden and isolated,with no contact with other people, and it is difficult for thepatient to move his/her body, although he/she is conscious.
A previous study aiming to solve this problem is an avatar
robot [1]. This avatar robot is placed in a place where ALS
patients cannot go, and it can communicate with a person
remotely by gestures. However, it is necessary to set up theavatar robot and remove it after the conversation. Therefore,
the system requires help from other people besides the ALS
patient.
In this study, we propose a system that enables ALS
patients to communicate with many people in an enjoyableway, so that they can keep in touch with the society andcommunicate with various people. We have developed asystem that enables ALS patients to talk with many people
by their own will by using an eye control technology and an
online conferencing tool[2]. Furthermore, the system allows
ALS patients to choose their favorite avatar character during
the conversation. We developed and evaluated a prototypeof the proposed system. In order to evaluate the165International Workshop on Informatics ( IWIN 2022 )Therefore, this study proposes a system that solves theseissues.

352326('6<67(0Chapter 3 provides a detailed overview of the avatar
remote communication system using video conferencing
tools proposed in this paper.

 2XWOLQHRIWKH3URSRVHG6\VWHPThe proposed system is intended for ALS patients withsevere symptoms who have difficulty in going out andenables them to easily and enjoyably talk with many people
remotely without requiring help from others. The proposedsystem uses an online conferencing tool [2], as a remotecommunication tool for the ALS patient and the other party
to have a conversation. The first reason for using an online
conferencing tool is that it is widely used around the world
as a video conferencing tool and is compatible with various
platforms. The second reason is that ALS patients can easily
communicate with other patients at home or in the hospital,
regardless of their location, as long as they own a terminalthat can use the online conferencing tool and have access to
a network. Next, the proposed system uses an eye tracking
interface [6] for the ALS patient's side. We chose the Eye
Tracking interface because it is less physically demanding
than the keyboard operation, since severely ill ALS patients
have significantly reduced physical capabilities. The text
input from the Eye Tracking interface was used as the textto-speech interface. The system also allows the user to add
and select a character for the voice dialogue agent that willbe the avatar of the ALS patient by utilizing the function ofadding and selecting voice dialogue agents in a previous
study[7]. The ALS patient's favorite character can be added
to the system by adding a GIF (Graphics Interchange
Format) file of an animated character using FaceRig[8], etc.
The system can also output different voices for eachcharacter by adding a voice model for each character andconfiguring the voice, speed of speech, and endings. The
system allows the user to remotely communicate with theother party via the voice dialogue agent by transmitting thedisplay screen of the voice dialogue agent selected by theuser on the shared screen of the online conferencing tool.
We call this proposed system Avatar Remote
Communication System for ALS Patients (ARCS-ALS).



Figure 2: ARCS-ALS Prototype System Overview.



352727<3('(9(/230(17Chapter 4 details the development of the ARCS-ALS
prototype system proposed in Chapter 3.

 2XWOLQHRISURWRW\SHV\VWHP
Figure 2 shows an overview of the prototype system. In
this prototype, distributed processing is performed by aWindows PC and a Raspberry PI3B+ (RPI). The Windows
PC is in charge of the main processing and the RPI is incharge of the sub-processing. The main processing on the
Windows PC includes the input processing of the Eye
Tracking interface (EyeTracke4C,Tobii)[6] used by the
ALS patient to operate the ARCS-ALS, the switchingprocessing of the video for the voice dialogue agent thatspeaks on behalf of the ALS patient, and the processing of
the video for the waiting and conversation. The subprocessing on the RPI includes switching the video between
the standby and conversational mode, and remotecommunication with the conversational partner via theonline conferencing tool. The RPI also outputs speechsynthesized content input from the Windows PC in
accordance with the voice of the voice dialogue agent andinstructs the Windows PC when to switch the live-action
video between the standby and conversational mode. The
speech synthesis output of the sub-processing (RPI) is input
to the microphone on the Windows PC side of the mainprocessing via the speaker output of the RPI. This results inthe synthesized voice being output to the other side via theonline conferencing tool. Open J Talk[9] was used for the
speech synthesis of the voice dialogue agent.

 $SSOLFDWLRQVFUHHQIn the prototype of the proposed system, ARCS-ALS, wedeveloped a voice dialogue application to be used by ALS
patients when conversing with each other. Figure 3 is a
general view of the prototype system developed for thisproject, and Figure 4 shows the screen of the application
developed this time. As shown in Figure 4, the large
character in the middle of the screen is the selectedcharacter. In other words, it will be a voice dialogue agent
that will carry out conversations on behalf of the ALS

Figure 1: ARCS-ALS overview.

166

International Workshop on Informatics ( IWIN 2022 )patient. When the patient wants to change the selected agent
to suit his/her preference, he/she can click on the character
display on the right side of the screen to change thecharacter automatically.
In this prototype, we prepared four characters: the user's
own image, a child character, and animal characters of a
dog and a cat. Two methods were prepared for the input of
the conversation. In the first method, frequently usedconversations such as replies, and greetings are registered inadvance and can be spoken at the touch of a button. The
second method is to use the keyboard, which is one of thefunctions of the eye control input interface, to input text. In
this method, the ALS patient inputs what he/she wants to
say using the keyboard and presses the "send" button to
speak. Although it takes some time to input the content of
the conversation, any kind of conversation can be sent.

Figure 4: Application screen for ARCS-ALS.

 (\HWUDFNLQJLQWHUIDFH
ARCS-ALS uses the Eye Tracking interface[6] as amethod for ALS patients to talk. The reason for using thisinterface is that even ALS patients who have limb
disabilities and have difficulty moving their bodies can use
the Eye Tracking interface for eye control if they do nothave eye problems. In our proposed system, we used an offthe-shelf eye tracking interface and the eye tracking
function[3] that comes standard in Windows 10. To use thesystem, the Eye Tracking interface is connected to a
Windows PC via a USB interface, and the main unit of the
Eye Tracking interface is attached to the bottom of themonitor. After installing the driver, adjusting the position of
the Eye Tracker 4C, and performing simple setups such as avisibility test, the system is able to detect the user's line ofsight on the Windows PC screen. Next, by activating theeye control function that comes standard with Windows 10
and later, the user can click, use the mouse, and performkeyboard input, so even the physically challenged canoperate the PC using only their line of sight. The Eye
Tracking interface should be installed at an appropriate
distance from the user, which is approximately 90 cm. If the
distance is too close or too far, the camera cannot becalibrated accurately and may not operate properly. Figure
5 shows the eye tracking interface used in the prototype andthe eye control function. The eye tracking interface attachedto the lower part of the monitor (circled in red) recognizes
the user's line of sight, and the eye control function (circled
in green) is used to control the operation. By looking at theFigure 5: Overview of Eye Tracking interface.
desired function for a certain period of time, the user can
instruct the eye tracking function to display icons to selectthe desired location for clicking and a keyboard for
inputting text, as shown in Figure 5. At first, it may be alittle difficult to operate the function you want to use, but as
you gradually become accustomed to it, you will be able tooperate it at will.

 9RLFHGLDORJXHDJHQWIn the ARCS-ALS system, a voice dialogue agent is usedwhich talks to the other party instead of the patient. The
voice dialogue agent was created by applying our previous
studies. In this prototype, we prepared four characters: acharacter created from the ALS patient's own image, a childcharacter, and animal characters (a dog and a cat). This
allows the user to choose one of the characters according tohis or her preference and environment, such as those whowant to show their healthy self when talking to otherpersons or those who want to appear to be a substitutecartoon character instead of their own image. The voice
dialogue agent can be created in two main ways: as a CG
character or as a live-action character. Figure 6 shows asummary of each method. First, CG characters were createdusing FaceRig[8] software, which allows the user to become
any character using a webcam, and two GIF images werecreated: one in the standby mode (as if the user is listening
to a conversation) and one in the conversational mode (as if
the user is talking). By displaying the two GIF images inaccordance with the speech state of the voice dialogue agent
using text-to-speech output and switching the character, themouth movements are synchronized with the video as if thecharacter is actually speaking. The live-action character isan application of our previous work on creating CGFigure 3: Prototype system developed for this project.

167

International Workshop on Informatics ( IWIN 2022 )characters. To create a live-action character, two live-action
videos were actually shot using video equipment, one in thestandby mode and the other in the conversational mode. As
in the previous study, a live-action character is easily
created by switching between the two GIF images inaccordance with the speech state. In this case, it is necessary
to shoot a short movie of 3 to 5 seconds and change themovie file to a GIF file. This makes it possible to repeat andplay back the short video, which allows for longer text
conversations. This method has the advantage that, once theuser becomes accustomed to it, a live-action character canbe created in about 10 minutes, making it possible to createa character of the person to be reproduced at a low cost. In
addition, it is considered to be better to create the image of
the ALS patient before the disease becomes severe, because
it is considered to be practically difficult for the ALS patient
to film himself after the disease has become severe.

Figure 6 : How to Create Voice Dialogue Agents.

 9RLFHV\QWKHVLVRXWSXWThe voice of the voice dialogue agent, which is the avatarof the ALS patient, is synthesized by Open J Talk [9], andits output is shown in Figure 7. In this system, the ALS
patient uses the eye control input interface to input thecontent of the conversation in text using the application
developed in this study. The input conversation content is
output as text-to-speech by Open J Talk [7], and the voice
dialogue agent speaks the content. Acoustic models areprepared for each of the voice dialogue agents selected by
the user, and the voice, speech style, and endings arechanged for each character to speak. The dog CG character
would end with " bow-wow" and the cat CG character
would say " meow". On the other hand, when using a liveaction character created from the ALS patient's own video,
we prepared an acoustic model that closely resembled thepatient's own voice, and then manually adjusted theparameters using Open J Talk to make the voice soundsimilar to the patient's own voice. Specifically, the voice
quality, which can be changed to feminine or masculine bychanging related parameter values, the pitch shift, whichchanges the tone of the voice, and the speech speed, whichchanges the speed of speech, were adjusted to match thepatient's voice. The values of these voice parameters wereused because they can be fine-tuned, which actually takes
time but produces highly reproducible voice. Other speechsynthesis technologies include those that utilize AI
technology to learn voices. Coestation [10] and LYREBIRD
[11] are representative of these technologies. With thesetechnologies, it is easy to produce a voice synthesizer that isclose to the user's own voice by reading a few examples ofsentences and having the AI learn those sentences. However,since the example sentences need to be learned repeatedly, it
is not possible to train the AI for ALS patients who havedifficulty in speech. Therefore, this system uses Open J Talk,
which allows the user to manually adjust the parameters of
the acoustic model based on the original voice.

(9$/8$7,21

In this chapter 5, a questionnaire evaluation was
conducted to evaluate the prototype of the proposed system.

168

Figure 7: Speech synthesis output mechanism.
This time, the evaluation was conducted the questionnaireevaluation for ALS patients. From the questionnaireevaluation, we investigated the effectiveness of the proposed
ARCS-ALS system, the functions required for the system,and points to be improved, and we describe the details.

 0HWKRG RI FRQGXFWLQJ TXHVWLRQQDLUHHYDOXDWLRQTo ask for an evaluation of the proposed system, we
contacted one ALS patient through an acquaintance who has
a close relationship with ALS patients. Because of theongoing COVID-19 situation and the difficulty of face-toface evaluation, we asked the patient to watch a video (about
9 minutes) summarizing the use of the prototype of the
proposed system and then to answer the questionnaire.

 (YDOXDWLRQ5HVXOWRI$/6SDWLHQWVTable 1 shows the evaluation results. In response to thequestion 1, "Would you like to use the system shown in thevideo?", the answer was "Yes". In response to question 2,"Would you like to use the system you saw in the video?",
the answers were "I can use it by myself," "I cancommunicate easily," "I can communicate with others as myfavorite character," and "I can talk with many people even if
I can't go out". In addition, "It is easy to listen to because itis speech synthesized and the pronunciation can beinflected" and "The predictive conversion seems easy touse". Next, to question 3, "What would you like to use as acharacter for the spoken dialogue agent in this system?", therespondent answered, "My current live-action movie" ,"Myfavorite animal character (dog, cat, etc.)", "My favorite
animation", and so on. To the question 4, "What would you
like to use as the voice of this system?", the answers were"my voice", "favorite star's voice", "favorite animation", andso on. There were no responses to the question 5, "WhatInternational Workshop on Informatics ( IWIN 2022 )improvements do you think are needed in the layout of theapplication screen of the system?". To the question 6, "Inaddition to the remote conversation function, what otherservice functions would you like to see in this system?",other comments included "a doorbell and zoom function,"and "more types of keyboards”. Next, there were no
responses to the question 7, "What do you think needs to be
improved in the future?" Finally, to the question 8, "Pleasefeel free to give us any other comments or requests you may
have”, the answer was "I use the Internet, YouTube,
Facebook, LINE, and recently Google Sheets instead of
Excel. I rarely use other apps, so I thought it would be betterto create a well-balanced system. Please let me reserve myjudgment until I actually use the system.

the practical application of the proposed system. We had
only one ALS patient as the respondent in the questionnaireevaluation this time, so we would like to increase thenumber of questionnaires in the future. In addition, we
would like to evaluate the proposed system from multiple
viewpoints by conducting questionnaire evaluation for
doctors who are engaged in ALS treatment.
Table1:Results of Questionnaire Evaluation of ALS
Patient.

&216,'(5$7,21

From the results of the questionnaire evaluation of the
ALS patient, in response to the question 1, "Would you liketo use the system you saw in the video?", the answer was"Yes" . This indicates that the system is probably effective
even from the viewpoint of ALS patients. Next, from theresults of Question 2, we can expect that the system can beused in various situations, since the respondent selected all
the items as the reasons why the patient would like to use
the system. Next, from the results of Question 3, therespondent selected all the options for the character of the
spoken dialogue agent in this system except for "live-action
videos of my past when I was healthy". This may suggestthat ALS patients do not want to return to what they were
before they became ill. Next, from question 4, therespondent selected all of the items the respondent wanted touse as the voice of the system, which suggests that the
respondent want the voice to match the character. Next,
from Question 5, the layout of the application screen of thesystem was evaluated to be good, and there was no point of
improvement from the viewpoint of the ALS patient. Next,
from the question 6, what kind of service functions otherthan the remote conversation function should be added tothis system, the respondent selected all of the options, which
confirms that more functions are required. In particular, it isconsidered necessary to add functions that make life moreconvenient, such as remote control of home appliances andhealth management functions. The result of Q7 suggests thatthe respondent found none of the listed items needed to be
specifically mentioned as an item that should be further
improved. Finally, the results of Question 8 showed that
ALS patients use various tools such as the Internet,
YouTube, and SNS. From this result, we can assume that the
ALS patient who responded to this questionnaire has
considerable knowledge about computers. We would like toincrease the number of subjects in the future, since we had
only one subject this time.

$&.12:/('*(0(176This study was approved by the Ethical Review Board forthe use of human subjects of Kanagawa Institute ofTechnology (No.20220715-01).

)8785(:25.

From the evaluation of the ALS patient’s questionnaires,we were able to understand what improvements arenecessary for the proposed system. Based on the results ofthe evaluation, we would like to make improvements toward

5()(5(1&(6[1] "Orihime Product Introduction ", < https://orihime.orylab.com/>, (Referred May.2022)㸬

169

International Workshop on Informatics ( IWIN 2022 )[2] “Zoom”㸪< https://zoom.us/jp-jp/meetings.html>㸪
(Referred May.2022)㸬
[3] “TALKING AID+”㸪
<https://www.talkingaid.net/products/ta-plus>㸪
(Referred May.2022).
[4] "LUCY Double Giken "㸪<http://www.jd.co.jp/fukushikiki-lucy.html> 㸪 (Referred May.2022) 㸬[5] "Metaverse Japan", <https://metaverse-japan.org/>,
(Referred May.2022)㸬
[6] Eye Tracker Product List|Tobii Technology
https://www.tobiipro.com/ja/product-listing>,( Referred
May.2022).
[7] Nagisa Kokubu Tetsuto Mukai, Keiichi Abe, " A
Proposal for a Low-Cost voice Dialogue System Using
Live-Action Video Contents ", 2021 IEEE Global
Conference on Consumer Electronics Proceedings(GCCE2021)㸪pp.340-341, Oct. 2021.
[8] FaceRig Live2D Module, <https://www.live2d.com/interviews/facerig>, (Referred May.2022).
[9] Open J Talk, < http://open-jtalk.sp.nitech.ac.jp>,(Referred May.2022).
[10] coestation, <https://coestation.jp/consumer/coestationapp/>, (Referred May.2022).
[11] Lyrebird, <https://www.descript.com/lyrebird>,(Referred May.2022).

170

International Workshop on Informatics ( IWIN 2022 )Path Planning considering Pedestrian Characteristics
Yuto Yada*, Shunsuke Michita*, Seiji Komiya* and Toshihiro Wakita**

Graduate School of Engineering, Kanagawa Institute of Technology, Japans2284001@cco.kanagawa-it.ac.jp

Abstract - Autonomous mobility in mixed traffic environments with pedestrians need functions to avoid contact withpedestrians. In this study, path planning method adapted topedestrian face direction was developed. For pedestrians who
are aware of mobility (forward facing walking), small avoidance path is generated. For pedestrians who are unaware ofmobility (downward facing walking), such as those who arewalking on their smartphones, large avoidance path is generated. Subjective evaluation experiments were conducted onfour items: distance, speed, smoothness of avoidance, and reliability. The subjective evaluation results showed that theevaluations improved for all items except speed, both for forward and downward walking. In particular, for downward
facing pedestrians the evaluation of the distance was considerably improved.

notice mobility with sound warnings, this method causes mobility to impede pedestrians' walking, and frequent warningsounds can make pedestrians uncomfortable. Especially whenautonomous mobility increases in the future, it is unlikely that
autonomous mobility will always be prioritized over pedestrians. As another means, a method of avoiding large can beconsidered. Although the method could avoid the pedestriansafely, such large avoidance would be excessive for pedestrians walking forward. If excessive avoidance is always performed, there is a high possibility that it will take a long time
to arrive at the destination or the route cannot be generated
and the mobility cannot move.
Therefore, in this study, a method to adjust the amount of
avoidance according to the face direction was attempted. After predicting the pedestrian's behavior, the risk of collision is
reduced by avoiding a small amount when the pedestrian's
face is in front of the vehicle and a large amount when theface is facing downwards. Despite avoiding pedestrians using
this method, if a collision is unavoidable, the mobility stops.
This avoidance strategy is similar to that used by humans
every day.

Keywords: Autonomous Mobility, Pedestrian behavior
prediction, Yolo

1

INTRODUCTION

In recent years, the practical application of autonomous mobility has been progressing worldwide in areas such as officebuilding security and package delivery. Autonomous mobility move in mixed traffic environments with pedestrians needa path planning function that avoids contact with pedestrians.
When humans pass each other, they unconsciously make eye
contact with each other and anticipate the other's movements
to ensure smooth movement. Therefore, we are working on
the realization of autonomous mobility that enables this typeof behavior.
DWA (Dynamic Window Approach) [1] and RRT (Rapidly
exploring random tree) [2] have been widely used as static
obstacle avoidance methods for mobile mobility. The problem with these previous studies was that dynamic obstacles
such as pedestrians could not be avoided because they were
not considered.
For dynamic obstacle avoidance, pedestrian prediction using the Kalman filter[3], pedestrian prediction and avoidance
using the potential method[4], and the application of ORCA
(Optimal Reciprocal Collision Avoidance) to the prediction
and avoidance of multiple pedestrians[5] have been studied.
One of the problem for previous studies is avoidance forpedestrians walking on their smartphones. It is difficult formobility to avoid pedestrians walking while gazing at theirsmartphones. This is because their walking path is unstableand behavior prediction is difficult. Also, pedestrians may besurprised when mobility suddenly appears in their field of vision when they pass by at close range while they are gazingat their smartphones. To solve the problem, a method of warning by sound can be considered. Although pedestrians may

2 METHOD
EQUIPMENT
2.1

AND

EXPERIMENTAL

MethodTo safely avoid a downward-facing pedestrian, the face direction of the pedestrian is recognized by face direction
recognition. Next, if the result of the face direction is a downward-facing pedestrian, a large avoidance path is generated.
We considered these two requirements for pedestrian avoidance.
The method is based on the following procedure.
Step1 Measurement of pedestrian position using 3D LiDAR,
pedestrian detection, tracking, and pedestrian behavior prediction.
Step2 Pedestrian face direction detection by image recognition .
Step3 Collision risk area calculation based on the pedestrian
behavior prediction and pedestrian face direction detection results.
Step4 Pedestrian avoidance route generation.
A schematic diagram of the proposed algorithm is shown below (Figure 1).

171International Workshop on Informatics ( IWIN 2022 )was about 99% and that of downward-facing was about 77%(Table.2).

Figure 1: Method of pedestrian cooperative path planning
As in previous studies [3], 3D LiDAR information and a Kalman filter were used for pedestrian recognition and pedestrian
behavior prediction.

2.2

Figure 3: Recognized example of face direction
Table 2: Recognition result of face direction
Recognition result
Forward
Downward
Human
Forward99%
1%
behav- Downward
23%
77%
ior

System Configuration

The autonomous mobility used in this experiment is shownin Fig. 2. It was equipped with a camera for face direction
detection of pedestrians and an omnidirectional laser sensor
for self-localization and obstacle detection. Data obtained
from these onboard devices is processed and controlled by acompact computer DH310 (Shuttle) and a Jetson Xavier NX(NVIDIA) (Table.1).

2.4

Route generation was performed by RRT* [7] based on theoccupancy grid map.
Based on the results of pedestrian face direction recognition and pedestrian behavior prediction, collision risk areas
were defined on the occupancy grid map used in route generation (Fig. 4). For pedestrian behavior prediction, the walking
speed of the tracked pedestrian was calculated using a Kalman filter, and the predicted position was calculated based on
the calculated walking speed.
The width of the collision risk area is the same as the widthof the body when avoiding a forward-facing pedestrian (hereinafter referred to as 'small avoidance'), and is wider when
avoiding a downward-facing pedestrian (hereinafter referred
to as 'large avoidance') so that a safe distance is maintained
between the pedestrian and the collision risk area. The collision risk area was treated like an obstacle in the route generation.

Figure 2: Mobility

CameraLiDAR
Computers

Path Planning

Table 1: System Configuration
C920n web camera (Logicool)
VLP-16 (Velodyne)
DH310 (Shutlle)
Jetson Xavier NX (NVIDIA)

Figure 4: Collision risk area

2.3

Face Recognition
3

To estimate whether pedestrians are aware of autonomous
mobility or not, this study assumes that pedestrians whose
faces are forward-facing are aware of autonomous mobility
and those whose faces are downward-facing are not aware.
Pedestrian face direction recognition was performed using
deep learning with Yolo [6]. An example of recognition isshown in Figure 3. First, 300 images were taken of each pedestrian with a forward face and a downward face. Next, 4000
epochs of deep learning by Yolo were performed using thecollected images. The recognition rate of forwarding-facing

EXPERIMENTAL RESULTS

Using the proposed path planning algorithm, a subjective
evaluation experiment on pedestrian avoidance was conducted in a laboratory. A course was created as shown in Fig.
5, and the autonomous mobility was moved at a translational
velocity of 0.5 m/s. Pedestrians were instructed to walk attheir natural walking speed and evaluate whether the autonomous mobility could avoid pedestrians.

172

International Workshop on Informatics ( IWIN 2022 )Mobility

Pedestrian

Figure 5: layout of pedestrian avoidance experiment

3.1

Figure 9: Trajectory example of without behavior prediction path planning.

Path Planning Results

The path planning results are shown below. For comparison, similar experiments were conducted under path planningwithout behavior prediction. Three types of path planningwere used: without behavior prediction (Fig. 6), small avoidance (Fig. 7), and large avoidance (Fig. 8). The bold lines are
the selected paths and the branches are the candidate paths. It
was confirmed that the system generated a largely avoidable
path for downward-facing pedestrians compared to forwardfacing pedestrians.

Mobility

Pedestrian

Figure 10: Trajectory example of small avoidance cooperative path planning.

Mobility

Figure 6: Path planning example of without behavior prediction path planning. The bold line represents the selected path and the branch line represent candidate path.

Pedestrian

Figure 11: Trajectory example of large avoidance cooperative path planning.
The paths also showed that large avoidance was avoided toa greater extent than small avoidance and that the timing of
avoidance was delayed without the collision risk area.

Figure 7: Path planning example of small avoidance cooperative path planning. The bold line represents the selected path and the branch lines represent candidate path.
The square represents the collision risk area.

3.2

Subjective Evaluation Results

Subjective evaluation of pedestrian avoidance performancewas carried out on nine subjects. Two trials of each condition
were made to each subject. Experiments were carried out
based on the approval of the Human Ethics Review Committee of Kanagawa institute of technology.
Subjective evaluation was performed with 4 evaluation
items. They are "distance from the autonomous mobility
when passing by", "speed of the mobility when passing by","smoothness of passing by (avoidance performance)", and"reliability when passing by". These items were evaluated infive levels, with the following ratings: 'good', 'a little good','undecided', 'a little bad', and 'bad'.
Subjective evaluation results are as follows (Figs. 12 and13). In the case of forward-facing pedestrians, the evaluationof all items improved in comparison without behavior prediction and small avoidance. In the case of downward-facing pedestrians, the evaluation values of all items improved in comparison without behavior prediction and large avoidance. Especially, in the case of downward-facing pedestrians, theevaluation of distance was considerably improved. No

Figure 8: Path planning example of large avoidance cooperative path planning. The bold line represents the selected path and the branch lines represent candidate path.
The square represents the collision risk area.
Figs 9, 10, and 11 show the trajectory examples for without
behavior prediction, small avoidance, and large avoidance,
respectively.

173

International Workshop on Informatics ( IWIN 2022 )Table 4: Result of significance test for downward facing
pedestrian (* p<0.05, ** p<0.01).

change was observed in the evaluation of speed, partly because the vehicles were driven at the same speed in all three
conditions.

item
Distance
Speed
Smoothness
Reliability

4 DISCUSSION: THE RESULTS OF THE
SUBJECTIVE EVALUATION

Figure 12: Subjective evaluation results for conscious pedestrians.

Both forward and downward-facing pedestrians improved
the subjective evaluation results for all items except speed.
Speed was highly rated in all three conditions, with no significant differences observed. It is considered that this is because
the mobility moved at a constant speed under all experimental
conditions.
In the subjective evaluation of forward-facing pedestrians,
the evaluation of large avoidance was slightly worse than thatof small avoidance. Several subjects commented on the poorsmoothness of large avoidance, such as "I felt poor smoothness" and "If I were a human, I would feel un-comfortable asif I were being large avoided”. From this result, it seems that
small avoidance is appropriate for for-ward-facing pedestrians.
Significant differences in distance and reliability were foundfor forward-facing walking. This result is thought to be dueto the fact that the mobility without prediction (no collisionrisk area) pass pedestrians at a close distance, while thosewith a collision risk area maintain a certain distance while
avoiding pedestrians.
In downward-facing walking, significant differences wereobserved in all items except speed. In forward-facing walking,the presence of mobility can be confirmed early on in the effective field of view, whereas in downward-facing walking,the effective field of view is narrower than in forward-facing
walking because walking is done while gazing at thesmartphone [9], and the pedestrian only confirms the presence of the mobility when it enters the peripheral field of view
just before passing by. Therefore, the evaluation of distance,
the reliability and smoothness decreased, whereas with thecollision risk area, the reliability also improved because therobot maintains a maximum safe distance and makes a larger
avoidance compared to forward-facing walking.

Figure 13: Subjective evaluation result.
Significant difference tests were conducted paired t-test witha significance level of 0.05 and a null hypothesis of 'no difference in mean values between the two groups. In thisstudy, subjective ratings were treated as interval scales.
Without behavior prediction and small avoidance were compared for forward-facing pedestrians, and without behavior
prediction and large avoidance were compared for downward-facing pedestrians. The p-values for each item areshown below (Tables 3 and 4).

Table 3: Result of significance test for forward facing pedestrian (* p<0.05).
item
Distance
Speed
Smoothness
Reliability

P-value0.005**
significantly different
0.040*
significantly different
0.014*
significantly different
0.006**
significantly different

P-value
0.02*
significantly different
0.173
0.046*
significantly different
0.021*
significantly different

5

CONCLUSION

In this study, we proposed a path planning algorithm that
adapts to the face direction of pedestrians and safely avoids
pedestrians who are walking while on their smartphones.
The effectiveness of this algorithm was confirmed by subjective evaluation.
This method would enable the operation of advanced collaboration between pedestrians and autonomous mobility on
campus.

174

International Workshop on Informatics ( IWIN 2022 )REFERENCES[1] D. Fox, W. Burgard, and S. Thrun, “The dynamic win-

dow approach to collision avoidance,” IEEE Robot. Autom. Mag., vol. 4, no. 1, pp. 23-33,(1997).
[2] S. M. LaValle, “Rapidly-Exploring Random Trees: A
New Tool for
Path Planning,” Tech. Rep., (1998).
[3] Takumi Goto, “Pedestrian behavior prediction and forecast circle generation using Kalman filter”(2019)[4] Hiroshi Hisahara et al, “Human Avoidance Function for
Robotic Vacuum Cleaner Through Use of Environmental Sensors - Roomba® Making Way for Humans” ,IEEE, Fifth International Conference on Intelligent Systems, Modelling and Simulation, pp.64-67.(2014)[5] Dongxiang Zhang, Zongjun Xie et al,“Real-Time Navigation in Dynamic Human Environments Using Optimal
Reciprocal Collision Avoidance”， IEEE International
Conference on Mechatronics and Automation.(2015).
[6] J.Redmon et al., “You Only Look Once: Unified, RealTime Object Detection”, 2016 IEEE Conference on
Computer Vision and Pattern Recognition (2016).
[7] “Plan Mobile Robot Paths Using RRT”,
https://jp.mathworks.com/help/nav/ug/plan-mobile-robot-paths-using-rrt.html.
[8] Chisato Shibata et al，“Driver characteristic estimation
using vehicle behavior fata while preceding vehicle decelerating”，Information Processing Society of Japan(2016).
[9] Daisuke Saito“Change in effective visual field using
smartphone with walking”，Biomedical Fuzzy System
Association (2019).
Title: Pedestrian Cooperative Autonomous Mobility -Pathplanning adapted to pedestrian face directionAuthor: Yuto Yada, Shunsuke Michita, Seiji Komiya , Tos
hihiro Wakita
Corresponding Author: Yuto Yada
Tel: 046-291-3091
Fax: 046-242-6806
E-mail: s2284001@cco.kanagawa-it.ac.jp

175International Workshop on Informatics ( IWIN 2022 )Implementation of Spatio-temporal Fencing for Crowd Sensing in a Smartphone
Applications
SHOTA SUZAKI† , NOBUHITO MIYAGAWA† , and KATSUHIKO KAJI††

Aichi Institute of Technology, Japan

Abstract - Crowdsensing platforms are available to solve
the problems of dedicated system development and operating costs associated with crowdsensing. Among them, there
are several crowdsensing platforms that provide incentive elements to motivate and maintain users. We build a crowdsensing platform (Lavlus) to mitigate disincentive factors thatcan be used in conjunction with incentive factors. This study
implements a smartphone application for Lavlus. The Lavlus
smartphone app needs to have the following functions: sensing project download, sensing request notiﬁcation, automatic
sensing, and sensor data upload. Among them, sensing request notiﬁcation and automatic sensing are based on Spatiotemporal fencing. Geofencing used latitude and longitude,
which are easy for users to visually recognize, but GPS is unstable indoors and in areas with many buildings. Therefore, amargin is provided on the geofence to determine whether theuser has entered or exited the geofence with certainty. Lavlus
may also target speciﬁc facilities, such as amusement parks orfactories, as actual use cases. In this case, the geofence may
be arbitrarily polygonal. We then draw a circle with the user at
its center and generate geofencing points at eight points on thecircumference of the circle. If one or more of the generated
points are inside the geofence, the system determines that it
is about to enter space-time and issues a notiﬁcation whether
or not it will cooperate with sensing. If all eight points are
inside, the application judges that it has deﬁnitely entered thearea and performs automatic sensing. If all eight points areoutside, the application judges that it has deﬁnitely exited andterminates sensing. This allows proper geofencing even when
the geofence is arbitrarily polygonal. In this paper, we implemented these functions and veriﬁed their operation. As aresult, the geofence worked properly for arbitrary polygons.
Keywords: Spatio-temporal fencing, crowdsensing, crowdsensing platform, smartphones

1 Introduction
In recent years, the number of smartphones equipped withsophisticated sensors has been increasing, and a wide variety of sensors are becoming available. Therefore, there are anumber of research that use smartphone sensors[1][2]．Crowdsensing is an attempt to utilize the sensing capability of smartphones[3][4][5]. Crowdsensing is currently being employed
in research and surveys[6][7][8]. The implementation of crowdsensing requires the development of a dedicated system, whichis expected to incur signiﬁcant initial and running costs. In
addition, in order to encourage many collaborators to cooperate in crowdsensing, there are issues such as reducing the timeand effort required for collaborators and eliminating their anx-

iety. As for the time and effort required for collaborators,
the burden of operating and communicating with smartphones
can be mentioned. The concerns of the collaborators include
worries about providing data due to privacy barriers of thecollaborators and distrust of sensing. Furthermore, since crowdsensing handles a lot of sensitive data such as sensor data,
security and privacy protection measures are essential.
The initial and running costs required to implement crowdsensing can be solved by a crowdsensing platform. However,crowdsensing platforms face several challenges, which wehave addressed by proposing a crowdsensing platform basedon Spatio-temporal fencing called ”Lavlus”. Create a crowdsensing platform to enable easy use of crowdsensing and diverse data collection to signiﬁcantly reduce costs (time, money,
and effort) in research and surveys. We also propose Spatiotemporal fencing, and expect those who use crowdsensing tocollect sensor data (hereafter referred to as ”requesters”) to
easily deﬁne the scope of sensing, and those who cooperatewith crowdsensing to provide sensor data (hereafter referred
to as ”collaborators”) to have a clear perception of the spacetime that is being sensed. When using Lavlus, the requester
uses a dedicated web application and the collaborator uses a
dedicated mobile application. This study is about mobile applications of Lavlus.

2 Related Research
There are several research that use crowdsensing to collect data from a large number of people for estimation andanalysis. For example, crowdsensing using mobile devices
is used to collect and share ambient sounds to conduct noise
surveys[6][7], and motion sensors such as accelerometers areused to collect data from car users to estimate road conditions
such as icy and paved roads, and road geometry such as ﬂatand hollow surfaces[8]. In these research, the development ofa crowdsensing system is expected to incur signiﬁcant costs.
To implement crowdsensing, it is necessary to develop a sensing smartphone application exclusively for the collaborator
and a server to manage the collected data.
Therefore, a crowdsensing platform would be very useful if
the requester wishes to use crowdsensing to collect data. For
the requester, it is no longer necessary to create and distributea dedicated smartphone application for sensing for each research, thus eliminating the time and effort spent on thesetasks. As for the collaborators, there is no need to install aseparate smartphone application for each research, and there
is no need to use separate applications for each research. In
addition, the use of a common smartphone application canlead collaborators to other crowdsensing applications, which
can lead to the acquisition of many collaborators.

177

International Workshop on Informatics ( IWIN 2022 )Next, we discuss related research on crowdsensing platforms. There are already some that operate as simple platforms mainly for the purpose of reducing system development
costs[9][10]. However, securing collaborators is very important for crowdsensing platforms. Related researches include
those to improve and maintain the motivation of collaboratorsand to secure collaborators.
This includes research using monetary incentives[11] andresearch that uses gamiﬁcation to provide non-monetary incentives[12]. There are also research that offer a ﬂexible choice
between monetary and non-monetary incentives. In this research, our approach is to remove disincentives for collaborators through Spatio-temporal fencing, and we are considering
introducing a mechanism that can provide incentives in the
future.
Next, we discuss related research on crowdsensing platforms. Since securing collaborators is extremely important
for a crowdsensing platform, it is necessary to devise ways
to increase and maintain motivation. There are two ways tomotivate collaborators: research [11] using monetary incentives and research [12] using non-monetary incentives. Thesemethods of increasing and maintaining motivation focus on
incentive factors, and not much is said about disincentive factors. This system aims to reduce the disincentive factors thatcan be used in conjunction with these incentive factors.
Smartphones are used for many of the sensing terminals in
the crowdsensing platform[9][10][11]．One of the advantagesof using smartphones as sensing terminals for crowdsensing
is that smartphones have high penetration rates and there is noneed to secure, distribute, or upgrade new sensing terminals.
In addition, if a mobile application dedicated to crowdsensing
is created, people around the world can participate in crowdsensing via the Internet. However, as a disadvantage, smartphones are likely to be carried at all times, and the sensor
data obtained may violate privacy if the collaborator senses atunintended places and times. In this study, Spatio-temporal
fencing is expected to provide sensing that does not violate
the privacy of collaborators.

3 Crowdsensing Platform Based on
Spatio-temporal Fencing
This section describes the details of a crowd-sensing platform (Lavlus) based on Spatio-temporal fencing. Each section is organized in the following order: deﬁnition of spacetime fencing, overall view of Lavlus

3.1 Deﬁnition of space-time fencing
Spatio-temporal fencing is deﬁned as ”a fencing methodthat extends geo-fencing by adding a time element. An overview
of Spatio-temporal fencing is shown in Figure 1. Geo-fencing
is a fencing method that uses location-estimation technologies such as GPS, Wi-Fi, and BLE beacons to generate virtual boundaries, and provides speciﬁc services when a user
enters or exits the boundary. With the increasing accuracy oflocation estimation and the widespread use of devices such
as BLE, which can easily construct geofences, various applications using geofencing have been realized. In other words,

Figure 1: Spatio-temporal fencing with additional geofencing and time elements
Spatio-temporal fencing is a fencing method that generates
virtual boundaries by specifying time and space, and performs
a speciﬁc service when a user enters or exits the boundary.
The speciﬁc service is ”sensing” in this study. Sensing is initiated when it enters within the virtual boundary and terminatedwhen it exits outside the boundary.
One advantage of Spatio-temporal fencing is that delimiting the boundaries by time and area allows the requester tospecify various situations for crowd-sensing. The range thatcan be deﬁned for Spatio-temporal fencing is the range within
which time and space can be properly separated. For example, a park from 3 p.m. to 5 p.m. or an amusement park duringopening hours. On the other hand, the extent to which timeand space cannot be properly separated cannot be deﬁned. For
example, it is not suitable only during rainfall, when time cannot be properly separated, or on a moving train, when spacecannot be properly separated.

3.2 Overall view of Lavlus
The overall view of Lavlus is shown in Figure 2. The sequence of steps in Lavlus is ”deﬁning a sensing project ina web app,” ”Spatio-temporal fencing,” ”accepting a sensing
request,” ”automatically sensing,” ”automatically uploading
under Wi-Fi,” and ”using the data. The requester deﬁnes thedetails of the sensing request in detail on the project management web application and creates a sensing project. The
collaborator downloads the sensing project created by the requester. Spatio-temporal fencing is performed by the collaborator according to the sensing project, based on the deﬁnitionin the 3.1 clause, and notiﬁcations are sent only if the collaborator is in the set space-time. If the collaborator agrees
with the requestor and the contents of the crowdsensing presented on the sensing request screen, and cooperates with thesensing, the collaborator accepts the sensing request. When acollaborator accepts the sensing request and enters the timespace, sensing starts automatically in the background. Whenthe sensing is ﬁnished and the collaborator is under Wi-Fi, thesensor data is uploaded.
This platform can be used only for crowd-sensing, where
space-time can be set appropriately and sensing is done inunconsciousness. For example, suppose that an amusement

178

International Workshop on Informatics ( IWIN 2022 )Figure 2: Overall view of Lavlus
park management company tries to sense the movement history of amusement park visitors in order to understand theirtrends. In this case, the time is set from the opening time tothe closing time of the amusement park, the space is set to theinside of the amusement park, and the required sensor data
are position information and acceleration. As another example, suppose a teacher wants to make environmental measurements to control the air conditioning of a classroom where heor she teaches a class. In this case, the time is set to the timeof day when the class is held, the space is set to the classroom,and the required sensor data are temperature and humidity.

4 Application for automatic sensing uponentry into a speciﬁc time-space
This section describes an application for automatic sensing upon entry into a speciﬁc time-space. In section 4.1, we
ﬁrst deﬁne the requirements speciﬁcation for Lavlus’ mobile
application. Section 4.2 describes the implementation of an
application that automatically senses when entering a speciﬁc time-space. Section 4.3 describes the implementation of
Spatio-temporal fencing. Section 4.4 describes the implementation of sensing request notiﬁcations. Section 4.5 describes
the implementation of automatic sensing.

4.1 Requirements speciﬁcation for Lavlus’
mobile application
The Lavlus mobile application requirements speciﬁcationis shown in Figure3. The Lavlus mobile app needs to be able
to request download, sensing request notiﬁcation, automatically sensing and sensor data upload.

The Lavlus mobile app needs to reduce the disincentive factor for collaborators. In order to reduce the physical costs
for the collaborator, it is necessary to notify the collaborator
and reduce the collaborator’s own operations, and not to overwhelm the data trafﬁc of the terminal. Notiﬁcation should be
issued only to collaborators who are likely to enter the spacetime to minimize notiﬁcation to collaborators. In addition,
once a collaborator accepts or rejects a sensing request, no notiﬁcation needs to be issued from that project. The requester
may feel annoyed if notiﬁcations are issued for crowdsensing that he/she cannot participate in or for crowdsensing thathe/she once rejected. Therefore, notiﬁcation to collaborators
should be kept to a minimum. To reduce the number of operations by the collaborator, the collaborator’s operations should
be limited to tapping the sensing request notiﬁcation and tapping the Accept/Reject button on the sensing request screen,with the exception of installing applications, etc. Therefore,
all Spatio-temporal fencing and sensing must be done automatically in the background. Collaborators may stop cooperating with crowdsensing if there is too much manipulation oreffort involved before they cooperate. When downloading asensing project, it is necessary to download only those sensing projects in which collaborators are likely to participate inorder not to overwhelm the communication and data capacityof the terminal. It also needs to be uploaded automaticallywhen connected to Wi-Fi after sensing is complete.
To reduce the psychological cost for collaborators, theyneed to be able to delete sensor data that they do not want
to be used. It is also necessary to be able to delete sensor datathat you do not want to be used even after uploading. Therefore, collaborators need to be able to reject a sensing project179

International Workshop on Informatics ( IWIN 2022 )Issue a sensing request notiﬁcation to collaborators who arelikely to enter the space-time or who are already inside thespace-time. Tap the sensing request notiﬁcation to display thesensing request screen. The collaborator checks the information about the requester and crowdsensing presented on thesensing request screen and chooses whether or not to cooperate with crowdsensing. If the collaborator agrees to cooperate with crowdsensing, he/she presses the ”Accept” button on
the sensing request screen; if not, he/she presses the ”Deny”
button. If the collaborator agrees to crowdsensing and is definitely inside the space-time, it is automatically sensed.

Figure 4: Overall view of an application for automatic sensing
upon entry into a speciﬁc time and space

4.3 Implementation of Spatio-temporal
fencing
Latitude and longitude, BLE beacons, and Wi-Fi are someof the methods used to generate geofences. Lavlus needs toensure that the requester and collaborator are aware of thegeofence. For this reason, the latitude and longitude wereused in this case because they are visually easy to recognize.

Figure 3: requirements speciﬁcation for Lavlus’ mobile applicationthat they have already accepted, or request the deletion of sensor data that they have already uploaded. If the data cannot bedeleted once sensed, collaborators cannot casually sense thedata. In this case, the number of collaborators may decrease
because it will be difﬁcult for them to participate in crowdsensing.

4.2 Implementation of an application that
automatically senses when entering a
speciﬁc time-space

Figure 5: Geo-fencing using point inside/outside determination

Implement a mobile app for sensing corresponding to thesensing project produced by the requester. This application
only implements Spatio-temporal fencing, sensing requests,
and automatic sensing out of those described in section 4.1.
The overall diagram of the implemented application is shownin Fig4．

Use the polygon interior/exterior decision algorithm to deal
with the case where the geofence is an arbitrary polygon. The
inside/outside judgment for a polygon of points is shown in
Figure5. To determine whether a point is inside or outside a
polygon, ﬁrst draw a straight line from the point (red circle)
to the polygon. If the number of intersections (blue circles)

180

International Workshop on Informatics ( IWIN 2022 )between the line and the polygon is odd, the line is inside
the polygon; if it is even, the line is outside the polygon. In
this case, if exactly a line overlaps an edge of a polygon ora line overlaps a vertex of a polygon, a wrong decision ismade. However, since latitude and longitude have seven decimal places and location information keeps changing, exactly
overlapping cases are not considered this time. No matter
how complex a polygonal area the requester sets up, such as
the lower part of Figure5, it is possible to determine whether
it is interior or not. Although not implemented in this case,
it is also possible to determine the area in the shape of a circle. In addition, since only GPS is used to determine the area,
only a ﬂat area can be determined in this case. Determinationof three-dimensional areas such as the ﬁrst and second ﬂoors
is a subject for future work.
In order to cope with various situations, such as sensing
only when the user is deﬁnitely inside the space-time interval, or issuing sensing request notiﬁcations to collaborators
who are likely to enter the space-time interval, we have implemented a margin that allows the space-time interval to be
expanded and contracted. If the space-time is to be determined with certainty, the space-time is reduced by a margin.
When determining whether or not the user is in the spacetime perimeter, an expanded space-time margin is used. If thegeofence is a rectangle made up of only lines parallel to thelatitude and longitude lines, the margin can be implemented
using addition and subtraction. To enlarge the geofence, add
a margin to the northernmost latitude and easternmost longitude, and subtract a margin to the southernmost latitude andwesternmost longitude. To reduce the geofence, subtract amargin from the northernmost latitude and easternmost longitude and add a margin to the southernmost latitude and westernmost longitude. However, this application needs to deal
with the case where the geofence is an arbitrary polygon. So
we let the collaborator, not the geofence, have the margin.

generated points are inside the geofence, the collaborator isconsidered likely to enter the geofence. If all generated points
are inside the geofence, the collaborator is deﬁnitely inside
the geofence. If all generated points are outside the geofence,the collaborator is deﬁnitely outside the geofence.

4.4 Implementation of sensing request
notiﬁcations
To reduce the number of sensing request notiﬁcations tocollaborators, sensing request notiﬁcations are issued to collaborators who are most likely to participate in crowdsensing. Collaborators who are close to the space-time set for thesensing project and already inside the space-time are considered likely to participate in that crowdsensing project. For example, a collaborator who is in the vicinity of crowdsensingthat has already begun, or a collaborator who will soon begin
crowdsensing where he or she is located. Therefore, we takea margin to widen the time-space, and issue a notiﬁcation if
the user is inside that time-space. Assume that collaborators
who are located far from the space set aside for the sensing
project are unlikely to participate in that crowdsensing. For
example, collaborators who are far away from the crowdsensing that has already begun. Issue a notice to a collaborator
who is unlikely to participate in crowdsensing, and even if
they accept the sensing request, they are unlikely to receive
sensor data. In addition, collaborators may feel uncomfortable with repeated issuance of crowd-sensing notiﬁcations inwhich they do not participate.

Figure 7: Example of issued notiﬁcation and sensing request
screenFigure 6: Based on a few points on the circumference centered on the collaborator, the algorithm determines if the collaborator is likely to enter the geofence, if the collaborator isdeﬁnitely inside the geofence, or if the collaborator is deﬁnitely outside the geofence.
A summary of the margins that the collaborator has is shownin the ﬁgure. First, draw a circle with a radius of the lengthof the margin centered on the collaborator. It then generates
several points equally on its circumference. Geofencing is
performed on all the generated points. If one or more of the

When the collaborator taps the sensing request notiﬁcation, the sensing request screen is launched and the requester’s
name, the sensor to be used, and the time and space are presented (Figure7). If the collaborator is satisﬁed with the information presented and decides to cooperate, the collaborator presses the ”Accept Sensing” button and cooperates withcrowdsensing. The collaborator conﬁrms the information presented on the sensing request screen. Examples of information presented on the sensing request screen include the requester’s name, the name of the sensor to be used and its frequency. At this time, collaborators can reject crowdsensing
by pressing the ”Reject Sensing” button if they have any distrust or disagreement with the information presented.

181

International Workshop on Informatics ( IWIN 2022 )4.5 Implementation of automatic sensing
To reduce the manipulation of the collaborator, sensing is
automatically performed in the background when the collaborator has deﬁnitely agreed to the sensing request inside thespace-time. Therefore, the collaborator does not need to consciously open the smartphone application, and the sensing
does not terminate during the process unless the application
itself is terminated. In addition to reducing the burden of thecollaborator’s operations, we expect the system to sense theuser’s usual behavior because it does not make the user awareof the sensing process.
In order to determine whether or not the collaborator is definitely inside the space-time of the collaborator, a narrower
space-time margin is taken on entry and a wider space-time
margin is taken on exit. If the geofencing is near the boundary
and the location information becomes unstable, the entry/exit
decision is repeated. If the location information is unstablearound space-time, it may be determined to be inside spacetime even though it is outside space-time. To prevent this,a margin is taken. However, if the space-time is small, too
narrow a margin makes it difﬁcult or impossible to enter the
space-time. For example, for a geofence that is less than 10
meters in either direction, a narrowing margin of 5 meters ormore will make it impossible to enter. Similarly, if the margin
to narrow the time is more than the set time, entry will not be
possible. Therefore, when narrowing the space-time interval,
the margin must take into account the size of the space-time
interval. At this time, margins are set for each sensing project.
As a crowdsensing platform, it supports many sensors andfree frequencies, and abstracts sensor data that would violate
privacy. Many sensors and frequencies need to be supported
to accommodate a variety of crowd sensing. For example, gait
estimation may require acceleration sensors, barometric pressure sensors, angular velocity sensors, and location information. Sound sensors may be needed for noise measurements,
and temperature and humidity for environmental measurements. Not only the type of sensor but also the frequency thatcan be freely set is necessary. The frequency of the same sensor varies depending on the application. For example, whena barometric sensor is used for gait estimation, it senses at ahigh frequency. On the other hand, weather estimation does
not require sensing at high frequencies, so sensing is done atlower frequencies. The type of sensor and the frequency foreach sensor are set in the sensing project created by the requester. This application allows sensing for multiple sensors
and sensor frequencies according to the sensing project. For
example, frequencies can be set on a per-sensor basis, suchas sensing acceleration at 50 Hz and barometric pressure at
10 Hz. It may also enter multiple time-spaces simultaneously
and participate in multiple crowdsensing activities. This application manages sensor types and their frequencies for eachsensing project, so that multiple sensing can be performed
simultaneously. For example, even when cooperating simultaneously with crowdsensing that takes acceleration at 50 Hz
and crowdsensing that takes acceleration at 20 Hz, it is possible to sense appropriately.

5 Experiment
We veriﬁed that the geofence works properly when it isarbitrarily polygonal, including the margins. The evaluation
items for the Experiment are shown in Fig.8．First, check to
see if notiﬁcations have been issued to collaborators who arelikely to enter the geofence. The collaborators most likely
to enter the geofence are as described in the4.4 section. As
described in section4.3, when one or more points with an inside/outside decision attached to a collaborator are inside ageofence, the geofence is considered to have entered an expanded geofence and a notiﬁcation is issued. Next, verify
that the sensing starts when you are deﬁnitely inside the geofencing. If it is working properly, when all the points withinside/outside judgments attached to the collaborator enter thegeofence, it judges that it is deﬁnitely inside the geofence
and starts sensing. Finally, check to see if the sensing isterminated when the geo-fence is deﬁnitely exiting the geofence. If it is working properly, when all the points withinside/outside judgments attached to collaborators have left
the geofence, it is determined that they are deﬁnitely outside
the geofence, and sensing is terminated. The experimental
environment for Experiment and the actual route moved areshown in Figure9. A polygonal geo-fence is set up as shownin Figure8, and the geo-fence is moved through (Figure8 orange arrows).

Figure 8: Experiment of geofencing operation
The results of the operation veriﬁcation are shown in Fig10.
First, the issuance of the notice was veriﬁed upon entry into
the expanded geofence. Next, we veriﬁed that sensing begins
upon entry into the reduced geofence. Finally, we conﬁrmedthat sensing was terminated when they exited the expanded
geofence. From the above, it can be said that proper operation
was conﬁrmed.

6 Conclusion
In this paper, we deﬁned the requirements for the functions
required for mobile applications in a crowdsensing platform.
From these, Spatio-temporal fencing, sensing requests, andautomatic sensing were implemented. The implementationof Spatio-temporal fencing used polygon inside/outside decisions to accommodate the case where the geofence is an arbitrary polygon. In order to cope with various situations, suchas sensing only when the user is sure to enter space-time, or
issuing sensing request notiﬁcations to collaborators who are

182

International Workshop on Informatics ( IWIN 2022 )on Spatio-temporal fencing, which was not implemented inthis study. It is necessary to implement the linkage with theserver such as sensing project download and sensor data upload, which could not be implemented this time.
One of the challenges of Spatio-temporal fencing is thatit uses GPS for geo-fencing, which makes the operation unstable indoors where GPS accuracy is degraded. Since only
latitude and longitude are used for geofencing at this time, geofencing cannot be performed properly when GPS blurring ismore severe than expected, such as indoors or in areas withmany high-rise buildings. Latitude and longitude alone cannot be used to determine the ﬁrst or second ﬂoor. Therefore,
in addition to geofencing in latitude and longitude, it is necessary to implement BLE and Wi-Fi.

REFERENCES
Figure 9: Veriﬁcation of geofencing operation

Figure 10: Results of geofencing operation veriﬁcation
likely to enter space-time, the system implements a margin
that allows expansion and contraction of space-time. In addition, to accommodate the case where the geofence is a complex rectangle, a margin is provided for the location information of collaborators. The ﬁrst step in implementing a sensing
request was to deﬁne collaborators who are likely to participate in the sensing request. Notiﬁcation is then issued only tocollaborators who are likely to participate in the sensing request. In addition, a sensing request screen was implemented
to determine whether or not the collaborator would participate
in the sensing. The company agreed to the sensing request
and implemented a function that automatically senses when
the user enters the Spatio-temporal space. In order to reliably
determine whether a collaborator enters or exits space-time, anarrow space-time margin is used for entry, and a wide spacetime margin is used for exit. As a crowdsensing platform,
it supports many sensors and free frequencies, and abstracts
sensor data that would violate privacy.
In the operation veriﬁcation, we checked whether the Spatiotemporal fencing was properly performed. As a result, Spatiotemporal fencing was performed properly. In addition, we
veriﬁed whether the sensing was appropriate for actual use
cases. As a result, various sensor data were collected.
Future work includes the implementation of necessary functions for mobile applications in a crowdsensing platform based[1] Suyama, A., Inoue, U.: Using geofencing for a disaster information system, 2016 IEEE/ACIS 15th International Conference on Computer and Information Science, pp.1-5, 2016.
[2] Sugimori, D., Iwamoto, T., Matsumoto, M.: A Study
about Identiﬁcation of Pedestrian by Using 3-Axis Accelerometer, 2011 IEEE 17th International Conferenceon Embedded and Real-Time Computing Systems andApplications, Vol.2, pp.134-137, 2011.
[3] Burke, J., Estrin, D., Hansen, M., Parker, A., Ramanathan, N., Reddy, S., Srivastava, M. B.: Participatory Sensing, Workshop on World-Sensor-Web: Mobile Device Centric Sensor Networks and Applications,
2006.
[4] Raghu, G., Fan, Y., Hui, L.: Mobile Crowd Sensing:
Current State and Future Challenges, IEEE Communications Magazine, Vol.49, pp.32-39, 2011.
[5] Liu, J., Shen, H., Zhang, X.: A Survey of Mobile
Crowdsensing Techniques: A Critical Component for
the Internet of Things, 2016 25th International Conference on Computer Communication and Networks, pp.16, 2016.
[6] Eiman, K.: NoiseSPY: A Real-Time Mobile Phone
Platform for Urban Noise Monitoring and Mapping,
MONET, Vol.15, pp.562-574, 2010.
[7] Nicolas, M., Matthias, S., Bartek, O.: Participatory
noise pollution monitoring using mobile phones, Information Polity, Vol.15, pp.51-71, 2010.
[8] Bin, P., Kenro, A.: Detecting the road surface condition by using mobile crowdsensing with drive recorder,
2017 IEEE 20th International Conference on Intelligent
Transportation Systems, pp.1-8, 2017.
[9] Tangmunarunkit, H., Hsieh, C. K., Longstaff, B., Nolen,
S., Jenkins, J., Ketcham, C., Selsky, J., Alquaddoomi,F., George, D., Kang, J., Khalapyan, Z., Ooms, J., Ramanathan, N., Estrin, D.: Ohmage: A General and Extensible End-to-End Participatory Sensing Platform, Association for Computing Machinery, Vol.6, No.3, 2015.
[10] Ferreira, D., Kostakos, V., Dey, A. K.: AWARE: Mobile
Context Instrumentation Framework, Frontiers in ICT,
Vol.2, pp.6, 2015.
[11] Jayarajah, K., Balan, R. K., Radhakrishnan, M., Misra,

183

International Workshop on Informatics ( IWIN 2022 )A., Lee, Y.: LiveLabs: Building In-Situ Mobile Sensing & Behavioural Experimentation TestBeds, MobiSys
16: Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services,pp.1-15, 2016.
[12] Shogo, K., Yuki, M., Hirohiko, S., Manato, F., Yutaka, A., Keiichi, Y.: Gamiﬁed Participatory Sensingin Tourism: An Experimental Study of the Effects on
Tourist Behavior and Satisfaction, Smart Cities, Vol.3,
No.3, pp.736-757, 2020.

184

International Workshop on Informatics ( IWIN 2022 )6OHHSEIIHFWVRQNRLVH
±&RPSDULVRQRISOHHSEIIHFWVbHWZHHQSWXGHQWVDQGWRUNLQJPHRSOH
Tatsuki Takahashi*,Masaaki Hiroe** , and Mari Ueda**

Kanagawa Institute of Technology, Japan
Kobayasi Institute of Physical Research, Japans2285013@cco.kanagawa-it.ac.jp**

subjective impressions, and it is difficult to grasp andcontrol the noise environment during sleep. On the other
hand, in the latter indoor experiment, it is possible to graspand control the noise environment presented to the subject
and observe the sleep effect objectively, but it is also
necessary to consider the effect of changes in the sleep
environment on the subject. Therefore, in order tounderstand the effects on sleep corresponding to real life, inthe experiment by Kaku et al., [4] traffic noise recorded at
actual noise sites was presented to sleeping subjects using a
CD boom box, and the effects on sleep were investigatedusing two methods: a hand-arm type body motion meter
called an actigraphy and a questionnaire. The results
confirmed that under conditions of equal LAeq presentationlevels, railroad noise had a greater impact on sleep than roadtraffic noise. In addition, the actigraphy evaluation
confirmed that the incidence of mid-sleep awakenings
increased significantly for railroad noise when LAeq,1min
exceeded 50 dB, although there was not necessarily a clear
relationship with the questionnaire.

$EVWUDFW - We have investigated how noise affect sleep

using recorded transportation noise in participants’ ownhome since 2019. In this study, we analyzed subjective andobjective sleep assessment for twenty participants consist ofeach ten persons of working people and university students.
The comparison between the two groups showed that theworking peoples were more affected by noise-induced sleepin the subjective sleep assessment. In addition, it confirmed
dose-response relationships between noise level andincidence of wake after sleep onset (WASO) derived fromthe objective sleep assessment differ from each traffic noise.

.H\ZRUGV: sleep effects, traffic noise, questionnaire, bodymovement, statistical analysis.

 ,1752'8&7,21
Ensuring good quality sleep is a very important issue intoday's world, where one out of every five Japanese citizens
is becoming increasingly dissatisfied with their sleep.
Traffic noise is one of the environmental factors that affectsleep. The 2018 European WHO Environmental Noise
Guidelines [1] recommended that Lnight 40-45 dB (energyaveraged nighttime noise level) should be used to protect
people's health, which became a major topic of discussion.
The authors also started an experimental study of noiseinduced sleep effects in 2019, and in the previous year's
study [2,3], the authors analyzed the association between
subjective and objective sleep ratings and noise exposure
regarding sleep effects. The results showed that subjective
and objective sleep ratings did not agree and showed
different trends. As a consideration of the cause of thisdifference, the previous study had conducted experiments on
working peoples, but in the previous year's study, most of
the participants were university students, and the differencein sleep habits was considered to be one of the reasons forthe difference.
Therefore, in this study, experiments were conducted on
10 working peoples and 10 university students. Twoevaluation methods were used: an objective sleep evaluation
using an actigraphy body movement meter and a subjective
sleep evaluation using two questionnaires.

 0(7+2'6
In this study, subjects were randomly presented with ninedifferent test sounds, obtained from three types of traffic
noise and three conditions of presentation level, and theeffects of noise on sleep were determined by two methods: aquestionnaire and an actigraph (wristwatch-type body
movement meter: GT3X series, AMI). The actigraph used isshown in Fig. 1.
The questionnaire was administered as early as possible
after waking up the next morning, asking whether and towhat extent the test sound affected sleep. No sound wasplayed on the first day of the experiment in order to get used
to wearing the actigraphy, and the CD was played from thesecond day to the tenth night of the experiment.
The actigraph is an instrument that continuously measures
the amount of body movement that occurs within a certain
period of time and determines wakefulness based on a set
sleep score algorithm. The Cole-Kripke method was used todetermine wakefulness or sleep. These experimental
methods have been approved by the ethical review
committee of the University.
Among the participants in the experiment, three working
peoples and two students had their noise exposure increased
by +5 dB, although the position of the CD player and thetype of sound source were the same. A sound level meter(RION, NL-52) was placed at the bedside during the /LWHUDWXUH5HYLHZ
Methods for studying the effects of noise on sleep include
social surveys and laboratory experiments. The former
method provides information on the effects of noise on sleep
in real life, but the extent of the effects is based on

185

International Workshop on Informatics ( IWIN 2022 )experiment, and room background noise and presentationlevels were measured continuously for 10 days. The soundg 4.
level meter used is shown in Fig.

The test sounds used as stimuli were road traffic
noise[RTN], conventional traffic noise[CRN], and aircraft
noise[ACN]. The road traffic noise was a fluctuating noisethat always sounded and was always present during sleep.
Railroad noise includes the sound of six trains running perhour. Aircraft noise includes four takeoff sounds of aircrafts
per hour. Table 1 shows a list of the types of test sounds.

7DEOH6WLPXOLW\SHDQGVRXQGSUHVVXUHOHYHO 63/ CD
SPL
SPL, +5 dB
Type of traffic noiseNo.
(LAeq,1h)(LAeq,1h)142.5 dB
47.5 dB
Road traffic noise2
35.0 dB
40.0 dB(RTN)
3
27.5 dB
32.5 dB
442.5 dB
47.5 dB
Conventional railway
535.0 dB
40.0 dBnoise (CRN)
6
27.5 dB
32.5 dB
742.5 dB
47.5 dB
8
Aircraft noise (ACN)
35.0 dB
40.0 dB
927.5 dB
32.5 dB)LJ$FWLJUDSK *7;VHULHV$0,  3$57,&,3$176
There were 10 working peoples (1 female) with a mean
age of 36.8 years (standard deviation: 15.2 years) and 10
students (2 females) with a mean age of 21.4 years (standard
deviation: 1.1 years).

)LJ'LDJUDPRIWKHH[SHULPHQW 68%-(&7,9(6/((3$66(660(17
In this study, two types of web-based questionnaires wereused in the subjective sleep assessment. One questionnaire
was used to measure usual sleep sensation and sound sleep
sensation. The usual sleep sensation questionnaire is shownin Table 2. The other questionnaire was administered as
early as possible after waking up the next morning so that
subjective sleep sensations would not be forgotten. The
questionnaire for subjective sleep sensation is shown in
Table 3.

)LJ&'SOD\HU :DYH6\VWHPVHULHV%26( 

7DEOH7KHTXHVWLRQQDLUHRQXVXDOVOHHSKDELWQ1
when do you usually enter your bed (bed or futon)
and try to sleep (turn off the light)?
Q2
How much do the hours of lights-out vary from day
to day?
Q3What time do you usually wake up on average?
Q4
How much does your waking time change from day
to day?
Q5
How much sleep do you usually get?
Q6
How much sleep do you get from day to day?
Q7
Do you get enough sleep?
Q8
What is the ideal amount of sleep for you?
Q9
Approximately how long do you usually stay in bed(bed or futon) before falling asleep?
Q10 Are you a good sleeper?
Q11 How many times per night do you wake up during
the night (while sleeping)? (Please consider on)LJ6RXQGOHYHOPHWHU 1/5,21  67,08/,

186

International Workshop on Informatics ( IWIN 2022 )

Q12
Q13
Q14
Q15
Q16
Q17
Q18
Q19

average)
What triggers you to wake up most often? (Waking
up in the middle of the night)
What triggers you to wake up most often? (Waking
up in the morning)
How do you usually feel when you wake up in themorning?
How deep do you usually sleep?
Do you drink alcohol?
How is the environment around your house atnight?
Do you hear any of the following sounds at yourhome at night?
Do you have trouble sleeping due to sound during
your normal sleeping hours?  (Please describe
other sounds that cause you to have trouble
sleeping)  

HSRFK

















6

6

6

:

:

:

6

6





6

6

6

:

:

6

:

6

6

6

6

:

6

:

6

6

6

6

6

:

:

6

6

6

6

6

6

:

6

:

6

:

6

6

ʕ

6

6

6

:

6

6

:

:

6

6

ʹ

0LGVOHHSDZDNHQLQJVʕʕt͗tĂŬĞĚ
^͗^ůĞĞƉŝŶŐ

ʹʹ

)LJ'LDJUDPIRUGHWHUPLQLQJPLGVOHHSDZDNHQLQJV

 5(68/76
We compared the responses of working peoples anduniversity students regarding their "usual sleep habits"(variation in waking and sleeping times, depth of sleep, etc.),
which they were asked to answer of questionnaire shown in
Table 2 (Mann-Whitney's U-test). The results showed that
there are significant differences in Q1 (*p<0.017), Q2(*p<0.019), Q3 (**p<0.007), Q4 (*p<0.041) and Q14(*p<0.013). Thus, it can be said that working peoples went
to bed and woke up earlier and lived more regularly (withless time fluctuation), but university students slept moresoundly.
Using the answers to a self-administered questionnaire
shown in Table 3 (subjective data) and wake/sleep judgment
based on the amount of body movement (objective data),night-related outcomes were compared between the twogroups (t-test). The results showed that there are significant
differences in Q3 (**p<0.006), Q5 (***p<0.001), Q12(**p<0.008), Q15 (*p<0.030) and Q16 (***p<0.001). The
results showed that the working people group was the worse
sleepers in Q.17. On the other hand, comparisons ofobjective data between the two groups showed there are
significant differences in several outcomes as follows: Total
Counts (*p<0.015), Total Minutes in Bed (**p<0.004),
Total Sleep Time (*p<0.014) and Mean Wake Time(**p<0.004). Since the total counts was affected by sleep
duration, it is possible that working peoples had a greater
amount of movement than students. In addition, a significant
difference was found in the mean awakening time for the
subjective sleep evaluation item (Q5). However, although
there was a significant difference in the mean awakening
time between the two groups (2.8 min for working peoples
and 2.2 min for students), considering the number of
awakening events (15.4 times for working peoples and 16.4
times for students), the time spent in the awake state (=number of awakening events x mean awakening time) for
both working peoples and students was 11-12% of total
sleep time (362 min and 331 min). The mean number of
awakenings was 43.1 min and 36.1 min, corresponding to
11-12% of the total sleep time (362 min and 331 min) for
both working peoples and students.
The association between subjective and objective sleep
ratings per night and presentation level LAeq,1h was thentested separately for working peoples and students(Spearman's rank correlation test). The results are shown in
Table 2, which indicates that the RTN participants were
aware that noise exposure had an effect on their sleep state
and arousal. None of the objective evaluation indices
showed significant correlations with the presentation level.
Finally, Fig. 6 shows the relationship between LAeq,1min,
and Wake After Sleep Onset (WASO) for the working

7DEOH7KHTXHVWLRQQDLUHIRUVXEMHFWLYHVOHHSVHQVDWLRQQ1What time did you get up this morning?
Q2
What time did you go to bed last night?
Q3
Were you able to sleep while listening to the CD?
Q4What time did you stop the CD?
Q5
Did you wake up during sleep?
Q6-1 How many times did you wake up last night?
Q6-2 What time did you awaken from sleep?
Q7
How were the sounds heard during theexperiment?
Q8
If you had trouble sleeping, what specifically wasit that disturbed you?
Q9
Did you drink alcohol last night?
Q10
Did you take sleeping pills or cold medicine lastnight?
Q11
Did you sleep better than usual last night?
Q12
Did you sleep better last night than usual?
Q13
Overall, was your sleep good last night?
Q14
Do you feel more tired this morning than usual?
Q15
How was the depth of your sleep last nightcompared to usual?
Q16
How many dreams did you have last night?

 2%-(&7,9(6/((3$66(660(17The actigraph is a measuring instrument that measures theamount of body movement that occurs within a certain
period of time in a series, and judges mid-arousal based on
the occurrence of events that exceed a set threshold value.
Fig. 5 shows the method proposed by Kageyama et al. [5],for judging mid-onset arousal. As shown in the figure, one
or two large body movements are not considered to be midarousal, but only three or more body movements under somerules are considered to be mid-arousal.

187

International Workshop on Informatics ( IWIN 2022 )peoples and students. The correlation coefficients calculated
assuming a simple increase in WASO with respect to onset
level were significant for RTN (r=0.888*) and ACN (r=0.879*) for working peoples and ACN (r=-0.934**) for
students. clearly different from RTN (multiple comparisonsof the difference in correlation coefficients using the
Bonferroni method), but one should be cautious about
negative correlations.

awakenings, it was determined that there was no significant
difference in the average wake time between university
students and working peoples, which was equivalent to 1112% of total sleep time (43.1 minutes for working peoples
and 36.1 minutes for university students).
In addition, a comparison of subjective and objective sleep
evaluations revealed a significant correlation between
railroad noise and aircraft noise for working peoples in thequestionnaire item about whether they were able to fallasleep while listening to CDs. In addition, significant
correlations were found between the two groups of working
peoples and university students in the questionnaire item ofwhether they were awakened during sleep and thepresentation level. Thus, it was clear that the subjects were
aware of being awakened during noise exposure. No
significant correlation was found between the objective
sleep evaluation and the presentation level.
Road traffic noise, railroad noise, and aircraft noise had
different correlations with each other. Road traffic noise waspositively correlated, railroad noise was uncorrelated, andaircraft noise was negatively correlated.
Comparison of subjective and objective sleep evaluations
between university students and working peoples revealed
that middle-aged and young working peoples had different
results in terms of sleep onset and depth of sleep. These
results were judged to be reasonable because they wereconsidered to be due to the decline in physiological
functions caused by aging, rather than to the effects of theworking-age population.

7DEOH&RUUHODWLRQFRHIILFLHQWEHWZHHQHDFKVXEMHFWLYHVOHHSDVVHVVPHQWDQGVRXQGVOHYHOL$HTK
RTN
CRN
ACN
0.503**
0.351
0.264
Q3
0.033
0.305
0.247
0.218
0.295
0.389*
Q5
0.039
0.413*
0.433*
0.294
0.237
0.257
Q12
-0.065
0.130
0.228
0.124
0.243
0.349
Q15
-0.159
0.151
0.296
0.152
0.116
0.167
Q16
-0.206-0.042-0.120Note: *p<0.05, **p<0.01, where p-value indicates the significanceprobability of correlation analysis. The upper value in each column iscorrelation coefficient for working people, and the lower one is that for
students.

5()(5(1&(6[1] World Health Organization Regional Office for Europe,
Environmental Noise Guidelines for the European
Region (2018).
[2] K. Nagai, F. Koyama, M. Ueda, H. Kasuga, and M.
Hiroe, Sleep effects on noise – Follow-up test on sleep
disturbance caused by traffic noise, 2020 Spring
Meeting Acoustical Society of Japan, pp.593-594 (inJapanese).
[3] K. Enomoto, K. Nagai, F. Koyama, M. Ueda, H.
Kasuga, and M. Hiroe, Sleep effects on noise part 2 –Discussions on the results of sleep analysis byactigraphy, 2020 Autumn Meeting Acoustical Societyof Japan, pp.373-374 (in Japanese).
[4] J. Kaku, M. Hiroe, S. Kuwano, and S. Namba, Sleep
disturbance by traffic noise, Jornal of sound andvibration, Vol.277, pp.459-464 (2004).
[5] T. Kageyama, T. Kurokawa and M. Kabuto, An
epidemiologic study on insomnia in metropolitan areas
IV: a relationship between incidence of waking
estimated by actimetry and nighttime road traffic noise,
Proceedings of the autumn meeting, the Institute of
Noise Control Engineering of Japan, pp.65-68, 1997
(in Japanese).

)LJ 'RVHUHVSRQVH UHODWLRQVKLS EHWZHHQ 63/
L$HTPLQDQGLQFLGHQFHRIPLGVOHHSDZDNHQLQJV

 ',6&866,21
From the results of this experiment, 10 working peoples
and 10 university students with different sleep habits were
exposed to traffic noise while sleeping, and their sleep states
were evaluated by both subjective and objective sleep
assessments. Subjective sleep evaluations revealed thatworking peoples were more likely to have regular sleep
habits, with less fluctuation in their sleeping and waking
times. However, the working peoples were found to havepoorer sleep conditions. The objective sleep evaluation
confirmed a significant difference in the average wake timeof working peoples, but considering the number of

188

International Workshop on Informatics ( IWIN 2022 )$6WXG\RQ$SSOLFDWLRQRI=HUR7UXVW$UFKLWHFWXUHWR,R7$FWXDWRUVNobuhiro KOBAYASHI**

Department of Information Security, University of Nagasaki, Japannkoba@sun.ac.jp$EVWUDFW -In recent years, the use of IoT systems in society
has been progressing. As an example of the Society 5.0 thatJapan is aiming for, it has been shown that information fromsensors in physical space will be accumulated (big data)
through IoT, analyzed by artificial intelligence (AI), andfeedback to physical space to provide high-value-added
information, suggestions, and control of devices. On theother hand, if cyber attacks on IoT systems cause service
outages, unauthorized operations, or other problems, it could
have a serious impact on our daily lives and economic andsocial activities. Therefore, in this paper, we especiallyconsider security in IoT actuators that exert physical action
as feedback from cyberspace to physical space. As a resultof the study, we also present the Zero Trust IoT Security
Framework, which applies the Zero Trust architecture to IoT
actuators.

$ 7UDQVGXFHUIXQFWLRQ
It interacts with the physical world and acts as an edge
between the digital and physical environments. All IoT
devices have at least one of two types of transducer
functions.
6HQVLQJIXQFWLRQ
Ability to measure and provide data on the physical world([DPSOHV temperature, optical sensing, audio sensing,radar$FWXDWRUIXQFWLRQV
Ability to create change in the physical world([DPSOHV heating coils, speakers, electronic door locks,
drone operations, servo motors, robot arms

.H\ZRUGV: IoT System, IoT Actuator, Cyber Security, Zero
Trust Architecture, CPS% ,QWHUIDFH)XQFWLRQV
Enable device-to-device interactions (device-to-device,device-to-human, etc.).

 ,1752'8&7,21

$SSOLFDWLRQLQWHUIDFH
Application programming interface, etc.

In recent years, the use of IoT systems in society has been
progressing. Society 5.0, which Japan is aiming for, isdefined as "a human-centered society that achieves both
economic development and solutions to social issues
through a system that highly integrates cyberspace andphysical space (the real world). As an example, informationis collected from sensors in the physical space through IoT(big data), analyzed by artificial intelligence (AI), and fed
back to the physical space with high-value-added
information, proposals, and control of devices. On the other
hand, if cyber attacks on IoT systems cause service outages,
unauthorized operations, or other problems, our daily lives
and economic and social activities may be seriously affected[1][2].
In this paper, we discuss, in particular, the results of ourconsideration of security in IoT actuators that exert physicalaction as feedback from cyberspace to physical space [3][4][5][6][7]. Future directions for security measures in IoTactuators will also be presented.

+XPDQXVHULQWHUIDFH
Functions for direct communication between IoT
devices and humans([DPSOHV Touch screen, microphone, camera, speaker
1HWZRUNLQWHUIDFH
Ability of IoT devices to use communication networks([DPSOH Ethernet, Wi-Fi, LTE, ZigBee, etc.
& 6XSSRUW)XQFWLRQV
The ability to support other IoT functions.
Examples: device management, cyber security features,privacy features
In this paper, A) IoT devices with actuator functions astransducer functions will be referred to as ,R7DFWXDWRUV.
IoT actuators are capable of influencing the physical world,which may lead to cyber attacks that may threaten humansafety and life, damage and destroy equipment and facilities,
and cause major disruptions such as the shutdown of criticalservices like social infrastructure. In Smart Cities forSociety 5.0, a large number of IoT actuators will be
deployed and utilized, and it is important to ensure theirsecurity.

 $%287,27$&78$7256
The total number of IoT devices worldwide is expected toreach approximately 35 billion by 2022. In terms of sectors,"medical," "industrial," "consumer," and "automotive andaerospace" are expected to show high growth, replacing"telecommunications," which has been increasing [8].
The following features of this IoT device have a different
impact on cybersecurity and privacy risks than traditional IT
devices [4].

 6&23(2)7+(678'<

191

International Workshop on Informatics ( IWIN 2022 )Since IoT systems are a fusion of IT and OT [4],cybersecurity issues related to IT are encompassed by issues
in IoT systems. An example is shown below.

(Japan's Ministry of Land, Infrastructure, Transport
and Tourism ensures safety through its vehicle
inspection system [10].)
࣭ Vendors tend to pursue planned obsolescence ofdevices to maximize profits through continued sales
rather than maintaining existing devices
࣭ Bankruptcy or dissolution of the company ororganization that managed the IoT device

࣭ Implement countermeasures for known
vulnerabilities
࣭ Consideration of countermeasures based on threat
analysis and risk assessment
࣭ Avoidance and detection of bugs and lack offunctionality during implementation
࣭ Communication security(FW, IPS/IDS, encryption, etc.)
࣭ Implementing authentication and access control
࣭ Protect and update credentials and trust anchors࣭ Logging and analysis

The risks associated with these conditions, characteristics,
and point out need to be considered for IoT systems,
including IoT actuators.

 ',5(&7,212)6(&85,7<0($685(6,1,27$&78$7256

The research on securing IT systems that has been
conducted over the years has accumulated results and can beutilized in IoT systems, and thus is excluded fromconsideration in this paper, except for "authentication andaccess control," which is discussed below [4]. In cases
where utilization is difficult due to limitations of individual
IoT devices (e.g., computing resources), it is necessary to
conduct a separate study. On the other hand, this paper will
focus on the following items [4], which are listed ascharacteristics of IoT systems.
a)
b)
c)
d)
e)
f)

 2XWSXW$FFHVV&RQWUROWR3K\VLFDO6SDFH
IoT systems are diverse, and it has been noted that evenwhen the IoT devices used and system configurations are
similar, the security measures required differ depending on
the purpose and application of the IoT system [11]. Prior
research on security countermeasures for output to physicalspace has focused on threats to loudspeakers, which areconsidered one of the IoT actuators. In that research, as acountermeasure against analog signal threats to sensors, theCyber-Physical Firewall (CPFW) framework, which detects
and regulates the analog signal output of the CPS using amechanism similar to a firewall, has been proposed [12]. It
has been shown that CPFW is effective as a countermeasureagainst threats posed by sound signals by controlling output
access to the digital signal output of the controller with
CPFW, and outputting only secure digital signal output fromthe output interface after digital-to-analog conversion. For
sound signals, the report also cites the wide variety of
threats presented among analog signals and formulates
access control policies for known attacks as a case study
evaluation.
On the other hand, the report states that, unlike
loudspeakers, the case of motors and other devices thatcause changes in the environment due to motion isconsidered a future issue, and furthermore, secondary effects
resulting from actuators require a mechanical engineering
approach.
With reference to the above, it is considered effective forsecurity measures in IoT actuators to introduce policy-based
access control mechanisms in the course of signaltransmission between the controller and the output interface.
If the policy formulation is to be specific to individual
threats, consideration should be given to increasing thenumber of policies and ensuring real-time performance.
In addition, the relationship between the information(ATTRIBUTE) that can be extracted from the analog signal
and the threat signal is considered to differ depending on thetype of actuator signal other than the sound signal, and thisrelationship needs to be clarified.
Furthermore, even if access control of output interfaces
could regulate threat signals, there is concern that secondary
effects could lead to more serious situations, so it is also

The scope and impact of the threat can be very broad
and large.
The demand for IoT systems, especially in operationand maintenance, is more than 10 years.
Monitoring and managing IoT devices can be very
difficult. There may be unmanaged IoT devices.
IoT devices may have difficulty being fully aware ofeach other's environment
IoT devices have limited functionality andperformance.
Potential connection of IoT systems not envisionedby the developer

These characteristics raise the following concerns: b)vulnerabilities discovered after the IoT device is shipped
may affect the response to vulnerabilities; c) vulnerabilities
may be left unpatched and un-updated; e) patching may bedifficult due to IoT device resources; f) the IoT device may
be vulnerable to security risks; and g) the IoT device may bevulnerable to security risks. e), there may be situations
where it is difficult to apply patches to IoT devices due toresource constraints. With regard to the provision of thesepatches and updates, the following points have been raised
from an economic point of view [9].
࣭ Currently, device vendors and manufacturers havelittle financial incentive to ensure continuous
upgrading of patches for IoT
࣭ Maintenance of IoT devices can reduce revenues
because company revenues come from device sales,
not maintenance࣭ Vendors are not legally responsible for the ongoing
maintenance of the device after the initial sale

192

International Workshop on Informatics ( IWIN 2022 )necessary to ensure consistency with high-level policies interms of service continuity for IoT systems.

6. Dynamic authentication and authorization of all
resources, strictly enforced before access is granted
7. Gather as much information as possible about thecurrent state of assets, network infrastructure, andcommunications and use it to improve security posture &RQWLQXRXV$FFHVV&RQWUROEDVHGRQ
'\QDPLF3ROLFLHV

Behavioral attributes, shown in 4. above, include analysisof the accessing subject and deviations from observed usage
patterns. Environmental attributes include the timing ofaccess and detection of currently active attacks. Since these
attributes are affected by changes in time, history, andcircumstances, access is determined by dynamically
applying a policy, a set of access rules that take them into
account.
Figure 2 is a conceptual framework model showing the
logical components that comprise a zero-trust architecture
and the basic relationships among their interactions.

Today's IT systems operate in complex network
environments, including integration with cloud services andthe use of terminals in mobile environments for remote work.
Such complexity makes it difficult to define a single
boundary between the inside and outside of an organization,
leading to the compromise of conventional perimeter
defense security measures. Therefore, a new concept known
as "Zero Trust" is currently attracting attention, and acybersecurity architecture called "Zero Trust Architecture"
based on it [7]=HUR7UXVW=HUR7UXVW$UFKLWHFWXUH
Zero Trust focuses on the protection of resources and isbased on the premise that trust is never implicitly granted,
but must be continually assessed. This is because it assumes
that attackers exist inside the perimeter and are just as
untrustworthy as those outside. It is also assumed that
resources are not limited to data, but include computational
resources and IoT actuators. And to reduce cyber attacks byattackers that are difficult to eliminate completely, the focus
is on authentication, authorization, and reduction of implicit
trust zones to maintain availability and minimize the timedelay of authentication mechanisms. Access control rules
are supposed to be as granular as possible in order to enforce
the minimum privileges necessary to execute a request. A
conceptual diagram of zero-trust access is shown in Figure 1.

)LJXUH&RUH=HUR7UXVW/RJLFDO&RPSRQHQWV>@
In the above figure, the PDP is decomposed into two
logical components: the PE (policy engine) and the PA(policy administrator).
z PE (Policy Engine)
The final decision to grant access to a subject's resources
is made by the Trust Algorithm (TA). Use external sources(CDM System, Threat Intelligence, etc.) and policies as
input to the Trust Algorithm to control access to resources.
In the United States, NIST is developing the Policy
Machine [13] as a reference implementation of NGAC
(Next Generation Access Control) [14], a standard for
ABAC (attribute-based access control) based onrelationships among data elements, with the goal of
supporting a wide variety of access control policies [15].

)LJXUH&RQFHSWXDO'LDJUDPRI=HUR7UXVW$FFHVVWhen a subject accesses a resource, it will go through a
PDP (Policy Decision Point) and a corresponding PEP(Policy Enforcement Point). However, no additional policies
can be applied beyond the PDP/PEP. Therefore, the implicit
trust zone should be as small as possible. The basic tenets ofZero Trust are as follows.

z PA (Policy Administrator)
The PA works closely with the PE and relies on the PE to
decide whether to allow or deny access; the PA causes the
PEP to establish or block communication paths between
entities and resources based on the PE's decision.

1. Consider all data sources and computing services as
resources
2. Protects all communications regardless of network
location
3. Access to resources is granted on a per session basis
4. Access to resources is determined by a dynamic policythat includes client ID, application service, state of therequesting asset, and other behavioral andenvironmental attributes
5. Monitor and measure the integrity and security
behavior of all assets

z PEP (Policy Enforcement Point)
The PEP works with the PA and receives policy updates
from the PA. It then applies the received policy to establish,
block, and monitor the communication path between thesubject and the resource.
z TA (Trust Algorithm)

193

International Workshop on Informatics ( IWIN 2022 )The thought process in determining access permissions in
PE is considered TA. This thought process is grouped into
the following categories.
A)
B)
C)
D)
E)

Access request
Subject database and history
Asset database
Resource policy requirements
Threat intelligence and logging

In addition, TA implementation methods are distinguishedby two main features.
1.
2.

)LJXUH([DPSOHRIDSRVVLEOH,R7DFWXDWRU
GHYLFHFRQILJXUDWLRQ

Criteria-based (binary)/score-based (weighted)
Unitary (individual request) / Contextual (history
consideration)

As shown in the figure above, an IoT device corresponds
to a controller. A motor unit consisting of a motor driver anda motor corresponds to an actuator. As a security measure
for IoT actuators, it is considered effective to introduce a
policy-based access control mechanism during the signaltransmission between the controller and the output interface.
Therefore, we will consider implementing a function
equivalent to CPFW in IoT devices.

Contextual score-based TA is thought to provide dynamic
and fine-grained access control.

 $33/,&$7,21 2) =(52 75867
$5&+,7(&785(72,27$&78$7256
In an IoT system, IoT actuators provide various added
values by cooperating with IoT services on the cloud. In thiscase, the application running on the IoT actuator controls theactuator in the physical space in response to requests fromthe IoT service. Note that the IoT actuator may have anothertransducer function, the sensing function. A schematic of an
IoT actuator and IoT system is shown in Figure 3.

)LJXUH([DPSOHRIDF\EHUDWWDFNDJDLQVW
DQ,R7DFWXDWRU
Malware that attacks IoT actuators is assumed to infectapplications (software) running on IoT devices and launch
cyberattacks against actuators. However, as pointed out in
Section 3, it is not always possible to update the vulnerable
programs as soon as possible. However, as pointed out in
Section 3, there are cases where the provision of programs
and patches for updates are delayed or not provided.
Furthermore, in recent years, zero-day attacks, in whichsoftware vulnerabilities are discovered and exploited before
patches or workarounds are released, have become a serious
problem [17].
Therefore, in order to prevent malware-infected
applications from controlling unauthorized actuators, afunction equivalent to CPFW is considered based on thezero-trust concept. First, PDP and PEP in in the zero-trustarchitecture, PDP is placed on the cloud side and PEP isplaced on the IoT actuator side. This PEP will act as a
CPFW to prevent unauthorized control. In addition, the PDP
can work with IoT services to obtain behavioral andenvironmental attributes of the subject application. This

)LJXUH6FKHPDWLFRI,R7DFWXDWRUDQG,R7V\VWHPOne of the most common types of actuators in the physicalspace is the motor. A motor is a device that uses electric
power to generate physical rotational motion. This rotationalmotion can be used to drive a pump in a river disasterprevention system or a water supply system for paddy fields,
or to rotate the propeller of a drone or the wheels of an
automatic delivery robot. Figure 4 shows an example of thedevice configuration of the IoT actuator with motor
envisioned in this paper [16].

194International Workshop on Informatics ( IWIN 2022 )allows the PEP to work with the PDP to provide
instrumental access control based on dynamic policies. A
schematic of the application of Zero Trust Architecture in an
IoT actuator is shown in Figure 6.

)LJXUH3URWHFWLQJ3(3ZLWK6HFXUH:RUOGThe above configuration makes it possible to prevent
unauthorized control of actuators by applications in which
the PEP is infected with malware. It is also possible toprevent the PEP itself from being infected with malware.
However, there is a risk of a denial of service attack (DoS
attack) in which a malware-completed app interferes withthe coordination between the PEP and PDP. In addition,
there is a risk that IoT actuators will not provide the services
they are supposed to fulfill because normal apps will not
control the actuators as they should.
Figure 9 shows an example of a DoS attack on an IoTactuator. Since IoT actuators are expected to be used inapplications where the availability of object motion isimportant, countermeasures against such DoS attacks arenecessary.

)LJXUH$SSOLFDWLRQRI=7$LQ,R7DFWXDWRU
However, there is concern that PEPs added to IoT
actuators will become a new target for malware attacks:even if PDPs are protected by cloud-side security measures,without proper access controls being enforced by PEPs,
attacks on the actuators could be carried out and damage tothe physical space.

)LJXUH([DPSOHRIDF\EHUDWWDFNDJDLQVW3(3Therefore, we focus on TEE (Trusted Execution
Environment) [18], which is a hardware security feature of
SoC (System on Chip), a kind of semiconductor chip that isbeing adopted in IoT devices such as IoT actuators.
TEE is an isolated execution environment independent ofapps and OSs, and the Secure World in the isolated
execution environment is not affected by the Normal Worldin which apps and OSs are executed. Therefore, PEP can beplaced inside Secure World to protect it from malware andother attacks. Examples of hardware that provide TEE
include Arm TrustZone [19], RISC-V Keystone [20], Intel
SGX [21][22], etc. A schematic of PEP protection by Secure
World is shown in Figure 8.

)LJXUH([DPSOHRIF\EHUDWWDFNFDXVLQJ'R6
Accordingly, Self PDP, which is a subset of PDP, isintroduced to the IoT actuator in preparation for the situation
where the PDP on the cloud side and the PEP in the IoT
actuator cannot be linked. However, since the input to the
PE of Self PDP is limited to within the IoT actuator, theprocess by the TA is limited. On the other hand, thecommunication resources required for the linkage between
PEP and Self PDP are no longer necessary, and real-time
performance and availability are expected to be improved.
In addition, in order to mitigate secondary damage to IoT
actuators caused by normal applications not controlling the195International Workshop on Informatics ( IWIN 2022 )actuators as they should, applications with Failsafe
functionality will be introduced to IoT actuators. For
example, provide a function to control the actuator to stopsafely while it is running, such as by gradually decreasing
the motor speed. In addition, as an application of theFailsafe function, it is also possible to realize EOL (End ofLife) for IoT devices. The management of EOL by the
Failsafe function is an advantage in preventing unmanaged
IoT devices from being abused by DDoS attack bots, etc.,
which are related to the characteristics b) and c) of the IoT
system described in Section 3. In addition, as acountermeasure against malware attacks, Self PDP and
Failsafe AP must be placed in the Secure World in the samemanner as PEP.
As a result of the final consideration, the relationshipbetween the components that apply the Zero Trustarchitecture to IoT actuators is shown in Figure 10 as the“Zero Trust IoT Security Framework”.

and Information Security (SCIS 2018), 4E2-1, January
2018
[3]
RON
ROSS,
MICHAELMcEVILLEY,
JANETCARRIER OREN, "NIST Special Publication
800-160 Systems Security Engineering," NIST
National Institute of Standards and Technology U.S.
Department of Commerce, November, 2016,
https://doi.org/10.6028/NIST.SP.800-160v1[4] Katie Boeckl, Michael Fagan, William Fisher, Naomi
Lefkovitz, Katerina N. Megas, Ellen Nadeau, Danna
Gabel O'Rourke, Ben Piccarreta, Karen Scarfone,"NISTIR 8228 Considerations for Managing Internetof Things (IoT) Cybersecurity and Privacy Risks,"
NIST National Institute of Standards and Technology
U.S. Department of Commerce, June, 2019,https://doi.org/10.6028/NIST.IR.8228[5] Michael Fagan, Katerina N. Megas, Karen Scarfone,
Matthew Smith, "NISTIR 8259 Foundational
Cybersecurity Activities for IoT Device Manufacturers", NIST National Institute of Standards andTechnology U.S. Department of Commerce, May,
2020, https://doi.org/10.6028/NIST.IR.8259[6] Michael Fagan, Katerina N. Megas, Karen Scarfone,
Matthew Smith, "NISTIR 8259A IoT Device
Manufacturers Cybersecurity Capability Core Baseline,
NIST National Institute of Standards and Technology
U.S. Department of Commerce, May, 2020,
https://doi.org/10.6028/NIST.IR.8259A[7] Scott Rose, Oliver Borchert, Stu Mitchell, Sean
Connelly, "NIST Special Publication 800-207 Zero
Trust Architecture," NIST National Institute ofStandards and Technology U.S. Department ofCommerce,
August,2020,
https://doi.org/10.6028/NIST.SP.800-207[8] "White Paper on Information and Communications,2020 Edition," Ministry of Internal Affairs andCommunications,
August2020.
https://www.soumu.go.jp/johotsusintokei/whitepaper/j
a/r02/pdf/index.html[9] George Corser, Glenn A. Fink, Mohammed Aledhari,
Jared Bielby, Rajesh Nighot, Sukanya Mandal,
Nagender Aneja, Chris Hrivnak, Lucian Cristache,Internet of Things (IOT) Security Best Practices:
whitepaper, IEEE, 2017,https://standards.ieee.org/wpcontent/uploads/import/documents/other/whitepaperinternet-of-things-2017-dh-v1.pdf[10] [Attachment 1] Permission System for Specified
Modification, etc. of Vehicles (Outline), Ministry ofLand, Infrastructure, Transport and Tourism, August
2020,
https://www.mlit.go.jp/report/press/content/001358067.pdf[11] IoT Security Guideline ver 1.0, IoT Promotion
Consortium, Ministry of Internal Affairs andCommunications, Ministry of Economy, Trade andIndustry,
July,
2016,
https://www.soumu.go.jp/main_content/000428393.pdf[12] R. Iijima , T. Takehisa, and T. Mori, "Proposal andVerification of a Security Framework to Detect and)LJXUH=HUR7UXVW,R76HFXULW\)UDPHZRUN &21&/86,21
This paper discussed, in particular, the results of a study of
security in IoT actuators that exert physical action as
feedback from cyberspace to physical space. The paper alsopresents the direction of future security measures in IoTactuators. Then, he proposed the "Zero Trust IoT Security
Framework," which applies the Zero Trust architecture to
IoT actuators. In the future, we plan to work on the detailed
design of the "Zero Trust IoT Security Framework",
implement it, and verify its effectiveness.

5()(5(1&(6[1] "Next Cyber Security Strategy (Draft)", Cabinet Cyber
Security Strategy Headquarters, September 27, 2021,
https://www.nisc.go.jp/conference/cs/dai31/pdf/31shir
you01.pdf[2] Zen Ishikura, Hideo Yamamoto, Takemi Nisase,
Fumihiko Magata, "Considerations on Security Designof IoT Systems," 2018 Symposium on Cryptography

196

International Workshop on Informatics ( IWIN 2022 )Regulate Threats by Analog Signals," Proc. ofComputer Security Symposium 2021, pp. 79 - 86, Oct.
19, 2021.
[13] David Ferraiolo, Serban Gavrila, Wayne Jansen,"NISTIR 7987 Revision 1 Policy Machine: Features,Architecture, and Specification," NIST National
Institute of Standards and Technology U.S.
Department
ofCommerce,
Oct.
2015,
https://doi.org/10.6028/NIST.IR.7987r1[14] INCITS 565-2020 - Information technology - Next
Generation Access Control, Information Technology
Industry Council, <ref> August 19, 2022, [15]https://standards.incits.org/apps/group_public/project/d
etails.php?project_id=2328[15]
PolicyMachineCore,
GitHub,https://github.com/PM-Master/policy-machine-core[16] Overview and Features of Stepping Motors, Oriental
Motor
Co,
https://www.orientalmotor.co.jp/products/stepping/ove
rview_1/[17] Ten Major Information Security Threats 2022, IPA,
July 2022, https://www.ipa.go.jp/files/000096258.pdf[18] Ariyasu Suzaki, Akira Tsukamoto, Kazumoto Kojima,
Kenta Nakajima, Hongtron Tuk, Akira Shiio: TEE
Comparison, Symposium on Cryptography andInformation Security (SCIS) 2020 (2019).
[19] S. Pinto and N. Santos, “Demystifying Arm
TrustZone: A Comprehensive Survey,” ACM
Computing Surveys (CSUR), vol. 51, no. 6, 2019[20] D. Lee, D. Kohlbrenner, S. Shinde, D. Song and K.
Asanovi´c: “Keystone: A framework for architectingtees”, arXiv (2019).
[21] V. Costan, I. Lebedev, and S. Devadas, “Secure
Processors Part I: Background, Taxonomy for Secure
Enclaves and Intel SGX architecture,” Foundations andTrends in Electronic Design Automation, vol. 11, no.
1-2, pp. 1-248, 2017.
[22] V. Costan, I. Lebedev, and S. Devadas, “Secure
Processors Part II: Intel SGX Security Analysis and
MIT Sanctum Architecture,” Foundations and Trends
in Electronic Design Automation, vol. 11, no. 3, pp.
249-361, 2017.

197

International Workshop on Informatics ( IWIN 2022 )Implementation on the Evaluation Platform for a Large-Scale Building Automation andControl System
Kohei Miyazawa*, Tetsuya Yokotani**, Hiroaki Mukai****Electrical and Electronic Science Engineering, Kanazawa Institute of Technology, Japan
c6100786@planet.kanazawa-it.ac.jp**
Engineering department, Kanazawa Institute of Technology, Japan
yokotani@neptune.kanazawa-it.ac.jp***
Engineering department, Kanazawa Institute of Technology, Japan
mukai.hiroaki@neptune.kanazawa-it.ac.jp

Abstract—In this paper, we propose an evaluation platform
based on a building automation and control system (BACS)
using a building automation control network (BACnet)
protocol. A BACnet connects electrical equipment with amonitoring center to manage and control multiple electricaldevices in a building centrally. Although a BACnet has
recently been widely deployed as a consensus solution, somesecurity concerns have been identified for implementation
expanded to the IP base protocol, that is, BACnet/IP. For
example, unauthorized remote access to control systems forelectrical equipment and denial of service attacks may lead toserious accidents, particularly in large buildings. Therefore,
security threats are typically detected by monitoring traffic on
BACnet/IP without complicated functions on the network
endpoints. However, investigating the detection mechanisms
in real building systems is relatively difficult. Therefore, we
propose an emulated evaluation platform for personal
computer (PC) systems. In this study, we clarify therequirements of a BACS, including the latest trends in the useof the platform in buildings, and describe an implementationof the proposed platform by PC clustering, which supports up
to 100,000 endpoints. Diagram of the configuration of theevaluation platform was designed. Emulators were
distributed at each level of the hierarchy to unify the platform.
The monitoring center reads the status of all locations in oneminute cycles to analyze the network traffic at the maximum
load. Finally, we show that the proposed platform can verify
the performance with the maximum network load without
using a real building environment.

As the demand for solar power generation and other electric
power facilities increases to achieve higher energy efficiency
in the future, the number of DC power feeders adopted is
expected to rise, and equipment related to these facilities will
also be controlled via a building automation control network(BACnet) [1]. However, when a BACnet is designed, it isalways recommended that BACnet devices be installed in a
separate segmented network. With the recent rise in theInternet of things (IoT), building automation system (BAS)
networks are now connected to the Internet for management
coordination [2]. Interconnectivity between networks
facilitates attacks between networks. When DC power supply
equipment, which often handles high voltages, is attacked insuch cases, it is suggested that arcing may occur in theswitchgear, leading to a fire accident, and the damage if suchan incident occurs in a large building is immeasurable.
However, currently, there is little awareness of the threat
posed by external attacks on DC power feeders. Moreover,
BACnet is used in small- and medium-sized buildings,
commercial facilities, airport systems, and large buildings to
perform control and management tasks in a single package,thereby enabling long-term operations. As previously
mentioned, it is necessary to evaluate the impact of an attack.
However, if the evaluation is conducted in an actual
environment using the electrical facilities of a large-scalebuilding, it is necessary to install electrical facilities that are
adapted to the assumed building environment, which can be
costly [3].
Thus far, the authors have focused their research only on thephysical impact of external unauthorized attacks via a
BACnet [4]. In this study, the previous evaluation platform
was modified from AC to DC, and the temperature
characteristics of the switchgear under a DC power supply
were verified. Based on the results, we design and emulate
the logical configuration of a large-scale building with
100,000 monitored points to respond to identified threats. The
emulated building is distributed at each level of the hierarchy
to unify the platform, and the network traffic under the
maximum load environment is investigated and analyzed.

Keywords: BACnet, Building automation and control system(BACS), Cyber physical system, Cyber physical security

 ,QWURGXFWLRQ

In recent years, communication protocols for building
management systems have become more open. particularly acommunication protocol called the building automation andcontrol system (BACS), which became an ISO standard in2003 as an international data communication protocol, hasbeen applied to the automatic control of equipment inbuildings. It has been applied to the automatic control offacilities in buildings. It is connected to various facilities,
such as air conditioning, electric power, lighting, crime
prevention, and disaster prevention, contributing to acomfortable environment for users. Additionally, as it can
automatically and centrally manage equipment from different
vendors, it contributes to improved convenience and safety.

 2YHUYLHZRID%$&QHW,3

A BACnet is a communication standard for building
networks that became an ISO standard in 1995, similar to
ANSI ASHRAE Standard 135-1995. The BACnet Operator
Workstation (B-OWS), a central monitoring device,interconnects building facilities through a common interface

199

International Workshop on Informatics ( IWIN 2022 )called the BACnet device to monitor and control the building[5].

An accumulator is used to monitor the total amount ofelectricity. These objects have a present-value property, andchanges in that value allow the device to physically control
and monitor its state. As these objects have a physical effecton the real environment when abnormal values are assigned
to them, it is necessary to be aware that an unauthorized attack
can pose a serious threat.
Table.1 Typical BACnet Objects

ObjectAnalog Input
Analog Output
Binary Input
Binary Output
Multi-state Input
Multi-state Output
Accumulator

Fig.1 BAS configuration

Property
Present Value
Present Value
Present Value
Present Value
Present Value
Present Value
Present Value

%$&QHWVHUYLFH

A BACnet/IP is a technology in which the upper protocolsof the BACnet standard are implemented on an IP network,
as shown in Fig.1. This implies that various BACnet-enabled
devices are connected to the Ethernet, which is commonlyused in all facilities. By using existing networks,interoperability between systems and the cost of configuring
network infrastructure can be reduced. The Category 5
cabling that a BACnet/IP uses for communications isavailable in virtually all areas of the building. All of thesefactors reduce costs in terms of initial deployment andongoing maintenance and management of the BAS network.
However, it is necessary to understand the problems
associated with a BACnet/IP. One particularly critical issue
is security: a BACnet/IP is a standardized, interoperable
network and is therefore not suitable for use on unprotected
networks. Therefore, the defense of building management
system networks with BACnet/IP in IP networks is critical.

Access to objects owned by BACnet devices is provided byservice requests. Various services are implemented, and themain ones used are those for creating and deleting objects(created object, deleted object), reading and writing
properties individually, and multiple properties at once (readproperty, write property), and receiving and sending change
notifications (change of value). There are equally services forreceiving and sending change notifications (change in value),etc. [8].
The number of devices and objects increases as the numberof buildings increases. When monitoring the status ofcontrollers at regular intervals, the amount of data to be readincreases, and the monitoring equipment may become
overloaded.

 (YDOXDWLRQRIWKHSK\VLFDOSKHQRPHQD

Many electrical devices in building facilities operate using
direct current (DC) power. However, current power
distribution facilities transmit power in AC, which must be
converted to DC when an electrical equipment is in operation,
resulting in energy loss. Eliminating the need to convert DC
to AC reduces energy loss. As an IT equipment that requires
a continuous power supply becomes more established, theamount of power consumed increases, and power
consumption needs to be reduced [9]. To solve these issues,energy savings by DC power supply are being promoted. Inline with this trend, an increasing number of energy-related
equipment, such as photovoltaic power generation equipment,is being incorporated as a part of construction facilities.
Consequently, the amount of photovoltaic power generation
installed has increased remarkably over the past 9 years. Fig.2
shows the cumulative power generation from renewableenergy sources in Japan (excluding large-scale hydropower)[10]. The installed capacity of solar power generation has
increased by approximately eightfold to 74.7 million kW over
9 years from FY2013 to FY2021. This indicates that thedemand for the DC power feed is high and is expected toincrease further in the future.

%$&QHWREMHFWA BACnet represents the target data as objects, which is an
abstract concept, and models the facilities connected to the
network as a collection of objects, which are classified into
basic input/output, device characteristics, notification
functions, life and safety, complex functions, file informationexchange, and others. BACnet objects are classified into
basic input/output, device characteristics, notification
function, life and life safety, complex functions, fileinformation exchange, and others. The ASHRE 135-2012
BACnet standard specifies 54 types of BACnet objects. Eachobject defines several attribute values, called properties, tofurther define the characteristics of that object in detail [6][7].
Table 1 shows the typical objects in the BAS handled in thisstudy: analog input and analog output are used to monitor
analog value information, such as room temperature and set
temperature (the former for monitoring, and the latter forcontrol). Multi-state input and output are used to control andmonitor multiple states, such as the operating mode andairflow rate.

200

International Workshop on Informatics ( IWIN 2022 )the electricity generated by solar panels will be used as direct
current, and the electrical equipment at the end of the line will
be controlled via BACnet.

 '&)HG%$&QHWHYDOXDWRU

In many existing buildings, electrical equipment is operated
by AC power transmission, and the evaluation platform
created last year was designed to evaluate the temperature risecharacteristics by operating the electrical equipment with an
AC power supply. However, as mentioned above, it isexpected that DC will be used to supply power to electricalequipment in the future instead of inefficient AC. To
accommodate this, the evaluation platform of the previous
year should be improved.
The BACnet controller is implemented using a
programmable logic controller and synchronized with theswitchgear, which is operated by sending high-frequency
on/off signals to the switchgear at regular intervals via
BACnet communication from the B-OWS. In this case, theshort-circuit and temperature rise characteristics owing to thewelding of the contact points when an arc discharge occurs inthe switchgear operated by the DC power supply areevaluated.

Fig.2 Cumulative power generation from renewable energy
facilities

+D]DUGVRI'&SRZHUVXSSO\Fig.3 shows the fire process for a hypothetical DC feed [11].

 'HVLJQ DQG LPSURYHPHQW RI D %$&QHWHYDOXDWLRQPDFKLQH

Fig.4 shows the configuration of the BACnet demonstration
unit designed and modified in this study. The areas shown in
red were changed to support the DC power supply. Figs.5 and6 show the transformer, rectifier, and resistors.
The existing system was designed to measure highfrequency cutoff temperature characteristics at low AC
voltages, and it was modified for testing at 280 V DC. The
following areas were focused on when improving the existing
demonstrator:
a. Selecting electromagnetic switches and breakers
compatible with high voltages and replacing them withexisting parts.
b. Modifying the electromagnetic switchgear by installinga transformer to allow the electromagnetic switchgear to draw
approximately 280 V.
c. Installing a rectifier and replacing the existing parts withnew ones.
d. Providing a rectifier to allow the electromagnetic switch
to draw DC power.
e. Installing breakers on the primary and secondary sides
to prevent overcurrent, leakage currents, etc. because large
currents are expected to flow.
f. Two exhaust fans are used to exhaust the generated heat.
Because the heat generation is expected to be large, the fanshave been modified to become more powerful.
g. Large resistors are installed to handle high voltages.

Fig.3 Fire process for DC feed
Although DC power supply has many advantages and is inhigh demand, it also has disadvantages. Generally, arcing
occurs when power is interrupted in both DC and AC;
however, in the case of AC, arcing stops spontaneously
because there is a point when the voltage reaches zero.
However, with direct current, the voltage is constant; as longas power continues to be supplied, arcing will not stop
spontaneously. The arc generates heat, which can cause
welding between the contacts and possibly cause a fire.
Furthermore, in the case of photovoltaic power generation,
the system is characterized by the fact that it continues togenerate power as long as it is exposed to sunlight. Therefore,
once an arc is generated in a photovoltaic power generation
system, it will not disappear spontaneously but will continue
to generate power, and there is a risk of damage escalation.
This also implies that the system will continue to generate
power in the event of a fire, which is dangerous from theperspective of firefighter protection.

 ([DPSOH RI %$&QHW XVH ZLWK '& SRZHUVXSSO\This is an example of a BACnet use in solar power
generation. Currently, the DC electricity generated by solar
panels is converted to AC by a power conditioner before
usage. However, as the shift to DC progresses in the future,201

International Workshop on Informatics ( IWIN 2022 )This experiment assumes a scenario in which an electricalfacility is under attack from the outside via a BACnet during
the night when the administrator is asleep, which is turned on
and off repeatedly at a high speed from the BACnet device
for a long period to evaluate the appearance of the switchgear
and the characteristics of the temperature rise near theswitchgear. Temperatures were measured at five locations:outside air temperature, at the switch connector, left side of
the switch, right side of the switch, and temperature inside thepanel.
 Fig.7 shows the locations of the temperature sensors andthe location inside the panel.

Fig.4 Configuration of BACnet evaluation machine

Fig.7 Temperature sensor position of evaluation machine

Fig. 5 Power transfer

([SHULPHQWVXPPDU\Fig.8 shows the switchgear issues due to arc discharge.

Fig. 8 Arc discharge
Fig.9 shows the experimental results at a 6-h/on/off intervalof 1 s.
Regarding the temperature rise characteristics, the connector
part of the switchgear exhibited a steeper temperature risethan the other parts, and the maximum temperature was also
the highest.
Fig.6 Rectifier([SHULPHQWFRQILJXUDWLRQ202

International Workshop on Informatics ( IWIN 2022 )Fig. 9 Temperature characteristics of switchgear

Fig.10 Proposed model of 100,000 monitored points

([SHULPHQWDOUHVXOWVThe model was created considering a building with 43 floors,
as shown in Fig.10, with 50 rooms per floor and fivecontrollers per room. Each controller was assigned anelement to monitor the equipment in the building, and thenumber of elements varied depending on the role of thecontroller. Fig.11 shows the specific role of each object of thecontroller.

BACnet evaluation equipment was designed and improved
to support the DC power supply. The improved equipment
was then operated to verify the temperature characteristics ofthe switchgear. In this experiment, the switchgear did notreach temperatures that could cause ignition; however, arcs
caused by switching the switchgear on and off wereconfirmed. Therefore, although it is unlikely to be a directcause of ignition, the degradation of the switchgear owing toarcing poses a certain threat during long-term operations.

 (YDOXDWLRQ SODWIRUP RQ ODUJHVFDOH EXLOGLQJ
V\VWHPVThe certainty of the attack was demonstrated in the previous
section, as the threat of the actual physical impact was
extracted from the heat rise caused by the continuous
operation of the switchgear owing to the DC power feed.
Such an attack on each BACnet device in a large building
would cause rapid traffic in the network between them. Thissignificantly degrades the network performance and disrupts
the actual environment.
Therefore, we propose and construct an environment toemulate a large-scale building with 100,000 monitoring
points of electrical devices, or properties of objects possessedby BACnet devices, in this evaluation environment building,and generate burst traffic by reading all monitoring points in
a 1-min cycle to evaluate the impact of the attack. Propose
and construct.

Fig.11 Attributes of each monitored object
Each of the five controllers was assigned 12 objects for air
conditioning, 4 for power, 4 for lighting, 6 for fire prevention,and 3 for security, each of which is used to monitor andcontrol the facilities. The number of monitored objects per
floor was multiplied by the number of objects in eachcontroller, multiplied by the number of floors.

0D[LPXPFRQILJXUDWLRQRYHUYLHZThe configuration of a large building with 100,000
monitored points, proposed to generate the maximum load, isshown in Fig.10.

(PXODWLRQRQHQYLURQPHQWFRQILJXUDWLRQThe emulation structure of the BACnet-style trafficgeneration model is shown in Fig.12.

203

International Workshop on Informatics ( IWIN 2022 )ᅗ 13 BACnet emulator

Fig. 12 Network configuration

The Internet Explorer-like tree makes visual operations easy.
Moreover, the tool’s largest function is the "panel," as shownin Fig.14. All BACnet device properties registered in thepanel can issue ReadProperty at the intervals set by the panel.
That is, it is possible to read the monitoring points of the
BACnet device at a certain cycle.

It is difficult for a single PC to emulate 43 levels of controllers
because of performance limitations. Therefore, as shown in
Fig.12, each client computer emulated 15 controller levels
and clustered them. To approximate the real environment,each controller is assigned a virtual NIC with an IP address(192.168.1.11-105) and a MAC address. They received
control monitoring from the B-OWS via a switch.
Additionally, a PC for traffic collection was installed to checkthe current status.

&RQQHFWLYLW\E\6RIW(WKHU931

To make it as close as possible to a real environment, it isnecessary to assign a network interface card (NIC) to each
BACnet device and allocate IP and MAC addresses to them.
Therefore, this platform uses L2-VPN software called
SoftEther VPN to generate and apply virtual NICs.
The role of the SoftEther VPN is to realize a VPN between
locations to create a virtual private network [12]. As shownin Fig.12, a virtual HUB is created on a server computer witha SoftEther VPN installed, and it accepts VPN connections
with client computers that have BACnet devices. NICs can becreated, and the computer that creates the virtual NICs canestablish VPN connections to the virtual HUBs created on the
VPN server. Totally, 105 virtual NICs were assigned to clientcomputers with 105 BACnet devices, each with a different IP
and MAC address. Additionally, SoftEther VPN has a feature
called a local bridge, which allows a bridge connection to beestablished between the virtual HUB of the VPN server andan existing physical Ethernet segment. Using this feature, it
is possible to bridge B-OWS and the VPN server, andexchange packets with the virtual HUB to which the clients’PCs are connected, thus achieving network connectivity on
the same segment.

Fig.14 Operation panel
In this environment, this function is used to read in a virtual
large-scale building with 100,000 monitored points. On thecontroller side, an emulator tool called ICONT Simulator isused to create a controller for each level of the hierarchy andexecute various BACnet services.

2SHUDWLRQRYHUYLHZThis section describes the operational verification of thelarge-scale building emulation environment constructedbased on the description in the previous section. This
generates a pseudo traffic burst. The traffic between the BOWS and client PC is then captured, and the communication
effect is investigated. Fig.15 shows the panel loading screen.

%$&QHWHPXODWRU The BACnet testing tool monitor (TTM), which iscommercially available, was used to emulate B-OWS forbuilding monitoring. Instead of the actual B-OWS, various
BACnet services, such as ReadProperty and WriteProperty,
are sent to the controller to monitor the building status. Fig.13
shows the BACnet TTM operation screen.

204

International Workshop on Informatics ( IWIN 2022 )5HIHUHQFHV

[1] Y. Takeuchi, “Current Situation and Future Prospects
Among NVDC (Medium Voltage Direct Current)
Distribution Network System” The Journal of the Institute ofElectrical Installation Engineers of Japan, Vol.38 No.7
pp.365-368 (2018)[2] O. Gasser, Q. Schitle, B. Rudolph, C. Denis, N. Schricker
and G. Carle "The Amplification Threat Posed by Publicly
Reachable BACnet Devices", Journal of Cyber Security andMobility, Vol.6, No.1, pp.77-104 (2017)[3] G. Stamatescu, I. Stamatescu, N. Arghira, and I.
Făgărășan “Cybersecurity Perspectives for Smart Building
Automation Systems”, In Proceedings of the 12th
International Conference on Electronics, Computers andArtificial Intelligence (ECAI), Bucharest, Romania, 25–27,
pp.1–5 (2020)

Fig.15 Panel loading
Due to the restriction that only 50 properties can be opened
per panel, it is necessary to open as many panels as the total
number of monitored points divided by 50. Furthermore, if
all monitored points are opened simultaneously, thesoftware will stop working due to its specifications, so all 50
panels must be opened at 120 s intervals.

[4] DG. Holmberg, “BACnet Wide Area Network Security
Threat Assessment”, Nist Interagency/Internal Report(NISTIR) (2003)

2SHUDWLRQUHVXOWV

[5] K. Miyazawa, T. Yokotani, and H. Mukai, “Developmentof a benchmark system on power-related control by
BACnet/IP”, IWIN2021, Session8, No.28 (2021)

Fig.16 shows the traffic transition when the model shown in
Fig. 10 is continuously loaded in 60 s cycles.

[6] ASHRAE.BACnet – A Data Communication Protocol for
BuildingAutomation and Control Systems Addendum 1352012aj (2016).
[7] K. Tomizawa, “Object and Property”, The Journal of theInstitute of Electrical Installation Engineers of Japan, Vol.32,
No.2, pp-115-118 (2012)[8] M. Nast, B. Butzin, F. Golatowski, and D. Timmermann,“Performance analysis of a secured BACnet/IP network,” in2019 15th IEEE International Workshop on FactoryCommunication Systems (WFCS), pp.1–8 (2019)

Fig.16 Read all monitored points

 &RQFOXVLRQVDQGIXWXUHZRUNVIn this study, we improved the BACnet evaluation system
we developed in the previous paper. As a result of assumingthat the high frequency of on/off signals is an unauthorizedattack, we found that physical abnormalities occur in theswitchgear in the DC power supply. We then proposed amaximum logical configuration that emulates a large-scalebuilding and proposed an environment in which the amountof network traffic can be verified by reading it at regularcycles. In the future, we plan to develop a tool that can easilychange the configuration of the maximum environment toenable flexible evaluation of network traffic according to thebuilding environment. [9] AM. Fedel, AM. Ibrahim, AK. Tamazin, RA. Hamdy, andAyman S. Abdel-Khalik, “IoT-Based Power Management for
DC Microgrids”, 21st International Middle East Power
Systems Conference (2019)[10] Agency for Natural Resource and Energy, “Act on
Special Measures concerning Promotion of Utilization of
Renewable Electric Energy”, https://www.fitportal.go.jp/PublicInfoSummary (2022)[11 ] Consumer Affairs Agency
https://www.caa.go.jp/policies/council/csic/report/report_01
2/pdf/report_012_190128_0002.pdf, (2021)$FNQRZOHGJPHQWVPart of this work was supported by the Cabinet Office(CAO), Cross-ministerial Strategic Innovation Promotion
Program (SIP), “Cyber-Physical Security for an IoT Society”(funding agency: NEDO). The authors sincerely thank themembers concerned.

[12] VPN Gate, “Connect to VPN Gate by Using SoftEther
VPN (SSL-VPN)”,
https://www.vpngate.net/en/howto_softether.aspx (2022)

205

International Workshop on Informatics ( IWIN 2022 ))LUPZDUH'LVWULEXWLRQZLWK(UDVXUH&RGLQJIRU,R7DHYLFHV
Takenori Sumi*†, Yukimasa Nagai* , and Hiroshi Mineno†*

Information Technology R&D Center, Mitsubishi Electric Corporation, Japan†
Graduate School of Science and Technology, Shizuoka University, Japan
Sumi.Takenori@dc.MitsubishiElectric.co.jp$EVWUDFW – Due to the increase in the number of IoT devices
connected to the internet, 920 MHz frequency bands for
wireless communication systems are attracting attention forvarious IoT applications, e.g., environmental monitoring,smart metering, process monitoring & control. With wirelesscommunication systems on 920 MHz having the features oflong distance, low rate and low power consumption, a huge
number of IoT devices distributed in wide area can beconnected to communication networks. When distributing thesame data to IoT devices such as firmware distribution during
operation, improving the efficiency of distribution method
becomes an issue. We propose a new firmware distribution
method with erasure coding for IoT devices. Our computer
simulation result shows that proposed method improves theefficiency of distribution by 1.7 times compared withconventional method and achieves higher spectrum
efficiency.

the simulation architecture and results for various conditions.
Finally, we conclude our paper in Section V.

 5(/$7(':25.
There are existing researches for wireless communications
using 920 MHz and firmware distribution. Since 920 MHz isnarrow bands compared to 2.4 GHz and 5 GHz for ISM band,
special regulation for “10 % transmission duty cycle” and“longer backoff mechanisms” are applied in Japan.
Throughput performance has been demonstrated in [1] and[2], which focus on the PHY and MAC protocol enhancement
for higher-throughput, protocol efficiency and delay via
simulation and measurement result using prototypes. For
example, D. Hotta et al. introduce the performance of multihop routing construction using Wi-SUN FAN (Field Area
Network) prototypes based on IEEE 802.15.4g FSK PHY [3].
Japanese standard ARIB STA-T108 (20 mW, unlicensed)
defines the use of IEEE 802.15.4g system from 920.5 –
928.1MHz (7.6 MHz bandwidth), but the ARIB STA-T107(250 mW, passive system) and the ARIB STD-T108 (250mW,
licensed/registered) also define operation from 920.5 – 923.5
MHz (3.0 MH). Therefore, 923.5 – 928.1 MHz (4.6 MHz
bandwidth) is the only reasonable frequency band for IEEE
802.15.4g applications in the unlicensed spectrum. IEEE
802.15.4g is regulated to operate over 200 kHz bandwidthchannel in the Sub-1 GHz band. Even low duty cycle
constraint applied in the Sub-1 GHz band, e.g., Japanese andEuropean standard allow up to 10% transmission duty cycle[4] - [7] for the number of IoT Devices increased with various
standards. Therefore, ensuring higher efficiency for spectrum
use in the Sub-1 GHz is clearly important.

.H\ZRUGV: IoT, Firmware Distribution, Erasure Code ,1752'8&7,21
In addition to smart phone, laptop and tablet connecting to
the internet, IoT devices such as sensors are getting connect
to the internet. Due to the increase in the number of IoT
devices connected to the internet, 920 MHz frequency bands
for wireless communication systems are attracting attention
for various IoT applications, e.g., environmental monitoring
application for location, temperature, humidity and water
lever, smart metering application, process monitoring &control application. Wireless communication systems on 920
MHz (Sub-1 GHz) have the features of long distance, low rate
(several 10 kbps – 100 kbps) and low power consumption forconventional standards, e.g., IEEE 802.15.4g, LoRa WAN,
SigFox, but the higher data rate up to several Mbps is also
considered for applications such as infrastructure monitoring,surveillance camera in IEEE 802.11ah / Wi-Fi HaLow. For
long life IoT devices, the firmware update for a large numberof IoT devices is also considered as new IoT application.
Since 920 MHz has the features of long distance from several
100 m to several km, a large number of IoT devices are
deployed in the area. Thus, firmware distribution for IoT
devices using 920 MHz with narrow band is a challenge. Thus,
efficient firmware distribution method for a large number of
IoT devices should be considered. In this paper, we focus on
920MHz for IoT applications and propose the new firmware
distribution method using erasure coding to achieve higher
efficiency for limited radio frequency.
The rest of this paper is organized as follows. Section II
presents related work. Section III describes the proposed
firmware distribution with erasure coding. Section IV shows 352326('0(7+2' &KDOOHQJHVLQ)LUPZDUH'LVWULEXWLRQ
Firmware distribution to IoT devices requires sending thesame firmware data to a large number of IoT devices via
wireless communication. Firmware distribution using unicast
is inefficient because the same firmware data is sent to eachdevice individually. Therefore, it is considered more efficient
to distribute to a large number of IoT devices using broadcast.
However, it is difficult to deliver all firmware data to all IoT
devices in a single communication because wirelesscommunication suffers from packet errors due to low
received signal power and/or interference from other wirelesssystems. As shown in Figure 1, packet errors occur randomly
in IoT devices, requiring large number of packets to be
retransmitted. Therefore, even when broadcasts are used forfirmware distribution, there are issues with distribution

207

International Workshop on Informatics ( IWIN 2022 )

efficiency. In this paper, we propose a method to reduce thenumber of transmitted packets in firmware distribution byusing erasure coding.

)LUPZDUH'DWD (UDVXUH&RGH

3DFNHW


3DFNHW


3DFNHW.

,R7 GHYLFH

3DFNHW


3DFNHW


3DFNHW


3DFNHW.

,R7 GHYLFH

3DFNHW


3DFNHW


3DFNHW


3DFNHW.

,R7GHYLFH1

3DFNHW


3DFNHW


3DFNHW


3DFNHW.

7UDQVPLWSDFNHWVWRWKH
ODUJHQXPEHURI,R7GHYLFHVIn this paper, RC QC-LDPC (Rate Compatible QuasiCyclic-LDPC)[8] is used as Erasure Code for firmware
distribution. As shown in Figure 2, RC QC-LDPC generates
redundant packets by erasure coding information packets that
divide the data to be transmitted into packets of a certain data
length. The data length of information packets and redundant
packets are the same. The number of information packets, K,
is a multiple of 36, and the number of redundant packets, M,
is equal to K or twice as large. RC QC-LDPC can attempt to
decode the transmitted data if the total number of received
information packets and redundant packets is K or more.
Even if some of the received information packets are missing,
the transmitted data can be recovered using the redundant
packets. Figure 3 shows the relationship between“redundancy rate” and decoding success rate. Here, the“redundancy rate” is defined as the sum of the number of
information packets Krx and the number of redundant packets
Mrx at decoding and the original number of information
packets K, using the following formula.

ܴ݁݀‫ ݁ݐܴܽݕܿ݊ܽ݀݊ݑ‬ൌ 

3DFNHW

7UDQVPLWWHU

௄ೝೣ ାெೝೣ
௄



7LPH

)LJXUH)LUPZDUH'LVWULEXWLRQIRU,R7'HYLFHV
'DWDWREHWUDQVPLWWHGʁ,QIRUPDWLRQ3DFNHW.

ʁ5HGXQGDQW3DFNHW(UDVXUH&RGLQJ






.







0

)LJXUH(UDVXUH&RGLQJ

(1)ĞĐŽĚŝŶŐ^ƵĐĞƐƐZĂƚĞ

Figure 3 shows that RC QC-LDPC does not ensure
successful decoding even when the redundancy is greaterthan 1. The higher the number of information packets K, thehigher the decoding success rate for the redundancy rate.
However, the larger the number of information packets K,
the more memory and CPU resources are required for the
decoding process.

 )LUPZDUH'LVWULEXWLRQZLWK(UDVXUH&RGH
In the proposed firmware distribution method, as shown in
Figure 4, the transmitter first erasure codes the firmware data
(K information packets in the figure) to generate M
redundancy packets (ᶅ in the figure). Then, the Transmitter
sends up to a total of K information packets and redundant
packets (ᶆ in the figure). After K packets have been sent,
depending on the number of packet errors in the IoT device,additional information packets and redundant packets are sent
until the IoT device can decode the firmware data (߉ in the
figure). Here, the additional packets to be sent are those notsent in ᶆ in the figure. The IoT device uses the received K or

ϭ
Ϭ͘ϵ
Ϭ͘ϴ
Ϭ͘ϳ
Ϭ͘ϲ
Ϭ͘ϱ
Ϭ͘ϰ
Ϭ͘ϯ
Ϭ͘Ϯ
Ϭ͘ϭ
Ϭ

<сϯϲ
<сϳϮ
<сϭϬϴ
<сϭϰϰ
<сϭϴϬ
<сϯϲϬ
<сϱϰϬ

ϭ

ϭ͘ϭ

ϭ͘Ϯ
ϭ͘ϯ
ϭ͘ϰ
ZĞĚƵŶĚĂŶĐǇZĂƚĞϭ͘ϱ

)LJXUH35HGXQGDQF\5DWHYV'HFRGLQJ6XFFHVV5DWH
)LUPZDUH'DWDʁ,QIRUPDWLRQ3DFNHW.

ʁ5HGXQGDQW3DFNHW⋇*HQHUDWHUHGXQGDQWSDFNHWVZLWKHUDVXUHFRGH(UDVXUH&RGLQJ






.







启7UDQVPLWDGGLWLRQDOSDFNHWVIRUHUDVXUHFRUUHFWLQJ⋈7UDQVPLW.SDFNHWV

more packets to decode and recover the firmware data (ᶈ inthe figure). As described in section 3.2, RC QC-LDPC may
fail in decoding. If decoding fails, additional packets are sentfrom the transmitter and the IoT device performs the
decoding process again.
To implement the proposed method, IoT devices need to
inform the transmitter which packets are missing. In the
proposed scheme, IoT devices notify the transmitter of a
NACK (Negative Acknowledgement) packet and an ACK

































,R7GHYLFH

,R7 GHYLFH,R7GHYLH1







7UDQVPLWWHU



7LPH

0

⋊,R7GHYLFHVFRUUHFWILUPZDUHGDWDZLWKUHFHLYHG
LQIRUPDWLRQSDFNHWVDQG
UHGXQGDQWSDFNHWV'HFRGLQJZLWK(&)LUPZDUH'DWD)LJXUH3URSRVHGILUPZDUHGLVWULEXWLRQZLWK(&208

International Workshop on Informatics ( IWIN 2022 )(Acknowledgement) packet in the sequence shown in Figure
5. The NACK packet is used to notify which packets are
missing; the ACK packet is used to notify that decoding was
successful. The packet formats of NACK and ACK packets
are shown in Figure 6. UDP is used to send NACK and ACK
packets, and Figure 6 shows only the UDP payload. The
NACK packet contains a packet type indicating whether it is
NACK or ACK and a bitmap indicating packet loss. The
length of the bitmap is the sum of the number of information
packets K and the number of redundant packets M. The ACK
packet contains only the packet type.

7UDQVPLWWHU

,R7
,R7
,R7
GHYLFH GHYLFH GHYLFH

7UDQVPLWLQIRUPDWLRQUHGXQGDQWSDFNHWE\EURDGFDVWʀʀʀ

3DFNHWORVV'HFRGLQJZLWK(&VXFFHVVIDLO

IDLO,R7GHYLFHDQGVHQG1$&.
SDFNHWIRUUHTXHVWLQJDGGLWLRQDO
LQIRUPDWLRQUHGXQGDQWSDFNHW,R7GHYLFHVHQGV$&.SDFNHWWRQRWLI\GHFRGLQJVXFFHVV
7UDQVPLWDGGLWLRQDO
LQIRUPDWLRQUHGXQGDQWSDFNHWIRU,R7GHYLFHV
DQGE\EURDGFDVW

 6,08/$7,21(9$/8$7,21
We evaluate the proposed method using computer
simulation. Table 1 shows simulation parameters. In thecomputer simulation, IEEE 802.15.4g [9], which is used in
IoT devices in the 920 MHz band, was used as the wirelesscommunication method and evaluated in an environment
where IoT devices are connected in a star network from atransmitter. The firmware data is divided into 248 Bytes eachand broadcast to the IoT device as the payload of UDP
packets. RC QC-LDPC was used as erasure code, and thenumber of information packets K and redundancy packets M
were set to 36 - 540. In the computer simulation, theconventional firmware distribution without erasure code alsobroadcasts the firmware data to the IoT devices by dividing it
into 248 Bytes each, as in the proposed method. Then, after
the K-packet transmission is finished, the IoT devices send
NACK or ACK to the transmitter to notify whether it needs
to retransmit data. This is to ensure that the opportunities tosend NACK and ACK packets are the same for theconventional and proposed methods. In the conventional
method, the number of redundancy packets M is 0 because
erasure coding is not performed for the firmware data.
First, the relationship between the number of information
packets K and effective throughput is shown in Figure 7,
which shows the simulation results when firmware is
distributed to 20 IoT devices and PER is 10%. Since a PERof 10% or less is often set for wireless communication
systems to take operations into account [9], the simulations inthis paper are based on an evaluation at a PER of 10%.
According to the figure, the throughput of the proposed
method is higher than that of the conventional method,
regardless of the value of K. As the value of K increases, theeffective throughput of the proposed method increases. As
mentioned in section 3.2, this is because the larger K is, thehigher decoding success rate at the same redundancy rate.
However, since the memory and decoding processing loadincreases as K increases, the subsequent simulation
evaluation will be performed for the case where the numberof information packets K is 180, where the effective
throughput increase rate by the proposed method is relatively
high.
Next, Figure 8 shows the simulation result on therelationship between PER and effective throughput whenfirmware is distributed to 20 IoT devices with the number of
information packets K set to 180. The figure shows that if the
PER is 1% or higher, the effective throughput is higher withthe proposed method than with the conventional method.
However, the effective'HFRGLQJZLWK(&
VXFFHVV VXFFHVV,R7GHYLFHDQGVHQG
$&.SDFNHWWRQRWLI\GHFRGLQJVXFFHVV,QIRUPDWLRQ5HGXQGWDQW 3DFNHW
1$&.3DFNHW
$&.3DFNHW)LJXUH3DFNHWVHTXHQFHRIWKHSURSRVHGPHWKRG
%\WH. 0 %\WH

1$&.3DFNHW

3DFNHW 3DFNHWORVVELWPDS
W\SH5HFHLYHG3DFNHWORVW$&.3DFNHW

3DFNHW
W\SH

%\WH)LJXUH3DFNHWIRUPDWVRI1$&.DQG$&.SDFNHW
throughput at a PER of 0% with no packet errors isapproximately 40 kbps for the conventional method, but 33
kbps for the proposed method, which is lower than theconventional method. As mentioned in Section 3.2, this isbecause RC QC-LDPC codes may not succeed in decoding
even if a total of K or more information packets andredundancy packets are received. Additional packets need to
be sent from the transmitter to the IoT devices. Since packet
errors occur in wireless communication systems, theproposed method can be applied to improve the effective
throughput in an environment with a PER < 10%, where
wireless communication systems are normally operated.
Finally, the relationship between the number of IoT devices
and effective throughput is shown in Figure 9, which shows
simulation results when the number of information packets K
is 180 and PER is 10%. The figure shows that the effective
throughput of the proposed method is higher than that of theconventional method when firmware distribution is
performed to two or more IoT devices. In the case of a single
IoT device, the effective throughput is lower than theconventional method due to the significant impact of thepossibility of unsuccessful decoding even if the number ofreceived packets is K or more. For 20 IoT devices, it is 1.60
times the effective throughput of the conventional method;for 50 devices, it is 1.73 times; and for 100 devices, it is 1.76
times. However, as the number of IoT devices increases, theeffective throughput decreases: 29.7 kbps for 20 IoT devices,24.8 kbps for 50 devices, and 19.8 kbps for 100 devices. Thisis likely due to an increase in NACK packets.
The transmission duty cycle of the transmitter in thesimulation of Figure 9 is shown in Figure 10. As mentioned

209

International Workshop on Informatics ( IWIN 2022 )in Chapter 2, Japanese and European standard allow up to
10% transmission duty cycle. Figure 10 shows that regardlessof the number of IoT devices, the transmission duty cycle ofthe transmitter exceeds 10% for both the proposed andconventional methods. The transmission duty cycle of the
proposed method is lower than that of the conventional
method, and it tends to be lower when the number of IoT
devices is larger. Therefore, in Japan and Europe, when thetransmission duty cycle is 10% or less, the difference between
the effective throughput of the proposed method and theconventional method is expected to widen.

ĨĨĞĐƚŝǀĞdŚƌŽƵŐŚƉƵƚ΀ŬďƉƐ΁ϰϬ

Firmware distribution packet length
Firmware distribution ACK packet
Firmware distribution NACK packet
6LowPAN and UDP header
PHY/MAC
MAC parameters
macMinBE
macMaxBE
LIFS
aUnitBackoffPeriod
phyCcaDuration
aTurnaroundTimetack
MAC header size
FCS
ACK frame size
PHY parameters
Data rate
Modulation
Modulation index
Frequency
Channel spacing

ϭϬ
ϱ
ϭϬϬ

ϮϬϬ

ϯϬϬ
<

ϰϬϬ

ϱϬϬ

ϲϬϬ

ĨĨĞĐƚŝǀĞdŚƌŽƵŐŚƉƵƚ΀ŬďƉƐ΁

ϯϱ
ϯϬ
Ϯϱ
ϮϬ

WƌŽƉŽƐĞĚ
ŽŶǀĞŶƚŝŽŶĂů

ϭϱ
ϭϬ
ϱ
Ϭ
Ϭ

ϱ

ϭϬ
WZ΀й΁ϭϱϮϬ)LJXUH3(5YV7KURXJKSXW .,R7GHYLFHVϰϬĨĨĞĐƚŝǀĞdŚƌŽƵŐŚƉƵƚ΀ŬďƉƐ΁M, the number of redundant packetsWƌŽƉŽƐĞĚ
ŽŶǀĞŶƚŝŽŶĂů

ϭϱ

ϰϬ

ϯϱ
ϯϬ
Ϯϱ
ϮϬ

WƌŽƉŽƐĞĚ
ŽŶǀĞŶƚŝŽŶĂů

ϭϱ
ϭϬ
ϱ
Ϭ
Ϭ ϭϬ ϮϬ ϯϬ ϰϬ ϱϬ ϲϬ ϳϬ ϴϬ ϵϬ ϭϬϬ/ŽdĞǀŝĐĞƐ

)LJXUH7KHQXPEHURI,R7GHYLFHVYV7KURXJKSXW.3(5dƌĂŶƐŵŝƐƐŝŽŶƵƚǇǇĐůĞ΀й΁

1 - 100
Rate-Compatible QC-LDPC
36, 72, 108, 144, 180, 360,
540
Conventional method: 0
Proposed method: same as K
248 Byte
1 Byte1+(K+M)/8 Byte
53 ByteIEEE 802.15.4g
Value
5
8
1000 us1130 us130 us1000 us1000 us
11 Byte2 Byte9 ByteValue
100 kbps
2-FSK
1.0
923.7 MHz
400 kHzϮϬ)LJXUH.YV7KURXJKSXW 3(5,R7GHYLFHV

7DEOH6LPXODWLRQSDUDPHWHUVThe number of IoT devices
Erasure code
K, the number of information packetsϮϱϬ

We proposed the new firmware distribution method using
erasure coding to achieve higher efficiency for limited radio
frequency. The performance of the proposed method wasevaluated by computer simulation and compared to theeffective throughput of conventional firmware distribution
methods. By applying the proposed method to firmware
distribution, the effective throughput was found to be 1.60
times higher than that with the conventional method when
there are 20 IoT devices receiving firmware data, 1.73 times
higher when there are 50 devices, and 1.76 times higher when
there are 100 devices. In this paper, we evaluated theperformance of the proposed method in a single-hop wirelessnetwork. Our future work is to investigate the application of
the proposed method to multi-hop wireless networks.
Value

ϯϬ

Ϭ

 &21&/86,21

Firmware distribution parameters

ϯϱ

ϭϬϬ
ϵϬ
ϴϬ
ϳϬ
ϲϬ
ϱϬ
ϰϬ
ϯϬ
ϮϬ
ϭϬ
Ϭ

WƌŽƉŽƐĞĚ
ŽŶǀĞŶƚŝŽŶĂůϬ ϭϬ ϮϬ ϯϬ ϰϬ ϱϬ ϲϬ ϳϬ ϴϬ ϵϬ ϭϬϬ/ŽdĞǀŝĐĞƐ

)LJXUH107KHQXPEHURI,R7GHYLFHVYV'XW\&\FOH.3(5210

International Workshop on Informatics ( IWIN 2022 )5()(5(1&(6[1] C.-S. Sum, F. Kojima, and H. Harada, ‘‘Performance
analysis of a multiPHY coexistence mechanism forIEEE 802.15.4g FSK network,’’ in Proc. IEEE
Wireless Communication Network. Conf. (WCNC),
Apr. 2013, pp. 41–46.
[2] F. Righetti, C. Vallati, D. Comola, and G. Anastasi,‘‘Performance measurements of IEEE 802.15.4g
wireless networks,’’ in Proc. IEEE 20th Int.
Symposium World Wireless, Mobile Multimedia
Network (WoWMoM), Jun. 2019, pp. 1–6.
[3] D. Hotta, R. Okumura, K. Mizutani, and H. Harada,‘‘Stabilization of multi-hop routing construction in
Wi-SUN FAN systems,’’ in Proc. IEEE 17th Annual
Consumer Communication and Network Conference(CCNC), Jan. 2020, pp. 1–6.
[4] 920MHz-Band RFID Equipment for Specified Low
Power Radio Station, ARIB STD-T107 version 1.1,
ARIB, Tokyo, Japan, 2017.
[5] 920MHz-Band Telemeter, Telecontrol and Data
Transmission Radio Equipment, ARIB STD-T108
version 1.2, ARIB, Tokyo, Japan, 2018.
[6] Y. Nagai, J. Guo, P. Orlik, Ben A. Rolfe, and T. Sumi,
Proposal for Section 6.1 Japan on Recommended
Practice, document IEEE 802.19-19/0049r0, IEEE
802.19, 2019.
[7] T. Sumi, Y. Nagai, J. Guo, and P. Orlik, 920 MHz
Status Update in Japan, document IEEE 802.1918/0084r0, IEEE 802.19, 2018.
[8] Wataru Matsumoto, Rui Sakai, Hideo Yoshida, “Rate
Compatible QC-LDPC codes,” SITA 2006, pp.387390 (2006).
[9] IEEE 802.15.4-2020, “IEEE Standard for Low ̺
RateWireless Networks,” IEEE 802.15.4-2020, IEEE,(2020).

211