IWIN2023
International Workshop on Informatics

Proceedings ofInternational Workshop on Informatics
September 1-4, 2023
Onuma, Hokkaido

Sponsored by Informatics Society

IWIN2023
International Workshop on Informatics

Proceedings ofInternational Workshop on Informatics
September 1-4, 2023
Onuma, Hokkaido

Sponsored by Informatics SocietyPublication office:Informatics Laboratory
3-41, Tsujimachi, Kitaku, Nagoya 462-0032, Japan
Publisher:
Tadanori Mizuno, President of Informatics
ISBN:
978-4-902523-50-8
Printed in Japan

Table of Contents
Session 1: Systems and Applications 1 ····················· 1( Chair: Katsuhiko Kaji ) ( 9:10 - 11:10, Sep. 2 )(1)

A Study on a Power Interchange Method for Realizing Net-Zero Energy for
Multiple Small Communities ························································ 3
Masashi Saito, Tomoki Nomura, and Yuichi Tokunaga(2)

Development of a Disaster Safety Confirmation Collection System using
Q-ANPI Service linked with Web Interface ······································ 11
Mitsuki Sano, Takuma Sato, Masao Isshiki, and Keiichi Abe(3)River Water Level Prediction Using Radar Rainfall with Deep Learning ··· 17
Futo Ueda, Hiroto Tanouchi, Nobuyuki Egusa, and Takuya Yoshihiro(4)

Developing an Infomation System to Enhance Intellectual Productivity in
Teleworking Environments ························································· 23
Kanae Matsui and Katsuma Takagi

Keynote Speech 1 ······························································ 29( Chair: Akihiro Hayashi ) ( 11:20 - 12:10, Sep. 2 )(I)Transforming Manufacturing with Information Technology ·················· 31
Dr. Hidetoshi MishimaiSession 2: Intelligent Transportation Systems ···· 51( Chair: Kozo Okano ) ( 13:20 - 15:50, Sep. 2 )(5)

Relieving Traffic Congestion Stably by Controlling Amount of Vehicles
Detouring from Congested Roads ·················································· 53
Arisa Kitano and Takuya Yoshihiro(6)

A Transfer Scheduling Method for ride-sharing Services to Alleviate Traffic
Congestion ············································································ 61
Shion Suzuki and Takuya Yoshihiro(7)

Development of a Framework for Jaywalking Risk Map to Reduce
Pedestrian-to-Vehicle Accidents ··················································· 65
Yuich Tokunaga, Atsushi Yamamoto, and Masash Saito(8)

Obstacle Avoidance with Extended Velocity Obstacles Algorithm ············ 71
Takumi Seita, Shunsuke Michita, Seiji Komiya, and Toshihiro Wakita(9)

A LiDAR Point Cloud Data Fusion from Connected Autonomous Vehicles to
Reduce Network Traffic Congestion at Server ···································· 77
Haripriya Pasupuleti, Yoshinao Wada, Kenjiro Yamada, Takuro Sone,
and Tomoya KitaniiiSession 3: Network and Security ······························· 85( Chair: Tomoya Kitani ) ( 16:10 - 18:40, Sep. 2 )(10)

A Scheduling Method for LoRaWAN Assuming a Mixture of Incompatible
Devices ················································································ 87
Tabasa Ishibashi and Takuya Yoshihiro(11)

Comparison on Function and Performance between MQTT and DDS for IoT
DEP ················································································ 93
Masaki Mitsuuchi, Kazuhiro Kosaka, Tetsuya Yokotani,
and Koichi Ishibashi(12)

Study of an Implementation Method of Point-to-multipoint Communication
for IoT Data Exchange to Reduce Traffic on an IoT Network ·············· 101
Takahiro Shiohara, Koichi Ishibashi, and Tetsuya Yokotani(13)

Design of Low-rate DoS Attack Detection in Robust WRED ··············· 107
Daisuke Sato, Hiroshi Inamura, Shigemi Ishida,
and Yoshitaka Nakamura(14)

A Two-Dimensional-Trust-Based Recommendation Method for Job
Placement Assistance ······························································ 113
Yoshinobu Kawabe and Tetsuhisa Oda

iii

Session 4: Sensing and Analysis ······························· 119( Chair: Takuya Yoshihiro ) ( 8:40 - 11:10, Sep. 3 )(15)Analysis and Sharing of Cooking Actions Using Wearable Sensors ······· 121
Ayato Kumazawa, Fuma Kato, Katsuhiko Kaji, Nobuhide Takashima,
Katsuhiro Naito, Naoya Chujo, and Tadanori Mizuno(16)

Design and Implementation of a Method for Estimating Bicycle Air Pressure
Decrease based on Vibration Sensing of Bicycles using Smartphone ···· 131
Rui Yamaguchi, and Katsuhiko Kaji(17)

Evaluation of Human Recognition Method by 2D-LiDARs ················ 135
Haruki Mochizuki, Takuya Watanabe, Yuya Sawano,
Yoshiaki Terashima, and Ryozo Kiyohara(18)

Estimating the Best Time to View Cherry Blossoms using Natural Language
Processing Tasks ··································································· 137
Ryoga Sato, Yusuke Takamori, Kenji Terada, Masaki Endo,
Tsuyoshi Tanaka, Shigeyoshi Ohno, and Hiroshi Ishikawa(19)

Relation between Time-series Forecasting Methods and Estimation of Best
Times to See Cherry Blossoms ··················································· 145
Asahi Iha, Kenji Terada, Masaki Endo, Tsuyoshi Tanaka,
higeyoshi Ohno, and Hiroshi Ishikawa

Keynote Speech 2 ···························································· 153( Chair: Yoshia Saito ) ( 11:20 - 12:20, Sep. 3 )(Ⅱ) A Compiler System Supporting Memory Shared by Heterogeneous Machines· ··························································································· 155
Prof. Dr.Hitoshi Aidaiv

Session 5: Multimedia and Communication ······· 171( Chair: Yuichi Tokunaga ) ( 13:20 - 15:20, Sep. 3 )(20)Comparative of Experienced and Inexperienced People by Shoot Form Analysis
from Free Throws of Basketball using 2-D Joint Position Information and Slow
Motion ················································································ 173
Tetsuto Mukai, Shota Ihigaki, and Keiichi Abe(21)

Disaster Victim Impact Analysis System using a Communication Failure
Emulator ········································································ 181
Kei Hiroi, Akihito Kohiga, Sho Fukaya, and Yoichi Shinoda(22)"I'm Going Right" or "Please Go to the Right": Disambiguation in Arrow
Display on Mobile Robots to Avoid Collision with Passersby ··············· 189
Yo Kuwamiya, Taketo Imagawa, and Minoru Kobayashi(23)

A Broadcast-based Online Machine Learning System for Environmental
Value Prediction ···································································· 199
Tomoki Yoshihisa, and Rikuto Kotougev

Session 6: Systems and Applications 2 ··················· 205( Chair: Kei Hiroi ) ( 15:30 - 17:30, Sep. 3 )(24)

DNN-based Fault Localization with Virtual Coverage based on Number of
Executions ··········································································· 207
Takuma Ikeda, Hitoshi Kiryu, Shinpei Ogata, and Kozo Okano(25)

A Spherical POV Heatmap using Mixed Reality in 360-degree Internet Live
Broadcasting ········································································· 215
Yoshia Saito and Junichiro Suto

(26)

Were You Speaking to Me?: A Trial to Use Physical Avatars to Establishing
Gaze Awareness in Hybrid Meetings ············································ 219
Kenta Ohnaka, Taai Tsukidate, Yo Kuwamiya, Kazuyuki
Iso, and Minoru Kobayashi(27)

Exploring Key Concepts in Arts and Mental Health: A Questionnaire-based
Analysis of Perception on Arts, Sports, and Therapy ·························· 227
Michi Komura, Hajime Kaneko, Midori Ishihara, and Akihiro HayashiviMessage from the General Chairs

It is our great pleasure to welcome all of you to Mori-machi, Hokkaido, Japan for the 17th
International Workshop on Informatics (IWIN2023). This workshop has been held annuallyby the Informatics Society. Since 2007, the workshops were held in Naples in Italy, Wien inAustria, Hawaii in the USA, Edinburgh in Scotland, Venice in Italy, Chamonix in France,
Stockholm in Sweden, Prague in Czech Republic, Amsterdam in Netherlands, Riga in Latvia,
Zagreb in Croatia, Salzburg in Austria, Hamburg in Germany, Wakayama in Japan(virtually), Fukui in Japan (virtually), and Wakayama in Japan respectively.
In IWIN2023, 27papers were accepted after peer reviewing by the program committee. Basedon the papers, six technical sessions were organized in a single-track format, whichhighlighted the latest research results in the areas such as Systems and Applications,Intelligent Transportation Systems, Network and Security, Sensing and Analysis, andMultimedia and Communication. IWIN2023 will also welcome two keynote speakers: Dr.
Hidetoshi Mishima, Executive Fellow, Corporate Research and Development Group,
Mitsubishi Electric Corporation and Professor Emeritus. Dr. Hitoshi Aida, The University ofTokyo. We greatly appreciate their participation in our workshop.
We would like to thank all the participants and contributors who made the workshop possible.
It is indeed an honor to work with a large group of professionals around the world to makethe workshop a great success. We are looking forward to seeing you all in the workshop. We
hope you enjoy IWIN2023.
September 2023

Yoshia Saito
Akihiro Hayashi

viii

Organizing Committee
General Co-Chairs
Akihiro Hayashi (Shizuoka Institute of Science and Technology, Japan)
Yoshia Saito (Iwate Prefectural University, Japan)

Steering Committee
Hitoshi Aida (The University of Tokyo, Japan)
Toru Hasegawa (Osaka University, Japan)
Teruo Higashino (Kyoto Tachibana University, Japan)
Tadanori Mizuno (Aichi Institute of Technology, Japan)
Jun Munemori (The Open University of Japan, Japan)
Yuko Murayama (Tsuda University, Japan)
Ken-ichi Okada (Keio University, Japan)
Norio Shiratori (Chuo University / Tohoku University, Japan)
Osamu Takahashi (Future University Hakodate, Japan)

Program Chair
Kei Hiroi (Kyoto University, Japan)

Financial Chair
Tomoya Kitani (Shizuoka University, Japan)

Publicity Chair
Yoshitaka Nakamura (Kyoto Tachibana University, Japan)

Program Committee
Hiroshi Sugimura

Akihito Hiromori (Osaka University, Japan)(Kanagawa Institute of Technology, Japan)

Akra Uchiyama (Osaka University, Japan)
Hideki Goromaru (Chiba Institute of Technology, Japna)

Hiroshi Yoshiura (Kyoto Tachibana University, Japan)

Hironobu Abe (Tokyo Denki University, Japan)

Kanae Matsui (Tokyo Denki University, Japan)

Hiroshi Mineno (Shizuoka University, Japan)

Katsuhiko Kaji (Aichi Institute of Technology, Japan)ix

Katsuhiro Naito (Aichi Institute of Technology, Japan)

Tetsuya Yokotani (Kanazawa Institute of Technology, Japan)

Kazuyuki Iso

Tomoki Yoshihisa (Shiga University, Japan)(Tokyo Information Design Professional University, Japan)

Tomoo Inoue (University of Tsukuba, Japan)

Kei Utsugi (Hitachi, Ltd, Japan)

Tomoya Kitani (Shizuoka University, Japan)

Keiichi Abe (Kanagawa Institute of Technology, Japan)

Tomoyuki Yashiro (Chiba Institute of Technology, Japan)

Ken Ohta (Miyagi University, Japan)

Toshihiro Wakita (Kanagawa Institute of Technology, Japan)

Kozo Okano (Shinshu University, Japan)

Tsukasa Kudo

Makoto Imamura (Tokai University, Japan)(Shizuoka Institute of Science and Technology, Japan)

Masaji Katagiri (iU, Japan)

Yoshia Saito (Iwate Prefectural University, Japan) Yoshiaki

Masaki Endo (Polytechnic University, Japan)

Terashima (Soka University, Japan)

Masaki Nagata (Shizuoka University, Japan)

Yoshinobu Kawabe (Aichi Institute of Technology, Japan)

Masashi Saito

Yoshitaka Nakamura (Kyoto Tachibana University, Japan)(Kanazawa Institute of Technology, Japan)

Yu Enokibori (Nagoya University, Japan)

Michiko Oba (Future University Hakodate, Japan)

Yuki Koizumi (Osaka University, Japan)

Mikiko Sode

Yukimasa Nagai (Mitsubishi Electric Corporation, Japan)(National Institute of Technology, Niihama College, Japan)

Yuichi Bannai (Kanagawa Institute of Technology, Japan)

Minoru Kobayashi (Meiji University, Japan)

Yuichi Tokunaga

Naoya Chujo (Aichi Institute of Technology, Japan)(Kanazawa Institute of Technology, Japan)

Nobuhiro Kobayashi (University of Nagasaki, Japan)

Yusuke Gotoh (Okayama University, Japan)

Orie Abe

Yusuke Ichikawa (Saitama University, Japan)(Information Communication Media Laboratory, Japan)

Yusuke Takatori (Kanagawa Institute of Technology, Japan)

Ryozo Kiyohara (Kanagawa Institute of Technology, Japan)
Satoru Matsumoto (Osaka University, Japan)
Shigemi Ishida (Future University Hakodate, Japan)
Shigeyoshi Ohno (Polytechnic University, Japan)
Shinji Kitagami (Fukui University of Technology, Japan)
Takaaki Umedu (Shiga University, Japan)
Takayasu Yamaguchi (Akita Prefectural University, Japan)
Takaya Yuizono(Japan Advanced Institute of Science and Technology)
Takuya Yoshihiro (Wakayama University, Japan)
Tetsushi Ohki (Shizuoka University, Japan)
Tetsuya Shigeyasu(Prefectural University of Hiroshima, Japan)xSession 1:
Systems and Applications 1(Chair: Katsuhiko Kaji)International Workshop on Informatics ( IWIN2023 )A Study on a Power Interchange Method for Realizing Net-Zero Energy
for Multiple Small Communities
Masashi SAITO*, Tomoki NOMURA** and Yuichi Tokunaga**Faculty of Graduate School of Engineering, Kanazawa Institute of Technology, Japan**
Graduate School of Engineering, Kanazawa Institute of Technology, Japan{msaito, y.tokunaga}@neptune.kanazawa-it.ac.jp,
c6101950@planet.kanazawa-it.ac.jp
Abstract - We have developed a simulator to achieve netzero energy through power sharing among multiple
households and communities. In multi-household
transactions, when the capacity of one household's
electricity storage system is depleted and power supply
becomes necessary, power can be purchased from otherhouseholds to meet demand without having to purchasepower from the grid, thus enabling power operations that do
not rely on grid power. In transactions the scale oftransactions is extended to multiple communities, a certain
increase in the renewable energy consumption rate isconfirmed even in communities with low storage battery
capacity and photovoltaic power generation capacity
through the inter-community exchange of electricity byscattering large-scale facility households with different
electricity consumption styles. In this study, only data fromthe Hokuriku region is used for electricity consumption andsolar power generation. By replacing these data, thefeasibility of net-zero energy can be verified for all regions,
showing that a wide variety of simulations in social systems
is possible.

rising, store the surplus in storage batteries without selling itto the grid, and discharge the stored electricity at night.
A house that focuses on self-consumption is similar to a
net zero energy house (henceforth ZEH). ZEH is a housethat aims for zero primary energy consumption with thethree characteristics of self-consumption, thermal insulation,and energy saving[2]. The use of air conditioners is reduced
by improving the insulation performance of the house, andthe entire house is made energy efficient by high-efficiency
equipment and HEMS (Home Energy Management System).
Since the electricity consumption is low, a combination of
PV power generation and storage batteries is expected to beeffective.
Storage batteries play an important role in realizing ZEH.
The price of storage batteries is on a downward trend, but
the price decline will slow down when the cost of raw
materials for lithium-ion batteries is taken into account.
Instead of installing storage batteries for home use, storage
batteries installed in EVs could be used as storage batteries
for home use. If EVs are the vehicles used on a daily basis,
this is V2H (Vehicle to Home) which links EVs andhomes[3]. The storage batteries of EVs that have reached
the end of their service life could be used as a substitute forhousehold storage batteries. In terms of used EVs, EVs thatwent on sale around 2010 have reached the end of theirservice life and are beginning to appear on the market.
Furthermore, the unstable global situation is causing
electricity prices to skyrocket. According to the Agency for
Natural Resources and Energy, the Japanese electricity
market, which was at about 8 yen/kWh until September
2021, recorded about 26.2 yen/kWh in March 2022[4].
Against this background, a shift to distributed energy
systems using renewable energy sources that do not dependon existing centralized energy systems is being promoted[5].
The operation of distributed energy systems is expected torealize "net zero energy".
Smart grids and microgrids are effective powertransmission systems for building distributed energy
systems. A smart grid is a power transmission system thatenables control of the electricity flow not only on the supply
side but also on the demand side, and this system makes it
possible for consumers to buy and sell electricity from the
supply side or between consumers. A microgrid is anelectric power network in which a smart grid is deployedwithin a certain region and energy is produced andconsumed locally within the community without relying on
external supplies. Microgrid operations that do not rely onelectricity sales from suppliers can be expected to secure

Keywords: Smart Community, Electric Power Management,
Net Zero Energy House, Renewable Energy, Simulation

1 INTRODUCTION
The introduction of renewable energies in Japan has
rapidly increased with the introduction of the feed-in tariff(FIT) program, which began in November 2009, requiring
electric power companies to purchase electricity generated
from renewable energies (solar, wind, hydro, geothermal,
and biomass) for a certain period at a price set by thegovernment. For residential PV (residential photovoltaic
power generation) of less than 10 kW, surplus power after
self-consumption is eligible for purchase for 10 years. The
system will gradually expire from November 2019.
According to the Agency for Natural Resources and Energy,
the total number of residential PV systems that will expire
under the FIT will be 730,000 (2.82 million kW) in 2020
and 1.65 million (6.7 million kW) in 2023[1].
The price of electricity sold during the FIT period was 48
yen/kWh in FY 2009. After the end of the FIT, the price will
be about 8 yen/kWh, and compared to the purchase price ofabout 20 yen/kWh, it is considered more economical toconsume the electricity generated on site. For this reason,
attention is focused on self-consumption houses thatgenerate electricity during the daytime when the sun is3

International Workshop on Informatics ( IWIN2023 )electricity and heat in the event of a disaster and to reduce
electricity costs.
In addition, the need to transmit electricity over long
distances in rural areas, including mountainous regions ofJapan, has raised concerns about the rising cost of
infrastructure maintenance. By operating a microgrid andbuilding an independent power infrastructure zone using
renewable energy generation within the surrounding area, it
is possible to operate power without relying on an external
power supply even when the area is remote from urban areas,
eliminating the need to manage the infrastructure of acentralized energy system. As a result, infrastructure
maintenance costs can be significantly reduced [6].
In this study, we have first developed a simulator thatassumes the installation of a PV power generation andstorage system in a house for the realization of ZEH andevaluate the economic feasibility by changes in the amountand cost of the system installation. Next, assuming a smallscale community in a mountainous area, we demonstrate thefeasibility of independent power operation and economical
power management only in the community and its
surrounding area when a microgrid is actually constructed,
and discuss the evaluation results using the simulator.

operation plan of a microgrid by calculating the operating
reserve based on the forecast error trends of power demandand PV power output. The validity of the proposed
framework is verified through numerical simulations. The
demand-supply operation planning problem is formulated asa demand-supply operation planning problem for a general
microgrid. In addition, they defined a demand-supply
operation planning problem that considers uncertainty byusing the probability distribution of the net demand forecast.
In the simulation, the operational cost for the forecasted
value of net demand resulted in worse solution resultscompared to the conventional method. On the other hand,the solution succeeded in reducing the expected value of
operational cost by approximately 20% compared to theconventional method.
Komiyama et al. [10] evaluated the optimal combinationof power generation facilities, considering the introductionof renewable energy and inter-regional power transmission.
Wind power generation facilities in Japan are concentrated
in regions such as Hokkaido and Tohoku, and there is a needto develop inter-regional power transmission lines.
Therefore, an optimal power generation model wasdeveloped as a linear programming model that considers thetransmission of power using inter-regional transmission
lines. As a result, it was shown that wind power can besupplied to the Tokyo metropolitan area, where solar poweris the main source of renewable energy supply, by
expanding the regional power grid. It was also shown thatthe introduction of more PV power generation facilities in
the Kyushu and Shikoku regions, which have long hours ofsunlight, should be promoted.

2 RELATED WORK
Ishikawa and Matsuo [7] conducted a simulation using
storage battery capacity, connection and charge/discharge
control methods, and water heater boil-up time asparameters to evaluate the primary energy savings andeconomic efficiency of installing storage batteries in PV
equipment in post-FIT homes. The best results wereobtained by shifting the boiling time of the heat-pump water
heater without installing the storage battery system, which
resulted in an annual profit of 0.37 million yen and areduction in primary energy consumption of 14 MJ (3.9
kWh). The paper also argues that if 3 kWh of storage
batteries are installed at a unit price of 8 yen/kWh, one of
the following conditions must be met: the storage battery
installation cost must be less than 150,000 yen, thedurability period must be extended to 28 years, or theelectricity price must rise to 186% of the current price.
Yabe et al. [8] evaluated the economic efficiency ofstorage batteries by determining the changes in the selfconsumption rate and the annual cost before and after the
introduction of storage batteries for each consumer based onelectricity demand data and actual solar power generation
data for residences and businesses with solar power
generation throughout Japan. The study concluded that a
5kWh storage battery at 60,000 yen/kWh can be installed ina house, and that the investment can be recovered in about
15 years. The payback period is about 11 years whenelectricity rates with late-night starting times are selected
and varies depending on the demand characteristics andother factors. For commercial customers, the payback period
is about 8 years when a 60,000 yen/kWh storage battery
with the PV power generation output for 30 minutes isinstalled.
Goto et al. [9] proposed a framework and a solution
method for the problem of determining the supply-demand

3 DEVELOPMENT OF ZEH ECONOMIC
EVALUATION SIMULATOR
3.1 ZEH Model
This section discusses a house in which a PV power
generation system and an energy storage system areinstalled. Electricity is supplied to the house from theamount of electricity generated by the PV power generation
system. If consumption exceeds the amount of electricity
generated, the storage batteries are discharged to make up
for the power shortfall. If the amount of electricity generated
and discharged from the storage batteries is still insufficient,
the remaining shortage is met by purchasing electricity fromthe grid. Surplus power from solar power generation ischarged into storage batteries, and surplus power that cannot
be charged due to full battery is sold to the grid. These
power transfers are controlled through HEMS.
We will also consider the use of electricity consumption
forecasting and PV power generation forecasting. Smart
meters measure residential electricity consumption, and PV
equipment sensors measure power generation and solar
radiation. The power consumption forecast, PV power
generation forecast, and remaining storage battery capacity
can be used to predict future excesses and deficiencies in
power, enabling power control to minimize power costs.

4

International Workshop on Informatics ( IWIN2023 )data is being collected as part of the experiment [11].
Measured solar-generated electricity in 2020 is used for the
power transfer simulation. Since the data is measured at 2second intervals, we calculated the amount of electricity byusing the average value of electricity generated every 30
minutes.

The assumed ZEH is shown in Figure 1. In this study,
the power transfer in Figure 1 is first simulated.

3.4 Electricity consumption data
The electricity consumption of eight detached houses inthe Hokuriku region (Niigata Prefecture) is used to create
electricity consumption data from the energy consumption
data published by the Architectural Institute of Japan [12].
The electricity consumption of the eight houses wasaveraged and aggregated by time and classified into three
categories. Using the consumption of eight houses as thebasic data used in the simulation may lead to a deteriorationin accuracy. In the future, if it becomes possible to obtain or
measure more data, the accuracy of the simulation can be
expected.
Electricity consumption and housing assignments areshown in Table 1.

Figure 1: ZEH Model

3.2 Overview of ZEH Economic Evaluation
Simulator

Table 1: Three distinguished power consumption models

The data used in the simulation, such as the amount ofelectricity consumption and solar power generation, arerecorded in CSV and DB formats. When parameters such as
the amount of installed PV and energy storage systems andinstallation costs are entered into the simulator and executed,
the data is read and the power transfer simulation is started.
Power transfer calculations are performed in every 30minute period over a year. After the simulation is complete,a graph is plotted showing the power transfer and profit/loss.
The resulting data can be downloaded in CSV format.
The function and configuration of the simulator developedin this study are shown in Figure 2.

ModelHouse ID
Night/Midnight: Concentrated use at night or
1,2,6
late at night when the user is away during theday, late-night hot water supply
Evening: Always at home and often used in the
4,5eveningMorning/Evening: Always at home and often
3,7,8
used in the morning and evening
The classified data are averaged and aggregated by week,
day of the week, and time, and scaled with the maximum
value set to 1, and used in the simulation. The simulation
reproduces electricity consumption according to households
by adjusting the size according to a parameter called theaverage monthly electricity consumption rate. The
electricity charges follow the metered B electricity rates of
Hokuriku Electric Power Company.

3.5 Power transfer simulation
Figure 3 shows an overview of the simulation algorithm.
First, a simulator using real data is developed, and later asimulation that takes the electricity supply-demand forecast
into account and a simulation in which storage batteries are
replaced by EVs are implemented.

4 SETTING UP A ZEH SIMULATION
Figure 2: ZEH Economic Evaluation Simulator
System Structure

4.1 Electricity consumption
Each power consumption model is used in the simulation
with the power consumption sized as an average monthly
power consumption charge of 10,000 yen. For the adjusted
power consumption, Figure 4 shows the monthly power
consumption and Figure 5 shows the average daily power
consumption.

3.3 Solar power generation data
An energy management experiment system for distributed
energy is being conducted at the Hakusanroku Campus of
the Kanazawa Institute of Technology, and energy-related

5International Workshop on Informatics ( IWIN2023 )6333 kWh, which is about 11% lower than the annual
consumption per household in the same area. In all cases,
electricity consumption is high in winter and low in spring
and fall.

4.2 Setting up PV power generation andenergy storage systems
We simulate combinations of PV outputs of 5, 10, 15, and
20 kW and storage capacities of 0, 5, 10, and 40 kWh,respectively. The storage capacity of 40 kWh is based on thecase where an EV is used as a storage battery. Degradationof the power generation and storage system is notconsidered in this paper. Table 2 shows the settings of theinstalled system.
Table 2: System parameters
Parameter
Value
Power conversion efficiency
95%
Maximum charge/discharge efficiency
95%
Maximum charge and discharge power
3kW

4.3 System pricing assumptions
The installation cost of the PV system is assumed to be
336,000 yen/kW, the main unit cost of the stationary storage
battery 140,000 yen/kWh, and the main unit cost of the V2Hsystem 493,000 yen [13]. The cost of the EV itself is notincluded. The construction cost of the energy storage system
is assumed to be 336,000 yen per installation. Table 3 shows
the unit installation costs of stationary storage batteries, PV
power generation, and energy storage systems, and Table 4shows the system installation costs in this paper.

Figure 3: ZEH Economic Evaluation Simulator
Algorithm Outline

Table 3: System costs per unit
SystemCost
solar power (generation)
336,000 yen/kW
Stationary storage battery
140,000 yen/kWh
V2H(EV)493,000 yen/unit
Energy storage system
336,000 yen per caseinstallation cost
Table 4: System installed costs in this paper
PV (generation)
Battery system
Spec.
Price(M Yen)Spec.
Price(M Yen)
5kW
1. 68
0 kWh
0
10kW
3.36
5 kWh
1.036
15 kW
5.04
10kWh
1.736
20kW
6.72 40kWh (EV)
0.829

Figure 4: Monthly total power consumption amountThe cost of using an EV for the energy storage system is493,000 yen for V2H plus 336,000 yen for construction,
regardless of the capacity.
Since electricity consumption tends to be low during the
day and increases from evening to night, we adapted
Hokuriku Electric Power Company's Kutsurogi-night 12[14]as the electricity rate plan for our simulation. Kutsurogi
Night 12 is a rate plan with low electricity prices during thenighttime (20:00~8:00). This report considers only the basic
charge (1,650 yen/month) and the electricity volume charge,
and does not include the renewable energy surcharge and theunit price of fuel adjustment. The unit price of electricity
sold is assumed to be 8 yen.

Figure 5: Average daily power consumption

Annual electricity consumption was 5621 kWh. The
annual electricity consumption per household in Hokuriku is

6

International Workshop on Informatics ( IWIN2023 )5 ZEH
ECONOMIC
SIMULATION RESULTS

EVALUATION

5.1 Profits and losses on electricity sales
The difference between the simulated profit/loss onelectricity sales and the conventional electricity rate(120,000 yen) was calculated and used as the profit/loss for
one year. Basic charges and system installation costs are notincluded.
Figure 6 shows the one-year power trading profits andlosses for the evening model.
Figure 7: 15-year profits and losses(Night/Midnight model)
power generation equipment and energy storage systems in
the neighborhood by using electricity exchange among themto identify how electricity sharing resolve the electricity
shortage. The multi-households model can also be used tosimulate diverse and realistic electricity transactions bychanging the electricity consumption patterns and the scaleof the PV power generation andstoragesystemsforeachhousehold. Figure 8 shows theelectricity exchange model for
multi-households’
electricity
exchange transactions.

Figure 6: Profits and losses from electric power trading(Evening model)
The maximum profit was 178,000 yen in the model with
20 kW of PV power generation, 10 kWh of storage capacity,and consumption in the evening. Although profit increases
with increasing PV output, it cannot be said that profit
increases with increasing storage capacity.

Figure 8: Multi-Households Electricity Sharing Model

6.2 Overview of Multi-Households Electricity
Sharing Simulator
We simulate the flexible distribution of electricity among
multiple households by running multiple simulators
simultaneously and having each household disclose its
electricity
statusandtradeelectricity
throughcommunication. In addition, we consider the power sharing
network among multiple households as a single communityand evaluate large-scale power sharing among communities
with different power consumption patterns and power
generation patterns.
In addition to the functions of the ZEH economic
evaluation simulator, the following functions in Table 5 are
implemented to simulate electricity sharing among multiple
households. Details are explained in Section 6.4.

5.2 15-year profit/loss including initial cost
A minimum loss of 26,000 yen was recorded for theconcentrated nighttime model with only 5 kW of PV power
generation, and a maximum loss of 6,365,000 yen wasrecorded for the concentrated morning and evening modelwith 20 kW of PV power generation and 10 kWh of storage
batteries. No profit was obtained, including the initial cost.
The 15-year gains and losses for the nighttime
concentration model are shown in Figure 7.
Under the conditions of this paper, it would be more
economical not to install the system. If the unit cost of
installing a PV power generation system is 330,000 yen, theNight/Midnight model with 5 kW of PV power generation
can reduce electricity rates by 0.4 million yen from theconventional electricity rate plus the basic rate.

6 ELECTRICITY DISTRIBUTION
MULTIPLE HOUSEHOLDS

Table 5: Additional Function
implementation function
Household-to-household
Reproduction of power
communication capability
trading
Algorithm for determining Process for determining
the amount of electricity
content at the time ofduring power sharing
transactionTO

6.3 Electricity calculationelectricity Sharing

6.1 Multi-Households ModelIn this section, we assume a situation such as acommunity with several households equipped with PV

methodforWhen conducting electricity sharing among multiple
households, the amount of electricity that each household

7International Workshop on Informatics ( IWIN2023 )wishes to trade is determined, and simulations are conductedin a form that more closely resembles realistic transactions.
As we explain in Section 3.4, the energy consumption usedin the simulation is classified into three patterns. These
power patterns were mixed in the community, and thesimulator is operated. In addition, a minimum power
configuration that assumes facilities that do not consume
power but only generate, store, and supply power is also
included.

6.4 Multi-Households
Algorithm

Electricity

Sharing

Figure 9 shows the algorithm outline of the multihouseholds’ electricity sharing algorithm. The left process
shows the requester and the right one is the responder.
Households whose remaining battery capacity has been
reduced below a certain level due to electricity consumption
are approached by other households to sell their electricity.
The households with sufficient remaining battery capacity
present their tradable electricity to the electricity-deficienthouseholds. Prospective buyers adopt the optimal power
trading details based on the offers from the responding
households and charge their batteries until a certain capacity
is met or the trading partner ceases to exist. If the battery
cannot be recharged, electricity is purchased from the powercompany to meet the remaining demand.

Figure 9: Multi-Households Power Sharing
Simulator Algorithm Outline

7.2 Multi-households ’and evaluation

simulation results

The simulator is operated from July 6 to 13, when the
supply to other households by PV power generation could
be verified, to confirm that sharing is achieved. Figure 10
shows the electricity demand-supply ratio for Household-1.
On days when PV power generation is high during theday, electricity is supplied from the storage battery inHousehold 2, indicating that peak-time electricity demand
could be met through sharing. However, on many days, thegrid power supply is still relied on.
Table 7 shows the amount of electricity purchased and theconsumption rate of PV power generation compared to thecase without electricity sharing.

7 SIMULATION OF ELECTRICITY
DISTRIBUTION AMONG MULTIHOUSEHOLDS
7.1 A Model Case for Simulation
We set up a household model case for one-to-one
electricity sharing among households in a community andconducted simulations according to this setting.
Table 6 shows the electricity consumption plans andhousehold equipment configurations used in the simulations.
The reason why we choose Morning/Evening andNight/Midnight households is that their average daily power
consumption is complementary.
Table 6: A Model Case Parameters
Case 1
Household-1
Household-2
Electricity
Morning/Evening Night/Midnight
Consumption Plan
Storage battery
55capacity (kWh)PV power
5
5
generationequipment (kW)In this case, we simulate the storage battery capacity andPV generation capacity settings that could realistically be
installed by an average household at this moment.

Figure 10: Household-1 Summer Electricity Supply Ratio
Table 7: Electricity Purchased and PV power
Consumption Rate for Household 1
TwoOnehouseholds
householdSharing
PV power consumption rate (%)
54.8
52.0
purchased from the grid amount
46.3
50.0
(kWh)

8

International Workshop on Informatics ( IWIN2023 )and 9 shows the electricity consumption plans andhousehold facility configurations used in the simulations.

The results show that the consumption rate of PV power
consumption decrease, and the amount of electricity
purchased from the grid increases because of the electricity
sharing. These results indicate that even small-scale
facilities can consume a portion of the demand throughelectricity sharing during the season when PV power
generation can be expected.

Table 8: Community-1 Parameters
Community-1
Household-1 Household-2 Equipment-1

Case 3
Electricity
Morning /Consumption Night/Midnight
Evening
Plan
Battery
capacity
5
5(kWh)Solar power
generation
5
5
(kW)

8 SIMULATION OF MULTI-COMMUNITYELECTRIC POWER SHARING
8.1 Overview of Electricity
between Communities

Transactions

We assume that there are communities with multiple
households in the vicinity to trade electricity. Since theelectricity storage pattern by PV power generation differsfrom community to community (Figure 11), a stableelectricity supply in winter, which is difficult to achieve in asimulation of only one community, can be expected.

a little
15
15

Table 9: Community-2 Parameters
Community-2
Household-3 Household-4 Equipment-2

Case 3
Electricity
Morning /Consumption Night/Midnight
Evening
Plan
Battery
capacity
5
5
(kWh)
PV
generation
5
5
(kW)

8.2 Smart Marginalized Communities
In this study, we define "Smart Marginalized
Community" as a community in a smart grid system that
achieves net-zero energy in mountainous areas and does notrely on external power supply.

a little
15
15

8.4 Multi-Community simulation results andevaluation
The simulator is operated for one year. Figure 12 andFigure 13 show the percentage of electricity demand
supplied by Household-1 and the amount of electricity
supplied by Facility 1 during the period from July 1 to July
31.

Figure 11: Model of inter-community electricity sharing
In this simulation, we are working to achieve a form ofenergy self-sufficiency by producing electricity and heat for
their own consumption, called net-zero energy. This study
aims to "achieve complete net-zero energy by means ofrenewable energy (mostly PV energy) and energy storage
systems that do not rely on external power," andhypothesizes that it is possible to achieve these goals andrealize smart marginal community. We then verify theconfiguration of the smallest marginal community that cansatisfy the conditions and the combination of power
generation and storage systems.
The optimal configuration and changes in the degree ofnet-zero energy realization is also examined when
considering the realization of net-zero energy in regionswith different demand and generation conditions.

Figure 12: Household1 Summer Electricity Supply Ratio
The results in Figure 13 show that household 1 receives alarge amount of electricity from Facility 1 and receives
electricity from Facility 2. Table 10 shows the amount ofelectricity purchased and the consumption rate of renewableenergy by Household 1 compared to the case without
electricity supply.
The consumption rate of renewable energy is significantly
increased, and the amount of electricity purchased from thegrid is also reduced through the sharing power supply
system. In addition to the intra-community transactions, thepower acquired through transactions with power generation

8.3 Model case in two Communities
A model case is then set up for two neighboring
communities to share electricity with each other, andsimulations are conducted according to this set-up. Table 89International Workshop on Informatics ( IWIN2023 )facilities in other communities is considered to have enabledmore stable operation that is not dependent on grid power.

[2] Agency for Natural Resources and Energy, Information
Disclosure on ZEH (Net Zero Energy House) - EnergySaving Housing, https://www.enecho.meti.go.jp/
category/ saving_and_new/saving/general/housing/index03.html, (accessed July 26, 2022).
[3] New Energy and Industrial Technology Development
Organization, "NEDO Renewable Energy Technology
White Paper, 2nd Edition," (2014).
[4] Agency for Natural Resources and Energy, "Recent
Trends in Electricity Supply and Demand andWholesale Electricity Market," https://www.meti.go.jp/
shingikai/enecho/denryoku_gas/denryoku_gas/pdf/046_03_02.pdf, (accessed July 15, 2022).
[5] Agency for Natural Resources and Energy, "Actions
Toward the Formation of Next-Generation Distributed
Energy Systems," https://www.meti.go.jp/shingikai/
enecho/denryoku_gas/denryoku_gas/pdf/046_04_02.pd
f, (accessed July 15, 2022).
[6] Agency for Natural Resources and Energy, "A Guide
for Establishing Regional Microgrids,"
https://www.meti.go.jp/shingikai/energy_environment/
energy_resource/pdf/015_s01_00.pdf, (accessed July20, 2022).
[7] Ren Ishikawa and Hironobu Matsuo, "Evaluation ofEnergy and Economy of Battery System Installation inresidential PV System after FIT," Journal of Japan
Solar Energy Society, Vol.46, No.5, pp.63-70 (2020)[8] Kuniaki Yabe and Yasuhiro Hayashi, "Economic
Evaluation of Increased In-House Consumption of
Photovoltaic Power Output by Storage Batteries," IEEJ
Transactions B (Journal of the Electricity and Energy
Division), Vol. 139, No. 5, pp. 363-371 (2019).
[9] Ryota Goto, Hirotaka Takano, and Hiroshi Asano, "Scheduling Method of Microgrids’ Operation
Considering Uncertainty in Values of Forecasted Net
Load," Transactions of the Japan Society of Energy
and Resources, Vol. 42, No. 1, pp. 11-17 (2021).
[10] Ryoichi Komiyama, Saeko Shibata, and Yasumasa
Fujii, "Analysis on Optimal Power Supply Structure
Considering Output Fluctuation of Photovoltaic andWind Power Generation and Interregional Electricity
Flexibility," IEEJ Transactions on Electric Power
Industry B Vol. 133, No. 3, pp. 263-270 (2013).
[11] Eiji Fukuda et al, "Construction of Energy IoT Data
Collection System (2)", Information Processing 82nd
National Convention, (2020).
[12] Committee for Research and Study on Energy
Consumption in Houses, Architectural Institute ofJapan, Database of Energy Consumption in Houses,
http://tkkankyo.eng.niigata-u.ac.jp/HP/
HP/database/index.htm, (accessed July 1, 2021).
[13] Nichicon Corporation, Product Specifications/EV
Power Station/Grid-connected V2H System,
https://www.nichicon.co.jp/products/v2h/about/specs/,(accessed Jan. 10, 2022).
[14] Hokuriku Electric Power Company, Hokuriku
Electric Power Company Customers at Home (unitprice of electricity), https://www.rikuden.co.jp/
ryokin/minsei.html, (accessed Dec. 3, 2021).

Figure 13: Amount of Electricity Sharing for
Equipment-1
Table 10: Electricity Purchased and Renewable Energy
Consumption Rate for Household 1
InterOne
community
householdSharing
PV power consumption rate (%)
37.3
54.0
purchased from the grid amount
3523
2484
(kWh)

9 CONCLUSION AND FUTURE WORKSIn this study, we have developed a simulator to evaluate
the economic efficiency of ZEH. The simulator is useful foreach household to consider the installation of equipment.
The simulator can be operated through a web interface, so it
can be operated by many people.
If economics is the only consideration when installing PV
power generation and storage systems, it is best to refrainfrom installing PV power generation and storage systems.
In a one-on-one simulation assuming intra-community
transactions, the amount of electricity purchased decrease,and the PV power rate increase when the storage batteries
and capacity of one household are increased, while thedemand during peak hours is only partially compensated by
the sharing of the other household when the size of thestorage batteries and PV generation facilities of bothhouseholds are small.
In the case of transactions in which the scale oftransactions is expanded to multi-communities, by scatteringlarge facility households with different electricity
consumption styles, the surplus electricity of large
households was passed on to small facility households, andas a result, a high renewable energy rate is confirmed.
However, in this study we only simulate very simple use
cases, so that the results are limited. We will set up somemore use cases and evaluate and analyze towards ZEH.
Another challenge for the future is to change theelectricity transaction price, which is fixed within andbetween communities, using an algorithm.

REFERENCES[1] Agency for Natural Resources and Energy, "Actions
toward the end of the FIT for residential photovoltaic
power generation equipment," (2019).

10International Workshop on Informatics ( IWIN2023 )Development of a Disaster Safety Confirmation Collection System using Q-ANPI
Service linked with Web Interface
MISTUKI SANO *, TAKUMA SATO *, MASAO ISSHIKI, and KEIICHI ABE *,*Kanagawa Institute of Technology, Atsugi City, Kanagawa, Japanabelab.sano.0914@gmail.com, abe@he.kanagawa-it.ac.jp

Abstract -In the future, when a wide-area disaster occurs
in Japan, there is a possibility of the loss of electric powerand communication infrastructure. In such a case, Q-ANPI,a service for collecting information on the safety of disastervictims using the Quasi-Zenith Satellite MICHIBIKI, which
can cover the entire country, would be very effective.
However, Q-ANPI currently has the problem that if thecommunication infrastructure is cut off due to a large-scaledisaster, the safety information cannot be registered on the
Q-ANPI shelter management PC from a smartphone owned
by a disaster victim if the dedicated application is notinstalled on the smartphone in advance. To solve thisproblem, this study proposes a method for inputting thesafety information of disaster victims using a Web browser
already installed on their smartphones. In addition, aprototype of the proposed system was evaluated from threedifferent perspectives: a simulated evacuation drill, aquestionnaire survey, and power measurements.

There are several services provided by this Quasi-Zenith
Satellite "MICHIBIKI", and Q-ANPI is one of them . This
study focuses on Q-ANPI. Currently, Q-ANPI[5] provides
two input interfaces for collecting disaster victim
information. One is to download a smartphone application
for registering safety information of disaster victims. The
other is to register information directly from the Q-ANPI
refuge management PC using a keyboard. However, the ”QANPI" currently has the problem that safety information
cannot be registered with the refuge management PC for ”QANPI" from a smartphone owned by a disaster victim
unless the dedicated application is installed on thesmartphone in advance, in the event that the smartphone
communication infrastructure is cut off due to a large-scaledisaster. For this reason, people will also be allowed toregister their safety information directly from the Q-ANPI
refuge management PC (Personal Computer) using akeyboard. In the midst of a large-scale disaster, a great dealof congestion will probably occur if a large number ofpeople register directly on the Q-ANPI refuge management
PC. Therefore, the authors believe that the current Q-ANPI
system cannot fully utilize the advantages of ICT. This
paper describes the proposed technology to solve thisproblem. In addition , we will also propose a method for
operating refuge management PCs with low power
consumption even in the loss of power infrastructure
disruption. We also describe the development of the
prototype system proposed in this paper and the evaluationof the effectiveness of the system from various perspectives,
including an actual simulated evacuation drill using theprototype system and the results of a demonstrationquestionnaire evaluation. In Chapter 2 ,we describe theposition of this study with respect to related studies, andChapter 3 gives an overview of the proposed system and thesystem flow. Chapter 4 evaluates the proposed system, andChapter 5 presents the conclusion.

Keywords: QZSS-MICHIBIKI, Victims information, QANPI service, RMS．

1

INTRODUCTION

In the future, when a wide-area disaster such as the Great
East Japan Earthquake occurs, there is a possibility of theloss of electric power and communication infrastructure. In
fact, when the Great East Japan Earthquake occurred, ICT(Information and Communication Technology) could not be
fully utilized, making it impossible to promptly disseminate
information on disaster victims, etc. [1]. In the Kumamoto
earthquake, it is reported that the number of victims inevacuation centers was managed at night, and the name list
itself did not function because of the heavy comings andgoings in the evacuation centers [2,3]. To solve these
problems, we have developed a Refuge Management
System (hereinafter referred to as RMS) [4]. This RMS [4]utilizes ICT to collect information on disaster victims ineach shelter and to manage the long-term health of each
victim, and can transmit the collected victim information via
simple radio communication, such as amateur radio, as wellas other means. The information collected by this RMS can
also be stored in a USB flash memory and take it out
manually. However, assuming a scenario in which a widearea disaster such as a major NANKAI Trough earthquake
occurs in Japan in the future, ”Q-ANPI”[5], a service forcollecting safety information on disaster victims using the
Quasi-Zenith Satellite "MICHIBIKI", which can cover theentire area of Japan, is considered to be very effective.

2

RELATED WORKSA previous study of disaster victim information
dissemination in response to a wide-area disaster is the RMS
development [4]. This RMS utilizes ICT to collectinformation on disaster victims in each refuge and to
manage the long-term health of each victim and can transmit
the collected disaster victim information. The RMS cantransmit information by simple radio communication andamateur radio, in addition to storing disaster victim
information on USB flash memory devices and manually
taking it outside. However, assuming a wide-area disastersuch as the Nankai Trough earthquake that is expected in the11International Workshop on Informatics ( IWIN2023 )future, simple radios, etc., would not be able tocommunicate over a wide area because their communication
range is too short. For this reason, the authors believe that it
would be effective to utilize a satellite communication
system using the Quasi-Zenith Satellite “MICHIBIKI”,
which can cover the whole area of Japan, as promoted by
the National Space Policy Secretariat, Cabinet Office,
Government of Japan. Therefore, this study focuses on the
Q-ANPI service. In a previous study on the Q-ANPI service,a system[6] is proposed that can collect safety information
on disaster victims more widely than the conventional QANPI system by combining Q-ANPI and ad hoc
communication using smartphones. This proposed system[6]is very useful for solving the problem of not being able toconnect to a shelter management PC when the connection to
the Wi-Fi access point is not known, and for collecting thesafety information of victims outside of the refuge. but thissystem requires the dedicated application to be downloadedin advance.
On the other hand, the current Q-ANPI[5] provides two
input interfaces for collecting information on disastervictims. One is to download a dedicated smartphone
application and register the safety information of disastervictims through the application, and the other is to input andregister the safety information directly from the Q-ANPI
refuge management PC using a keyboard. However, in theevent that the smartphone communication infrastructure iscut off due to a large-scale disaster, the safety information
cannot be registered with the refuge management PC for ”QANPI" from a smartphone owned by a disaster victim
unless the dedicated application is installed on thesmartphone in advance. For this reason, the current system
additionally allows people to register their safety
information directly from the refuge management PC using
a keyboard. Since it is assumed that one such PC will beplaced in each refuge, a great deal of congestion willprobably occur if there are a large number of people
registering directly on the refuge management PC.
Therefore, the authors believe that the current system cannot
fully utilize the advantages of using a smartphone.
Therefore, To solve this problem, the authors propose amethod for inputting the safety information of disastervictims using a Web browser already installed on
smartphones.

3

we also considered that the power infrastructure will be cut
off at the same time during a large-scale disaster, and aimed
to realize a practical safety information collection systemthat is robust against power infrastructure cutoffs by linking
Q-ANPI with a system built using an embedded
microcomputer board that can operate with low power
consumption, instead of building a Web server and DB(Data Base) in the refuge management PCs.

3.1

Overview of proposed system

Figure 1 shows an overview of the Q-ANPI safety
information collection system linked with the web interface
proposed by this research. The web interface proposed bythis research uses a small, low-power-consumption
microcontroller (Raspberry PI 3B+; hereinafter referred toas RPI) with a web server and DB (Data Base). The RPI is
used as an intermediary between smartphones owned bydisaster victims and the refuge management PC. When asmartphone owned by a disaster victim connects to the webinterface from its web browser, a page for inputting thevictim's safety information is displayed. When the safety
information is entered, it is automatically registered in the
DB in the web interface. This collects the safety informationof multiple victims in a refuge. The system then sends theinformation to the refuge management PC in a batch. Whenthe person in charge of the refuge management presses thedata transmission button on the application for the refuge
management PC, the data are transmitted to the server of thecontrol station via the Quasi-Zenith Satellite “MICHIBIKI”.
The system enables relatives of disaster victims to confirm
the safety of the victims by accessing the server of thecontrol station via the Internet. By introducing the Webinterface proposed in this research, the application for the
refuge management PC used in the current Q-ANPI can beinstalled without any modification. In addition, anyone who
has a smartphone with a Web browser installed can input
victim information without having to download the
dedicated application for inputting safety information. To
reduce the power consumption of the proposed Web
interface, we used RPI that can operate with lower power
consumption than PC. The power supply used for the entire
system is a 240Wh battery equipped with an inverter for AC
power output as well as a commercially available
photovoltaic panel, assuming that the power infrastructure iscut off.

PROPOSED SYSTEM

3.2
How to display the safety information
input page from a smartphone

In this study, we decided to use Q-ANPI, which uses the
Quasi-Zenith Satellite “MICHIBIKI” and can cover theentire area of Japan, as a solution to the problem of the lossof communication infrastructure in the event of a large-scaledisaster. However, as described in Chapter 3, it is necessary
to consider a scenario in which the communication
infrastructure is cut off in the event of a large-scale disaster,and the dedicated application for inputting safety
confirmation for Q-ANPI cannot be downloaded, in whichcase the advantages of Q-ANPI cannot be fully utilized. To
solve this problem, this study proposes a method for
inputting the safety information of disaster victims using a
Web browser already installed on smartphones. In this study,

As with Q-ANPI, the system in this study forms a LAN(Local Area Network) with commercially available Wi-Fi
wireless routers. In addition to receiving safety informationfrom smartphones and the Web interface, the refuge
management PC serves as a gateway to the Q-ANPI
terminal (920 MHz). For the connection to the web interface
from a smartphone, the SSID of the Wi-Fi wireless router(in this case, "QANPI_179") was made open to allow
connection to the network without a password. A printout of
the URL of the safety information input page converted into12

International Workshop on Informatics ( IWIN2023 )3.4
How to operate the power supply of the
proposed system

a QR code was prepared in advance. By holding the QR
code over the camera of a smartphone, anyone can easily
display the safety information input page on the Webinterface.

3.3

Assuming that power and communication infrastructures
are cut off in a large-scale disaster, power supply isimportant in a real refuge. To operate the proposed system,
it is considered necessary to use one solar panel and two or
more battery units. One battery unit is used for the refuge
management PC, web interface, wireless router, Q-ANPI
terminal, etc. The other battery unit is charged withelectricity generated by the solar panel, and the batteries
should be exchanged alternately. The Q-ANPI terminal andthe refuge management PC consume a lot of power, so when
they are not transmitting data through the Quasi-Zenith
Satellite “MICHIBIKI”, the Q-ANPI terminal can be turned
off and the refuge management PC can be put in sleep mode
for long-term operation.

Prototype system of web interface

Figure 2 shows an overview of the prototype of the webinterface proposed in this paper. The web server was built
on RPI using nginx, and MariaDB was used for the DB. The
reason for using nginx for the web server is to achieve highprocessing speeds that allow multiple inputs of victim
information at the same time. The application for the victim
safety information input interface shown in Figure 2 wasdeveloped in PHP and HTML languages. The "disastervictim information transmission application," whichtransmits the safety information of disaster victims
registered in the DB server to the refuge management PC,
was developed in python 3.7, and is an application thattransmits all safety information registered in the DB. Since
the application for the refuge management PC automaticallydiscards duplicate information, we decided to use a simple
application that sends all DB data at once. Figure 3 shows
the appearance of the prototype developed this time. This is
a prototype with a web server and DB (Data Base) built on
RPI.

4
4.1

DEMONSTRATION EVALUATION
Demonstration evaluation of the system

We evaluated the effectiveness of the prototype of thesystem proposed in this study from three perspectives: ademonstration evaluation based on a simulated evacuation
drill, a questionnaire evaluation based on a demonstration,
and a measurement evaluation of the prototype's power
consumption. In the evaluation through a simulated
evacuation drill, we conducted a questionnaire to determine
whether the proposed system is practically usable, andcompared it with the conventional Q-ANPI system todetermine how easy it is to input safety information. In
addition, a questionnaire survey was conducted by
exhibiting a prototype demo to investigate the ease of use of
the proposed system and find points to be improved. We
also evaluated the power consumption of the prototype
system using the Web interface proposed in this study.

Figure 1: Overview of Proposed System.

4.2

Simulated evacuation drill

A prototype of the proposed system was utilized andevaluated through a simulated evacuation drill. The
simulated evacuation drill was conducted on August 5, 2022,
on the premises of the Kanagawa Institute of Technology.
Figure 4 shows the evacuation route of the simulated drill.
Table 1 shows the schedule of the actual evacuation drill
and the actual action time required. Before the simulated
evacuation drill, one Q-ANPI terminal, one refuge
management PC, one web interface system, one solar panel,and one battery with 240Wh capacity were placed in theformer gymnasium of the university, which served as theevacuation site. It was assumed that a major earthquake
occurred at 14:00. All participants in the evacuation drill
initially hid under desks, and after the earthquake subsided,
they descended the stairs from the classroom (E602) on the
6th floor of Building C2 to the first floor, and then followed
the evacuation route shown in Figure 4 to the entrance of theuniversity's soccer field. After taking roll call at theevacuation assembly, all the evacuees were moved to the old

Figure 2: Overview of Web Interface.

Figure 3: Prototype of Web Interface.

13

International Workshop on Informatics ( IWIN2023 )9 minutes, which indicates that the evacuation wascompleted fairly quickly. The time required to open theevacuation site was about 8 minutes. In the schedule shownin Table 1, attention should be paid to the "process of
inputting disaster victim information.” The proposed system
required only about 3 minutes, while the conventional QANPI system required about 11 minutes. The proposedsystem took only about a quarter of the time of theconventional system. Factors that caused the conventional
Q-ANPI system to take such a long time include the timerequired to download the dedicated application and the timerequired to establish a Wi-Fi connection. The proposedsystem uses a QR code for easy access to the safety
information input page, which is a major factor. Therefore,
the proposed system is superior in terms of the time requiredfor on-site use in actual disasters.

Figure 4: Evacuation route for simulated evacuation drill.

4.3
Questionnaire evaluation after
simulated evacuation
In the demonstration evaluation through the simulated
evacuation drill, a questionnaire survey was conducted after
the simulated evacuation drill, since personal safety
information was collected by two systems, i.e., theconventional Q-ANPI and the proposed system. The actual
number of participants in the simulated evacuation drill was
9, and 6 males (67% response rate) responded to thequestionnaire. Figure 6 shows the results of thequestionnaire. To Question 1, "In the event of a large-scaledisaster, would you like to actually use the proposed systemused this time, which uses a browser for data input, in an
evacuation center?", three respondents answered "Yes, I
would like to use it," and three respondents answered "Yes, I
would like to use it somewhat.” This resulted in a highaverage score of 3.5 on the 4-point scale in which thehighest score of 4 was given to "would like to use" and thelowest score of 1 was given to "would not like to use". Next,
to Question 2, "Which system was easier to use, theconventional system or the proposed system?”, 83% of the
respondents answered that the proposed system was easier
to use. Next, in response to Question 3, "In Question 2, what
was easy to use about the conventional system?”, the answer
was "Because there is no need to search in a browser.” In
Question 4, "In Question 2, what was easy to use in the
proposed system?”, five respondents answered "The factthat it can be used without downloading a dedicatedapplication." To Question 5, "Is there any other information
on disaster victims that you think is necessary besides theexisting information (phone number, name, date of birth,
gender, admission status, availability, address, injuries,
assistance needed, disabilities, pregnant women)?”, One
respondent answered, "Information such as pre-existing
medical conditions, family doctor, and medications used ona regular basis would be helpful.” Next, to Question 6,"What do you think needs to be improved in the future
regarding the proposed system? As shown in Figure 6, themost common response to this question was "Type ofevacuee information" (five responses), followed by "Ease of
text input" (two responses).

Figure 5: Overview of simulated evacuation drill.
Table 1: Schedule for the Demonstration Experiment.

gymnasium, which was the evacuation site, where threeevacuation supervisors set up a proposed system and opened
the evacuation desk (Figure 5). Table 1 records the timefrom immediately after the earthquake until the equipment
was removed. The time required from the occurrence of amajor earthquake to the gathering at the evacuation site was14

International Workshop on Informatics ( IWIN2023 )available 240Wh battery, it is estimated that the RPI alone
can operate for about 3.3 days. A more power-saving
operation could be achieved by using the RPI as a standalone device without the LCD display screen, with theterminal software for the RPI installed on the refuge
management PC so that the RPI's "disaster victim
information transmission application" could be run remotely.
It is also considered a good idea to turn on the power of the
Q-ANPI terminal and the refuge center management PC
only when communicating via the Quasi-Zenith Satellite“MICHIBIKI”, and turn them off and in sleep mode for the
rest of the time. Assuming such an operation scenario, only
the RPI unit (3W) and the wireless router (2.5W) would be
used in constant operation, so the 240Wh battery would lastapproximately 1.8 days. If the two battery units are operated
alternately, long-term operation is possible even when the
power infrastructure is cut off.
Table 2: Power consumption of each device.

5

CONCLUSION

This paper addresses issues related to Q-ANPI, a safety
confirmation collection service for disaster victims using the
Quasi-Zenith Satellite "MICHIBIKI". It describes how tosolve the problem that when the cell phone communication
infrastructure is cut off, safety information cannot beregistered from a smartphone owned by a disaster victim to
the refuge management PC used by Q-ANPI unless a
dedicated application is installed on the smartphone inadvance. We proposed a method to solve this problem byusing a web interface that enables the input of safety
information of disaster victims using a web browser alreadyinstalled in their smartphones. This web interface collects
the safety information of multiple victims and sends it to the
refuge management PC at any given time, which then sends
the data to the control station server via the Quasi-Zenith
Satellite “MICHIBIKI”. The system is a general interface
that allows safety information to be input using a Web
browser already installed on a smartphone. The evaluationof the prototype through simulated training showed that thetime required to input safety information was shorter withthe proposed system than with the conventional Q-ANPI. In
a questionnaire survey conducted after the simulation
training, many respondents answered that they would prefer
to use the proposed system rather than the conventional QANPI system, and many of them answered that the proposedsystem was “easier to use.” However, some respondents
pointed out the following points for improvement of the
proposed system: "Ease of text input" and "Types of victim
information.” In the event of an actual large-scale disaster, it

Figure 6: Questionnaire results for simulated evacuation
drill.

4.4
Measurement and evaluation of
prototype power consumption
Table 2 summarizes the results of power consumption
measurements for each device used in the proposed
prototype system. It can be seen that the Q-ANPI terminal
consumes the largest amount of power (15W) among thedevices used in the prototype system. The second highest is
the refuge management PC, which has a maximum power
consumption of 12 W. The third highest is the RPI, and thefourth highest is the wireless router. The RPI alone
consumes approximately 3 W, which is a significantly low
power consumption. Considering the use of a commercially15

International Workshop on Informatics ( IWIN2023 )can be assumed that far more people will access the systemthan in this simulated evacuation drills. For this reason, weplan to conduct load tests with a larger number of people
than in this simulated evacuation drills. The prototype of the
proposed web interface was developed using RPI. Powerconsumption was measured, and it was found from thisevaluation that the power consumption was low among thedevices used. Therefore, for the sake of power conservation,
the RPI can preferably be used alone, in which case theproblem of the screen display can be solved by remotely
controlling the RPI through terminal software installed on a
refuge management PC.
In the future, if the proposed system and Q-ANPI's
previous research [6] are all successfully linked, it will bepossible to develop a highly realistic system that cansmoothly collect safety information of disaster victims at
actual disaster sites where people are in confusion.

ACKNOWLEDGEMENTS
We would like to express our sincere appreciation to the
National Space Policy Secretariat, Cabinet Office,
Government of Japan, and the Space Systems Division of
NEC Corporation for their guidance and cooperation inproviding the Q-ANPI service equipment.

REFERENCES[1] Reconstruction Agency, Cabinet Office (in charge ofdisaster prevention), and Fire and Disaster
Management Agency: "Number of deaths related toearthquake disaster in the Great East Japan
Earthquake",<http://www.reconstruction.go.jp/topics/
main-cat2/sub-cat26/20171226_kanrenshi.pdf>,(Referred:2023-06).
[2] Ai Urano, Executive Director, Rescue Stock Yard
(RSY): "Actual conditions and issues of shelter
management in the Kumamoto earthquake comparison with shelter management in pastearthquakes."<https://www.isad.or.jp/pdf/information_
provision/information_provision/h28/H28_dai3bu1.pdf> , (Referred:2023-06).
[3] Ministry of Land, Infrastructure, Transport andTourism, "Issues and Discussion Points in Light of the
Kumamoto Earthquake.",<https://www.mlit. go.jp/common/001135911.pdf>, (Referred:2023-06).
[4] K. Akasaka, T. Namba, M. Isshiki, K. Abe,
"Proposal on Victims Information Management
System", International Journal of Informatics Society(IJIS), Vol.12, No.1, pp.17-27, Aug. (2020).
[5] Cabinet Office, Government of Japan, "Q-ANPI
Satellite Safety Confirmation Service.", <https://qzss.go.jp/overview/services/sv09_qanpi.html>,(Referr
ed:2023-06).
[6] N. Ohuchi, M. Nishiura, S. Aikawa, A. Yumura, T.
Hamada, "Disaster Safety Confirmation Collection
System Using QZSS Michibiki and Smartphones",
The Institute of Electronics, Information andCommunication Engineers, Technical Report,
Vol.119, 266, pp.27-32, Nov. (2019).

16

International Workshop on Informatics ( IWIN2023 )River Water Level Prediction Using Radar Rainfall with Deep Learning
Futo Ueda† , Hiroto Tanouchi‡ , Nobuyuki Egusa‡ , Takuya Yoshihiro‡†

Graduate School of Systems Engineering, Wakayama University, Japan‡
Faculty of Systems Engineering, Wakayama University, Japan{s246034, tanouchi, egusa, tac}@wakayama-u.ac.jp

Abstract - Accurate prediction of river water levels duringﬂoods is important to help residents make appropriate evacuation decisions and is the subject of much research. Traditional river water-level predictions are based on physical models. Recently, several studies reported that deep-learning-basedmodels achieve more accurate water-level predictions. Existing deep learning-based water level prediction methods predict downstream water levels based solely on rainfall and water levels at upstream observatories. However, in many cases,
there is not a suﬃcient number of upstream observatories tomake accurate water-level predictions.
In this study, we propose a river level prediction model basedon deep-learning models, CNN and LSTM, with radar rainfall
data. We predict the river levels up to several hours after thecurrent time with the proposed model that uses the historicalwater levels at the prediction place and the radar rainfall data,
without using the data of upstream observatories. The evaluation results showed that the proposed model using the historical water level at the prediction place and the radar rainfall inthe basin can predict water levels with a small error, even if it
does not use the data of the upstream observatories.
Keywords: river water level prediction, deep learning, radar
rainfall, convolutional neural network(CNN), long short-term
memory(LSTM)

1 INTRODUCTION
Because Japan has a lot of mountains, there are many rapidﬂowing rivers. Since river water-level in Japan rises rapidly
in a short period of time, it is important to evacuate quickly incase of heavy rainfall. In particular, elderly people, who may
take a long time to evacuate, need to evacuate earlier. In addition, the recent climate change leads to frequent and short-time
heavy rainfall. Because the short-time heavy rainfall rapidly
rises river water levels, the time to ﬂood became short andthere is a higher risk of extensive ﬂooding damage.
For this reason, river water-level prediction has been actively studied as a disaster countermeasure. Accurate prediction of river water-levels during heavy rainfall is important
for residents to make appropriate evacuation decisions. Traditional river water-level predictions are based on physical models that represent the rainfall-runoﬀ process [1][2]. Recently,
several studies reported that deep-learning-based models achieve
more accurate water-level predictions [3]-[10]. These studies typically learn models from the rainfall and the water-levelmeasurements at upstream observatories to predict downstream
water levels.

17

However, rainfall measurements at observatories don’t include all rainfall information in the river basin. Using all rainfall information in the river basin will improve accuracy to predict river water levels. In addition, in many cases, there is nota suﬃcient number of upstream observatories to make accurate water-level predictions. Many small rivers in Japan do noteven have a single observatory upstream. Typhoon Lionrockin 2016 caused ﬂooding in Komoto River in Iwate Prefecture
where there is only one water level observatory, resulting in
numerous human casualties [11]. Thus, highly accurate riverwater-level prediction even in small rivers is required. Even
when there is no upstream observatory, highly accurate riverwater-level prediction enables early evacuation even in small
rivers.
As other rainfall data sources, radar rainfall observed by
Japan Meteorological Agency [12] is available in Japan. In
this data, rainfall strength values are observed at 1 km meshwhole of Japan based on radar reﬂections. Unlike rainfall
at observatories, radar rainfall contains rainfall information
for the entire watershed because it has been observed at 1km
mesh. So even when there is no rainfall observatories at theriver watershed, we can obtain all rainfall information in theriver basin by using radar rainfall. Water-level prediction method
using radar rainfall with deep-learning-based models has been
proposed [10], but it also requires a number of rainfall andwater-level observatories upstream, and no method without
using upstream observatories has been proposed in the literature.
In this study, without using rainfall and water-level observatories upstream of a prediction place, we predict river waterlevels up to several hours after the current time using only historical water-level at the prediction place and the radar rainfall
in the basin. We achieved highly accurate water-level prediction without using upstream observatories, which shows thatwater-level prediction is possible even for small rivers with
only one water-level observatory.
This paper is organized as follows. In Section 2, we present
related work. In Section 3, we describe the proposed method.
In Section 4, we evaluate the proposed method by using actual
measurements of the river. Finally, in Section 5, we summarize this study.

2 RELATED WORKIn Japan, Hitokoto et al. proposed a model for river waterlevel prediction using deep learning, which showed that it can
predict the river water level more accurately than the conventional prediction methods that are based on physical models[8]. This prediction model is built with MLP (Multi Layer

International Workshop on Informatics ( IWIN2023 )Perceptron), and predicts the downstream water level value by
using the rainfall and the water-level data at observatories upstream of the prediction place. Yamada et al. showed that
LSTM (Long Short-Term Memory), which is a recurrent neural network, is capable of predicting river water-level more accurately than the conventional MLP models [9], meaning that
LSTM can learn chronological characteristics of the waterlevel and the rainfall observation values. However, because
these methods use only the rainfall and the water level of theobservatory upstream of the prediction place, they do not include all rainfall information in the river basin.
Araki et al. proposed a model based on CNN (Convolutional Neural Network), which uses radar rainfall data in addition to the rainfall and the water level observation values upstream of the prediction place [10]. Each channel of the CNN
layer in this model is each time point of radar rainfall measurements. Each of the rainfall and the water level in each observatory forms one independent channel, and adjusts the matrix
size by ﬁlling the time-series values into the matrix to makethe same-size matrix as the radar rainfall mesh. These data are
input to the CNN to predict river water level. However, in thismethod, since each channel contains the time-series rainfall
or the water level values, it cannot learn chronological characteristics directly. Furthermore, these deep-learning-basedmodels depend on a number of observatories upstream of theprediction place.
In a similar study using the radar rainfall data, Kimura etal. proposed a model for predicting the water level in sewer
culverts [13]. In this model, radar rainfall information is input
into CNN, and the water level at close observatories of theprediction place are combined and input into fully connected
layers. The output of the fully connected layer is the prediction
value. This study shows that using radar rainfall can predict
more accurately than without using radar rainfall. However,this study predicts the water level in sewer culverts, so it diﬀers
from river water level prediction.

results are compiled through LSTM, and a water-level prediction value is ﬁnally output through the fully connected layers.

3.2 Convolution Layer
The ﬁrst layer of the prediction model is the convolution
layer. In the convolution layer, we convolute the radar rainfall
data, and compress its information without losing the areal
rainfall information of the radar rainfall data. We use ReLU
as an activation function.
Convolution and pooling layers are built for each LSTM
step. Rainfall and water-level observation values are at hourlyintervals, whereas the radar rainfall data is at 10-minute intervals. Thus, one step of LSTM corresponds to a set of one
hour of radar rainfall data including 6 time-points of the 10minute-interval data, where each channel of the convolution
and pooling layer is each time point of the radar rainfall data.
Because 10-minute intervals of data are for 1 hour, there are
6 channels. As we will discuss later, a target watershed is divided into 40 × 30 areas with 1km mesh. Therefore, the inputof radar rainfall data into the convolution layer at each LSTM
step is 40 × 30 × 6 tensors.

3.3 Pooling Layer
The second layer of the prediction model is the poolinglayer. In the pooling layer, the rainfall information compressedat the convolution layer processes pooling, and summarizes information without losing the areal rainfall information of radar
rainfall data, and reduces size. Those output values are input
to LSTM as compressed information of the radar rainfall. In
the proposed method, the pooling process uses max pooling.
Because max pooling can extract the maximum values for eachdivided area, we expect that the model can learn the local rainfall intensity of radar rainfall.

3 PROPOSED METHOD
3.1 Overview of the Proposed Prediction
Method

3.4 LSTM Layer

In this study, we propose a deep-learning-based model forriver water prediction using radar rainfall data. Radar rainfall
data are input to convolution and pooling layers, compressing rainfall information. The compressed rainfall information,and measurements (rainfall and water level values) of the observatories upstream and the prediction place, are input to the
LSTM layer. Each time step of the LSTM layer is an hourlyinterval because the interval of observatories’ measurements
is one hour. Thus, we combine radar rainfall data, which is
10-minutes interval, into hourly time series, and input theminto CNN and each LSTM step. Figure 1 shows a diagramof the proposed river water-level prediction model. In Fig. 1,
the left side is the input, and the process proceeds to the right
direction, to output at the right side. Each row represents theprocess for a single time-point of the time series, and an onehour dataset is processed in the convolution and pooling layers
and is input to one step of LSTM. The each step, the learning

The ﬁnal layer of the prediction model is the LSTM layer
and the MLP layer. In the LSTM layer, rainfall information,
which is compressed by the convolution and pooling layer, andthe observatories’ observation values are input to the LSTM
layer, and learn information on each step in chronological order. In the last step of the LSTM layer, outputs of the LSTM
layer are input to a fully connected MLP layer, which outputs
the water-level prediction value. As we have already noted,
each time step of the LSTM layer is an hourly interval, andradar rainfall information that is compressed for one hour. Also,
the rainfall and water-level observation values are measured
for one hour. Those hourly data are input to each LSTM step.
Furthermore, the number of LSTM steps is determined by the
number of historical points used in prediction. Speciﬁcally,in this study, because we use 11 hours historical data to learn
models, the LSTM layer is built with 11 time steps.

18

International Workshop on Informatics ( IWIN2023 )Figure 1: Proposed Prediction Model

4 EVALUATION

Japan. There are 5 water level observatories and 12 rainfall
observatories. The locations of these observatories are shownin Fig. 2. In this area, there are no ﬂood control facilities upstream of the Hiwatashi point, which is the prediction place.
There are a lot of water level and rainfall observatories upstream.

4.1 Evaluation Method
In this study, we predict the water level using river proposedmodel, and show that it is possible to predict it with high accuracy even when we do not use rainfall and water-level observatories upstream of a prediction place. Speciﬁcally, we
compare the following ﬁve prediction models.
A. Prediction model with MLP, which uses rainfall and water level data obtained by the observatories located at
upstream and the prediction place.

4.1.2 The Dataset
We acquire water level and rainfall data, which are hourly data,from the website of the hydrology and water-quality database[15]. We acquire radar rainfall data, which are 10-minutesinterval, from Japan Meteorological Business Support Center[16]. Radar rainfall data are observed at 1km mesh whole ofJapan, and we use as input data 40 km long and 30 km wide
regions shown in Fig. 2.

B. Prediction model with LSTM, which uses rainfall andwater level data obtained by the observatories locatedat upstream and the prediction place.
C. Prediction model with CNN and MLP, which uses radar
rainfall data in addition to rainfall and water level data
upstream and at the prediction place.

The prediction place is the Hiwatashi water level observatory, and the target period is the 13 periods from 2006 to 2021
when the Hiwatashi water level observation value is higher
than the dangerous water level 1 (5.4m) [17][18] deﬁned ateach river observatory by the government. (The actual numberof such periods is 14 periods, but we exclude one period because radar rainfall has missing values.) We deﬁne that a such
period is 120 hours long, from 72 hours before to 48 hours after the time of peak water level of Hiwatasi observation value
during the target periods. Among them, we select three evaluation periods for water level prediction (July 3, 2010, September 20, 2011, and July 3, 2019) when the Hiwatashi water level
observation value is higher than the dangerous water level 3(8.3m) [17][18] in the 13 periods. We evaluate the prediction
accuracy of the models A-E by predicting one of the 13 periods with the other 12 periods used as the training data, which
is applied for evaluating the three test periods.

D. Prediction model with CNN and LSTM, which uses radar
rainfall data in addition to rainfall and water level data
upstream and at the prediction place.
E. Prediction model with CNN and LSTM, which uses radar
rainfall data and water level data at the prediction place(without data from upstream observatories).
We train the prediction model using values up to 11 hours
before the reference time for prediction, and predict 1 to 12hours after the reference time. We use Mean Squared Error(MSE) to evaluate the accuracy of our model.
4.1.1 Study Area
In this study, the study area is the Hiwatashi point of the Oyodo River, which is a ﬁrst-class river in Miyazaki prefecture,

19

International Workshop on Informatics ( IWIN2023 )Table 2: The Number of Hidden Units for MLP

No Radar Rainfall (Model A)

Using Radar Rainfall (Model C)

Hidden Units
30-20-1050-30-10
100-50-20
150-100-50
500-300-100
800-400-200
1000-500-100
2000-1000-500
2000-1000-500
3000-1000-500
5000-3000-1000
7000-4000-1000
9000-5000-1000
10000-6000-2000

Table 3: The Number of Hidden Units for LSTM
Figure 2: Study Area
(Created Using GIS Map of Geospatial Information
Authority of Japan [14])Table 1: Structure and Parameters of the ModelClassiﬁcation
Optimizer
EpochError function
Batch size
Language
Library

No Radar Rainfall (Model B)

Parameter
Adam (0.0001)
Early Stopping (50 times)
MSE (Mean Squared Error)
90
PythonPyTorchUsing Radar Rainfall (Model D,E)

4.2 Details of the Prediction Model

Hidden Units
6
10
14
18
50
100
150
200
500
200
300
500
1000
2000
3000

4.2.1 Normalization of Training Data
If training data contain bias, they impose a negative eﬀect onlearning. Therefore, in this proposed method, we normalize
and convert to the range of [0, 1] the training data before learning the model. Normalization is made as shown in the following Eq. (1).
xi − min(x)
,
(1)
x′i =
max(x) − min(x)Table 4: Parameter of CNN
Parameter

Convolution Layer

where x′i is the normalized value of the i-th observation value,
xi is the raw observation value, min(x) is the minimum value,max(x) is the maximum value in the training data. This normalization is applied to all the input data, i.e., water level andrainfall data of the observatories, and the radar rainfall data.

Pooling Layer

Kernel size
Number of Filter for MLP
Number of Filter for LSTM
StrideActivation Function
Classiﬁcation
Window size

Value
3×3
128
6
2
ReLU
Maxpooling
2×2

learning parameter adjuster. To prevent overlearning, we determine the number of epochs by using Early Stopping [20],and stop learning if accuracy does not improve 50 times. We
use stochastic gradient descent (SGD) to learn the network,and use backpropagation to calculate the gradient of the error
function.

4.2.2 Structure and Parameters of the Model
We show the structure and parameters of our learning model inTable 1. We use Adam [19], which is a well-known dynamic

20

International Workshop on Informatics ( IWIN2023 )Table 5: Parameter of the Best Accuracy
Prediction Model

Hidden Units

Number of Layers

Dropout

A. MLP with upstream observatories
B. LSTM with upstream observatories
C. CNN+MLP with upstream observatories and radar rainfall
D. CNN+LSTM with upstream observatories and radar rainfall
E. CNN+LSTM with water level at prediction place and radar rainfall

50-30-10100
9000-5000-1000
500
1000

2
1
1

0.1
0.3
0.1
-

4.2.3 Parameter Evaluation
In this study, we tried a variation of the parameter value foreach of the ﬁve prediction models A-E, and compare the prediction accuracy to determine the optimal parameter values.
For the MLP model (model A and C), we use ReLU as an activation function, and build a ﬁve-layer network consisting of
an input layer, three hidden layers, and an output layer. Table
2 shows the number of hidden units in the prediction model ofusing MLP. Each number in the triplet expresses the numberof neurons of an input, intermediate, and output layer, respectively. The dropout parameter is set as 0.1 or 0.3. We predictwater levels with all combinations of the number of hidden
units and the dropout, and evaluate prediction accuracy.
For the LSTM model (model B, D, and E), we use the number of hidden units shown in Table 3. The number of layers is
1, 2, or 4, and in the case of 2 or 4 layers, a dropout is set as
0.3. We predict water levels with all combinations of the number of hidden units and the number of layers of the LSTM, andevaluate prediction accuracy.
We show the CNN parameter for the prediction model withradar rainfall data in Table 4. For the prediction model with
CNN and MLP (model C), each channel of the CNN is 10minute intervals of radar rainfall data. In prediction modelC, because 10-minute intervals of data are input for 11 hours,
there are 66 channels. Therefore, the input of radar rainfall
data is a 40 × 30 × 66 tensor.

Figure 3: MSE of Each Prediction Modelat Three Prediction Periods

5 CONCLUSION
In this study, we propose a river level prediction model basedon deep-learning models, CNN and LSTM, with radar rainfall
data. We predict the river levels up to several hours after thecurrent time with the proposed model that uses the historicalwater levels at the prediction place and the radar rainfall data,
without using the data of upstream observatories.
The prediction accuracy of the proposed model is evaluatedby comparing it with the models that use the upstream waterlevel and rainfall measurements. The results showed that the
proposed model using the historical water level at the prediction place and the radar rainfall in the basin can predict waterlevels with a small error, even if it does not use the data of
the upstream observatories. This means that water level prediction is possible even for rivers with only one water-level
observatory, which are often the case of many small rivers inJapan.

4.3 Results
4.3.1 Parameter Results
We show the parameter values with the best accuracy for eachof the prediction models A-E in Table 5. For each prediction
model, we predict water levels by using these parameter values
to evaluate the prediction accuracy in the next section.

ACKNOWLEDGEMENT
This work was supported by JST, PRESTO Grant Number
JPMJPR1939, Japan.

4.3.2 Accuracy for the Three Prediction Periods
We show the prediction accuracy of each prediction modelfrom 1 to 12 hours after the reference time for the three prediction periods in Fig. 3. The vertical axis is MSE, and thehorizontal axis is the prediction time. Figure 3 shows that theprediction accuracy in all ﬁve cases is not signiﬁcantly diﬀerent. In particular, the MSE of the prediction model E, which
does not use the measurements of the upstream observatories,
does not increase MSE signiﬁcantly compared to the MSE of
A-D.

REFERENCES[1] Y. Tachikawa, T. Sayama, K. Takara, H. Matsuura,
T. Yamazaki, A. Yamaji, and Y. Michihiro, “Development of a Real-Time Runoﬀ Forecasting System Using a
Physically-Based Distributed Hydrologic Model and its
Application to the Yodo River Basin, ” Natural Disaster Science J. JSNDS, Vol. 26-2, pp. 189-201, 2007 (inJapanese).

21International Workshop on Informatics ( IWIN2023 )[2] Y. Nakamura, T. Koike, S. Abe, K. Nakamura, T.
Sayama, and K. Ikeuchi, “Development of Real-Time
Flood Prediction Utilizing the RRI Model with a Particle Filter, ” Journal of Japan Society of Civil Engineers,
SER. B1 Hydraulic Engineering, Vol. 74, No. 5, I_1381I_1386, 2018 (in Japanese).
[3] S-S. Baek, J. Pyo, and A. Chun, “Prediction of Water
Level and Water Quality Using a CNN-LSTM Combined
Deep Learning Approach,” Water, 12, 3399, 2020.
[4] P. Li, J. Zhang, and P. Krebs, “Prediction of Flow Basedon a CNN-LSTM Combined Deep Learning Approach,”
Water, 14, 993, 2021.
[5] X. Li, W. Xu, M. Ren, Y. Jiang, and G. Fu, “Hybrid
CNN-LSTM models for river ﬂow prediction,” Water
Supply, 22(5), 4902, 2022.
[6] B. Alizadeh, A.G. Bafti, H. Kamangir, Y. Zhang,
D.B. Wright, and K.J. Franz, “A Novel Attention-based
LSTM Cell Post-processor Coupled with Bayesian Optimization for Streamﬂow Prediction,” Journal of Hydrology, 601, 126526, 2021.
[7] C. Chen, J. Jiang, Z. Liao, Y. Zhou, H. Wang, and Q. Pei,
“A Short-term Flood Prediction Based on Spatial Deep
Learning Network: A Case Study for Xi County, China,”
Journal of Hydrology, 607, 127535, 2022.
[8] M. Hirokoto, M. Sakuraba, and Y. Sei, “Developmentof the Real-Time River Stage Prediction Method Using
Deep Learning, ” Journal of Japan Society of Civil Engineers, SER. B1 Hydraulic Engineering, Vol. 72, No. 4,
pp. I_187- I_192, 2016 (in Japanese).
[9] K. Yamada, Y. Kobayashi, M. Nakatsugawa, and J.
kishigami, “A Case Study of Flood Water Level Prediction in the Tokoro River in 2016 Using Recurrent Neural Networks, ” Journal of Japan Society of Civil Engineers, SER. B1 Hydraulic Engineering, Vol. 74, No. 5,
pp. I_1369- I_1374, 2018 (in Japanese).
[10] T. Araki, K. Hakoishi, M. Hitokoto, T. Shimamoto, andK. Fusamae, “River Water Level Prediction by Convolutional Neural Network Using Radar Rainfall, ” Advances in river engineering, Vol. 25, pp. 297-302, 2019
(in Japanese).
[11] Ministry of Land, Infrastructure, Transport andTourism:River Information Policies <http://www.river.or.jp/kougi2020_5_rev.pdf>(accessed2023-06-05).
[12] Japan Meteorological Agency Weather Radar <https://www.jma.go.jp/jma/kishou/know/radar/kaisetsu.html>(accessed 2023-06-05).
[13] Y. Kimura, “Prediction of Water Level in Sewer Pipe Using Convolutional Neural Network, ” Journal of the Society of Instrument and Control Engineers, Vol. 58, No.
6, pp. 449- pp. 453, 2019 (in Japanese).
[14] Ministry of Land, Infrastructure, Transport and Tourism:
Geospatial Information Authority of Japan, <https://www.gsi.go.jp/top.html >(accessed 2023-06-05).
[15] Ministry of Land, Infrastructure, Transport and Tourism:
the hydrology and water-quality database, <http://
www1.river.go.jp/ >(accessed 2023-06-05).
[16] Japan Meteorological Business Support Center, <http://www.jmbsc.or.jp/jp/ >(accessed 2023-06-05).
[17] Ministry of Land, Infrastructure, Transport and Tourism:
List of Water Levels Related to Flood Prevention for
Directly Controlled Rivers, <https://www.mlit.go.jp/
river/toukei_chousa/kasen_db/pdf/2021/12-1-8.pdf>(accessed 2023-06-05).
[18] Ministry of Land, Infrastructure, Transport andTourism: Water and Disaster Management Bureau
Implementation Guidelines for Revision of Disaster Prevention Information System for Floods,<https://www.mlit.go.jp/river/shishin_guideline/
gijutsu/saigai/tisiki/disaster_info-system/ >(accessed2023-06-05).
[19] D. P. Kingma and J. Ba, “Adam: A method for stochastic
optimization, ” arXiv: 1412. 6980, 2014.
[20] L. Prechelt, “Early Stopping - But When?, ” Springer
Berlin Heidelberg, Berilin, Heidelberg, pp. 53- pp. 67,
2012.

22

International Workshop on Informatics ( IWIN2023 )Developing an Information System to Enhance Intellectual Productivity in
Teleworking Environments
KANAE MATSUI†* KATSUMA TAKAGI†***

School of System Design and Technology, Tokyo Denki University, Japan**
Graduate School of Informatics, Tokyo Denki University, Japan{matsui@mail, 21amj12@ms }.dendai.ac.jp

Abstract –The COVID-19 pandemic highlighted thenecessity for efficient telework settings, as intellectual
productivity is closely linked with workplace satisfaction.
Addressing the lack of productivity systems in homeenvironments, we created an IoT tool that uses temperature,humidity, and CO2 measurements to provide relevant
feedback. A two-month trial demonstrated its effectiveness inpromoting behavioral changes, such as using humidifiers,
ensuring its operational reliability.
Keywords: IoT application, Healthcare, Data collection

1changes in thermal
environment and lighting

Both workers and employers observed over an 80% decline
in telecommuting productivity due to communication
challenges, inadequate equipment, and home disturbances.
Environmental factors such as temperature, humidity, andCO2 levels, crucial for productivity, were often overlooked,
with their improvement in the U.S. potentially offering $40$200 billion in economic benefits (Table 1). Consequently,
many firms are now implementing systems like BEMS to
improve workspace productivity.
Additionally, about 70% of telecommuters faced home air
quality issues, underscoring the need for better environmental
conditions and information systems. Despite known benefits,
few companies have optimized home thermal conditions for
productivity. Our research utilized the “Netatmo Weather
Station [2]” IoT device to monitor environmental factors,
leading to an IoT-based system to enhance home work
environments and productivity.

INTRODUCTION

The COVID-19 outbreak led to a significant shift fromoffice to homework, with a survey showing 80% of Japanesefirms implementing telework (Figure 1). However, over 80%
of both employees and employers reported a productivity
drop in teleworking, due to factors like poor communication,
inadequate equipment, and home distractions. Environmental
factors, such as temperature, humidity, and CO2 levels,
though initially underestimated, are proven to be crucial for
productivity [1]. In the U.S., improving these aspects could
result in economic gains of $40-$200 billion annually (Table1), prompting companies to integrate BEMS and smart offices
for better office productivity.

2

RELATED STUDY
2.1 Telework situation in JapanThe Ministry of Internal Affairs and Communications
describes telework as a “flexible approach leveraging ICT for
efficient time and place usage [3].” There are three primary
types of telework based on work location: telecommuting,mobile work, and satellite office work. Due to COVID-19,
telework adoption has surged, with many companies making
home-based telework standard practice [4]. Owing to its
adaptability, telework offers numerous benefits for society,corporations, and employees. Our research specificallyexamines telecommuting within these telework categories.

2.2 Novelty of our study Indoor environment
and intellectual productivity
Fig 1．Telework implementation rate at companies inTokyo (30 or employees) [2]

Hamanaka et al. [5] developed a teaching bot designed to
optimize indoor comfort. Integrated into the chat tool “Slack,”
the bot offers real-time feedback on indoor temperature andhumidity. It sources data from a sensor on the Raspberry Pi 3,
recording temperature and humidity values every minute via
Wi-Fi. Upon querying the bot, users receive
recommendations for enhancing comfort, derived from realtime data and optimal comfort ranges. This bot boasts twocore features: querying the database to generate and sendgraphical data on demand.

Table 1．Economic effect of improving air quality in the
United States
Source of productivity gain
Reduced respiratory illness
Reduced allergy and asthma
Reduced sick building
syndrome symptoms
Improved worker performancefrom

Potential US annual savings$6-$14 billion/year$2-$4 billion/year$10-$30 billion/year$20-$160 billion/year23

International Workshop on Informatics ( IWIN2023 )unlimited storage and API access for data integration into
custom systems.
Our developed platform centralizes data from multiple
Netatmo devices, linked to their respective accounts, ensuring
data consolidation and accessibility. Data is updated every
five minutes and stored in BSON format using MongoDB,
with Heroku as the application server. The development
environment details are in Table 3.

2.3 Novelty of our study
The majority of IoT systems for indoor optimization target
office spaces, with a gap in catering to individual
telecommuters or small teams. Systems like HEMS focus onresidential energy use rather than telecommuting needs. Our
research aims to create a health-conscious, efficient
workspace by developing a recommendation system tailoredfor home telecommuting. This system faces challenges inuniversal adaptability due to the variety of household airconditioning devices. It provides customized information to
improve working conditions based on indoor parameters,
allowing manual adjustments to air-conditioning. The systemdesign prioritizes intuitiveness, with easily understandable
information even during work hours. We will detail thissystem's structure, technical aspects, experimental methods,
evaluation, and our findings in the following sections.

3

Table 3．Development environment of Platform
OS

Server-side
MacOS

Language

Node.js (v10.18.1)

Library
Database

SYSTEM DESCRIPTION

express，ejs，
crypto，Socket.IO
MongoDB

Client-side
MacOS
HTML，CSS，(SCSS)，JavaScript
Vue.js，Bootstrap-Vue，Core
UI，chart.js，Axitos

3.2.2 Information generator
This component analyzes real-time and historical data tocreate insights for an optimal work environment. It generates:
1) Provides immediate values of indoor environmental
factors with feedback for improvement.
2) Displays daily fluctuations in indoor factors usinggraphs, with comments to ensure optimal conditions.
3) Summarizes the previous day’s and week’senvironment suitability, with averages andimprovement suggestions.
4) Offers advice on maintaining or enhancing the indoor
environment.
Our system uses building environmental hygiene control
standards to define ideal workspace conditions (Table 4) [7].

3.1 Overview
Our system uses strategically placed sensors to monitor
workspace conditions, generating notifications formaintaining or enhancing the work environment. Workers canact upon these alerts to foster productivity.

3.2 System configuration
The system, outlined in Fig. 2, relies on IoT devices across
telecommuting setups, sending environmental data to acentral server. This data, both real-time and historical, is used
to create recommendations, delivered to workers as
notifications. Upcoming sections detail the system's design
and functionality.

Table 4．Ideal working environment
Category
Carbon dioxide
concentration

3.2.1 Data acquisition partOur system’s data acquisition relies on Netatmo devices,consisting of a primary indoor module for internal
environment monitoring and a secondary outdoor module.
The primary device sends data directly to Netatmo’s server
via Wi-Fi, while the secondary device transmits data
indirectly through the primary device. Netatmo’s server offers

Temperature
Relative humidity

Fig 2．System overview

24

DetailLess than 1000 ppm
Range: 17℃ to 28℃
If the indoor temperature is lower
than the outdoor, the difference
should be significant.
Range: 40% to 70%
1)2)

International Workshop on Informatics ( IWIN2023 )This presents hourly updates on the current state of theworkspace environment, specifically focusing on temperature,humidity, and carbon dioxide concentration. Relevant
comments are determined based on conditional branching,rooted in the previously defined criteria. Here are the tables
for humidity and CO2.
Table 5 ． Conditional branching and comments inhumidity
ConditionLess than 40%
From 40% to70%More than 70%

Fig 3．Actual condition of information 1

Comment
The room is dry. Dryness increases thenumber of times you blink, causing
discomfort and reducing work efficiency.
It’s recommended to humidify.
Humidity is optimal. Let’s keep thishumidity.
Humidity in the room is high. In a humid
environment, the sympathetic andparasympathetic nerves are out of balance,
making it easier to feel tired. Let’s
dehumidify.

Fig 4．Actual condition of information 2 (CO2)Information of type 2 provides daily visuals and commentson the fluctuation of temperature, humidity, and CO2 from8:00 AM to 7:00 PM. Here are the conditions and comments
for temperature, humidity, and CO2.
This displays how closely the workspace met the building
environmental hygiene standards. It presents the “indoor
environment adequacy rate”, which is calculated using thefollowing formula [A].

guided articles) at set intervals, focusing on work hours
for minimal distraction.
2) Threshold Alerts: Monitors environmental data every
30 minutes and alerts users when humidity and CO2
levels exceed set thresholds, offering concise
evaluations and suggestions.
3) Interactive Chatbot Response: Provides real-time
feedback to users’ queries about their current work
environment. It utilizes the Predicted Mean Vote
(PMV) index to assess thermal comfort, consideringfactors like temperature, humidity, air velocity, andpersonal factors such as clothing insulation andmetabolic rate. This allows tailored advice based on
environmental data from Netatmo and user-specificinputs like clothing type, enhancing comfort andproductivity. The system's approach to measuring andresponding to thermal comfort is detailed, including
predefined clothing insulation levels for accurate PMV
calculations.

𝐼𝑛𝑑𝑜𝑜𝑟 𝑒𝑛𝑣𝑖𝑟𝑜𝑛𝑚𝑒𝑛𝑡 𝑎𝑑𝑒𝑞𝑢𝑎𝑐𝑦 𝑟𝑎𝑡𝑒
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑚𝑒𝑒𝑡𝑖𝑛𝑔 𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑𝑠
𝑑𝑢𝑟𝑖𝑛𝑔 𝑤𝑜𝑟𝑘 ℎ𝑜𝑢𝑟𝑠=
𝑇𝑜𝑡𝑎𝑙 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑑𝑢𝑟𝑖𝑛𝑔 𝑤𝑜𝑟𝑘 ℎ𝑜𝑢𝑟𝑠∗ 100 … [𝐴]The system keeps telecommuters informed about theirworkspace environment, allowing them to make necessary
adjustments for maximum comfort and productivity. Every
day at 12:00, a report is generated highlighting researchfindings on the correlation between indoor environment andproductivity, along with suggested measures to enhance theindoor ambiance in information of type 4. By grounding
suggestions in scientific evidence and providing explicit
action steps, the aim is to foster a heightened awareness andproactive attitude towards the workspace environment amongusers.

When a user mentions the term “working environment” intheir designated channel, the bot prompts them to specify theircurrent clothing [8 and 9]. After receiving the user’s clothing
input, the system incorporates the Predicted Mean Vote
(PMV) value into the data from Presentation Information 1
and then notifies the user accordingly.

3.2.3 Information notification partIn our system, we integrated three key notification functions
using Slack [6] as the communication platform (Figure 3 and
4). Each participant had a dedicated Slack channel forreceiving notifications from the “Indoor Environment
Notification Bot.” The functionalities include:

4

EXPERIMENT

4.1 Overview of the experiment
The experiment aimed to collect indoor environmental data
for telecommuting using Netatmo, and utilize it to enhanceindoor conditions and influence behavior. Effectiveness wasevaluated through interviews before, during, and after the

A feature enabling periodic dispatch of the four types of datapreviously detailed.
1) Periodic Data Dispatch: Sends the four types of data
(current status, daily trends, historical analysis, and25

International Workshop on Informatics ( IWIN2023 )experiment, which lasted 33 days from November 10 to
December 28, 2022.

3

4.2 Participant demographics

4

Three participants from the Kanto region, aged 20s to 50s
and working from home, were involved. Their demographics
are detailed in Table 14, including age, gender, occupation,
residence, and workspace specifics.

5
6
7

Table 5．Participants attributes of the study
A
50’s
male
Engineer

Age
Gender
Occupation
Residence status
Workplace
Size of workplace

Apartment
Study
18 m2

8

B
40’s
male
Technical
sales
Apartment
Living
22 m2

C

9

20’s
Female10

Office work

11

Apartment
Living
14 m2

12
13

4.3 Experimental method

14

4.3.1 ParticipantsParticipants set up accounts on Netatmo and Slack andreceived device kits (Table 7). They installed indoor modules
in their workspaces and outdoor modules in sheltered outdoor
areas.

15
16

17

Table 7．Contents of the experimental kit
No
1
2
3
4
5

Details
Netatmo Weather Station outdoor module
Netatmo Weather Station indoor module5

AC adapter for Netatmo Weather Station indoor
moduleKit bag
Covering letter

EVALUATION
5.1 System evaluation

The system’s effectiveness in enhancing telecommuting
environments was assessed. Notifications were periodicallysent to participants, including alerts for sub-optimal humidity
and CO2 levels. The frequency and types of thesenotifications, both during and outside work hours, are detailedin Table 17. An “information provision rate” was calculated
to measure the system's efficiency in delivering notifications(Table 9).

4.3.2 Pre-experiment questionnaire
A survey, inspired by Daikin Industries [6], assessed
participants' initial perceptions of their telecommuting
environment. It inquired about housing types, environmental
factors, and potential improvements.

Table 9．Number of notifications

4.3.3 Questionnaire during the experiment
Daily surveys via SurveyMonkey asked about work
location, duration, and environment evaluations, focusing ondays spent telecommuting.

A

4.3.4 Post-experiment questionnaire
A post-experiment questionnaire (Table 8) evaluated thesystem's effectiveness and any shifts in participants'
environmental improvement awareness. It included questions
on device placement, installation ease, notification frequency,
and the system's impact on work environment quality.

Notificat
ionfunction

Table 8．Questionnaire after the experimentNo.
12

Please specify the dimensions of the room where thedevice was set up. (Provide the measurement in square
meters.)Did you find the installation process for Netatmo to be
intuitive?
Was the notification frequency of once every hoursatisfactory?
Could you provide further clarification on your answer toquestion 5?
Was receiving a daily summary notification at 19:00
suitable for you?
Please elucidate your response to question 7.
Was the daily 8:00 am summary notification, recapping
the previous day/week, convenient for you?
Please tell me if there is a reason for answering question9.
Was the 12:00 noon notification for the column function
apt for your needs?
Please tell me if there is a reason for answering question
11.
Kindly expound upon your response to question 11.
Which set of indoor environmental data was deemed mostvaluable by you?
In your estimation, did the system enhance the work
environment’s conduciveness?
When telecommuting, are there particular facets of yourhome setting that present concerns? (You may selectmultiple options.)
Looking ahead, are there specific components orresources you’re considering investing in or improving
for an optimized telecommuting experience?

Details
Where did you position the indoor module?
Where was the outdoor module placed?

26

Type ofinformati
on 1Type ofinformati
on 2Type ofinformati
on 3

B

Duri
ng
offic
e
hours

Out
of
the
offi
ce
hou
rs

182

296

C

Duri
ng
offic
e
hours

Out
of
the
offi
ce
hou
rs

Duri
ng
offic
e
hours

Out
of
the
offi
ce
hou
rs

160

349

82

134

20

20

9

13

14

5

International Workshop on Informatics ( IWIN2023 )Alert
function
Chatbotfunction

Type ofinformati
on 4
Humidit
y
CO2
-

13

14

5

42

149

0

0

0

0

94

7

255

537

7

45

0

0

0

0

0

0

To assess the system’s operational efficacy, we compared
the actual frequency of notifications to an ideal count for thedays participants worked from home, deriving a metrictermed the “information provision rate”. The calculation forthis rate, denoted by equation [B], is as follows:
𝐼𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 𝑝𝑟𝑜𝑣𝑖𝑠𝑖𝑜𝑛 𝑟𝑎𝑡𝑒 =∗ 100 … [B]𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑐𝑡𝑢𝑎𝑙 𝑛𝑜𝑡𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝑠
𝐼𝑑𝑒𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑛𝑜𝑡𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝑠

Table 18 encapsulates the information provision rates
deduced using the formula.
Table 10．System availability
Type of information 1
Type of information 2
Type of information 3
Type of information 4

A
84%
83%
100%
100%

B
84%
83%
100%
100%

C
83%
83%
100%
100%

5.2 Questionnaire evaluation
The post-experiment questionnaire assessed participants’adjustments to their work environments, focusing on air
conditioning, humidification, and ventilation (Table 11).
Despite prior management of temperature and humidity, there
was room for improvement in environmental management, as
revealed in Table 12. The system was found to significantly
contribute to a better work environment, with participants
intending to continue improving their telecommuting spaces.

Fig 10．Environmental data of each participants
temperature but variable humidity and CO2 levels. Participant
B maintained stable temperature and humidity, but CO2
levels fluctuated. Participant C had a consistently stableenvironment.

Table 11．Questionnaire about notification frequency
Type of information 1
Type of information 2
Type of information 3
Type of information 4

A
4
5
3
4

B
4
5
5
5

C
4
4
4
4

5.3.2 Qualitative evaluation
Feedback from the post-experiment questionnaire was used
to evaluate improvements in the working environment.
Participants confirmed using air conditioning, humidifiers,and ventilation, and monitoring environmental parameters.
Despite prior efforts to manage the environment, there were
gaps in effective utilization of data and environmental control.
Table 14 shows participants’ ratings of the system’s impact
on their work environment and their intentions for continued
improvement. Scores on a 5-point Likert scale indicated apositive contribution of the system to the work environment
and a strong intention to maintain these improvements.

Table 12．Questionnaire about system usefulnessA

B

C

Was the type of information 1 useful?

3

5

4

Was the type of information 2 and 3 useful?

5

5

4

Was the type of information 4 useful?

3

5

4

5.3 Improvement of work environment
5.3.1 Quantitative evaluation
We assessed how the system influenced participants’ work
environments using data on temperature, humidity, andcarbon dioxide levels. Participant An experienced stable

27

International Workshop on Informatics ( IWIN2023 )Table 14 ． Questionnaire about improvement oftelecommuting environment
System’s contribution to a good work
environment?
Intent to continue improving the work
environment?

6

A

B

C

5

4

5

5

5

5

6.2.3 Questionnaire Considerations
The study assessed the system’s utility and the shift inworkspace environment awareness. Questionnaire responses
indicated potential improvements in notification timings anddata presentation. Participants showed an increased drive forenvironmental improvement, validating the experiment's
impact. Daily task inquiries were found useful for data
analysis and could lead to more tailored suggestions in the
future.

DISCUSSION

7

6.1 Information notification system

The system, aimed at improving the telecommuting
environment, faced challenges, especially with Slack as thenotification platform due to its chat-based design. Overall, thesystem proved useful but highlighted areas for further
improvement.

We discuss our system, which transmitted four types ofdata and included a chatbot feature for enhancing work
environments. Despite a non-operational period, due to
Heroku account issues, the system was generally efficient.
However, the chatbot feature was underutilized, possibly due
to participants’ work constraints.

Acknowledgments
This work was supported by JSPS KAKENHI Grant
Numbers JP22H03572.

6.2 Discussion derived from questionnaire
results

References[1]

This segment deliberates on the insights obtained fromquestionnaires administered pre, during, and post-experiment.
6.2.1 Information notification system
Usability insights of the information notification system,
sourced from the post-experiment questionnaire, are explored
here. Feedback on the four pieces of presented information
was generally favorable. Notification intervals, the shortest
being hourly, were well-received due to their concise andinstantly comprehendible design.

[2]

[3]

6.2.2 Improvement of work environment
Participants understood their workspace conditions better,leading to conducive work environments. For example,
Participant A used a humidifier post-notification, showing abehavioral change, while Participant B did not significantly
alter behavior despite high CO2 levels. Challenges such as
ventilation impacts and external noises were noted,
highlighting the limitations of relying solely on information
notifications for behavioral change. The study suggests a needfor automated environmental regulation but acknowledges its
complexity.

[4]

[5]

[6]
[7]

Table 13 ． Type of work and the total days duringexperimentType of work

A

B

C

Office work
Meeting
Programming
Write documentation
Thinking
Other

0
11
12
13
5
0

2
14
0
12
5
0

5
3
0
3
0
0

CONCLUSION AND REMARKS

[8]

[9]

28

Kawaguchi, G. et al., “Intellectual Productivity Evaluation in Indoor
Environments: (Part 8) Impact of Improved Thermal Environment
Satisfaction through Quantitative Methods on Intellectual Productivity,”
Journal of Air Conditioning and Sanitary Engineering of Japan, 2004.
Netatmo， ”Smarthomeweatherstation” ，URL：
https://www.netatmo.com/en-gb/weather/weatherstation/indoormodule, (Accessed November15, 2023)．
Nishino, J. et al., “Research and Development of AI & IoT-basedIntellectual Environment Control System to Realize Spaces that
Enhance Intellectual Productivity (3rd Report) Verification Evaluationof the Effect of Work Efficiency Improvement by Air Conditioning
Control Based on Sleepiness,” Proceedings of the Air Conditioning &Sanitary Engineering Society Conference, Vol. 8, pp. 381-384, 2019.
Ministry of Health, Labor and Welfare, “Guidelines for the Proper
Introduction
andImplementationofTelework,”
URL:
https://www.mhlw.go.jp/content/000759469.pdf,(Accessed
November15, 2023)．
Hamanaka, G and et al., “Implementation and Evaluation of a Teaching
Bot for Improving Indoor Comfort,” Research Report on Ubiquitous
Computing Systems, No. 9, pp. 1-6, 2017.
Slack，URL：https://slack.com/intl/ja-jp/, (Accessed August 3, 2023)．
Ministry of Health, Labor and Welfare, “Standards for Building
Environmental
HealthManagement,”
URL:
https://www.mhlw.go.jp/bunya/kenkou/seikatsueisei10/,(Accessed
November15, 2023)．
National
InstituteofHealthandNutrition,
URL：
https://www.nibiohn.go.jp/eiken/programs/2011mets.pdf, (Accessed
November15, 2023)．
FREE
HAND,
“Clo
Value
Laboratory,”
URL：
http://freehandjapan.com/, (Accessed November15, 2023)．

Ke�eech 1:
Dr. Hidetoshi Mishima(Mitsubishi Electric Corporation
Corporate Research and Development
Group Executive Fellow)(Chair: Akihiro Hayashi)International Workshop on Informatics ( IWIN2023 )31

International Workshop on Informatics ( IWIN2023 )32

International Workshop on Informatics ( IWIN2023 )33

International Workshop on Informatics ( IWIN2023 )34

International Workshop on Informatics ( IWIN2023 )35

International Workshop on Informatics ( IWIN2023 )36

International Workshop on Informatics ( IWIN2023 )37

International Workshop on Informatics ( IWIN2023 )38

International Workshop on Informatics ( IWIN2023 )39

International Workshop on Informatics ( IWIN2023 )40

International Workshop on Informatics ( IWIN2023 )41

International Workshop on Informatics ( IWIN2023 )42

International Workshop on Informatics ( IWIN2023 )43

International Workshop on Informatics ( IWIN2023 )44

International Workshop on Informatics ( IWIN2023 )45

International Workshop on Informatics ( IWIN2023 )46

International Workshop on Informatics ( IWIN2023 )47

International Workshop on Informatics ( IWIN2023 )48

International Workshop on Informatics ( IWIN2023 )49

Session 2:Intelli

stems(Chair: Kozo Okano)International Workshop on Informatics ( IWIN2023 )Relieving Trafﬁc Congestion Stably by Controlling Amount of Vehicles
Detouring from Congested Roads
Arisa Kitano† , Takuya Yoshihiro‡†

Graduate School of Systems Engineering, Wakayama University, Japan‡
Faculty of Systems Engineering, Wakayama University, Japan†e-mail: s236086@wakayama-u.ac.jp‡
e-mail: tac@wakayama-u.ac.jp

Abstract - Automobile trafﬁc jam causes serious problems
such as loss of time for drivers on congested roads, trafﬁc accidents, and environmental pollution. Therefore, trafﬁc jam
is a problem that needs to be solved. One method to solving these problems is encouraging vehicles to detour aroundcongested roads. In this method, the system detects the occurrence of trafﬁc jams and suggests the drivers a route to detour
around the congested road segments. This approach reduces
the number of vehicles entering the congested road, thereby
relieving the congestion. However, when congestion is relieved by detours, the system terminates the detour or rapidly
reduces the number of detours, which increases the numberof vehicles entering the road, and causes re-congestion. In
this study, as an extension to the conventional methods ofdetouring vehicles to reduce trafﬁc jams, we propose a new
method that balances the number of vehicles entering and exiting a congested road. This proposed method improves the
efﬁciency of the entire trafﬁc network and prevents repeatedtrafﬁc jams. We implemented the proposed method in a trafﬁc ﬂow simulator and evaluated its performance. The results
showed that the average vehicle travel time reduces with theproposed method, indicating that the proposed method controls road trafﬁc more efﬁciently during trafﬁc jam resolution.

However, this approach causes re-trafﬁc congestion when thenumber of vehicles entering the road exceeds the trafﬁc capacity.Due to mitigate this adverse effects, it is essential toresolve trafﬁc congestion with stable trafﬁc condition transitions, not only when the system detect a trafﬁc jam but also
after the congestion relief.
In this study, we expand the conventional methods of resolving trafﬁc jams by detouring vehicles and propose a new
method that balances the number of vehicles entering a congested road and the number of vehicles exiting the road. This
proposed method improves the efﬁciency of the entire trafﬁc
network from the occurrence of trafﬁc jam to its resolution.
Speciﬁcally, we propose a method of controlling the numberof vehicles that detour around congested roads. The basic
idea is to stabilize the change in trafﬁc volume on congested
roads when relieving congestion by keeping the number of vehicles entering a congested road at a certain level lower than
the number of vehicles exiting the road. This approach provides stable control overall trafﬁc between occurrence of a
trafﬁc jam to its resolution and prevents re-trafﬁc congestion
and temporary trafﬁc disruptions when the number of vehicles
detouring around the congested road is changed. Efﬁcient andstable congestion resolution reduces vehicle travel time due tocongestion.
We implemented the proposed method in a trafﬁc ﬂow simulator and evaluated its performance. The results showed thatthe proposed method was more stable than the conventional
method in the process between the occurrence of a trafﬁc
jam and its clearing. In addition, the results of the proposed
method showed that the average vehicle travel time was lower,
indicating that the proposed method made road trafﬁc more
efﬁcient during trafﬁc jam resolution.

Keywords: automobile, trafﬁc jam, detour route guidance,
ITS(Intelligent Transport Systems), trafﬁc control

1 INTRODUCTION
Automobile trafﬁc jam causes problems such as loss oftime for drivers on congested roads, trafﬁc accidents, and environmental pollution. Therefore, trafﬁc jam is a problem thatneeds to be solved. Trafﬁc jam occurs when the trafﬁc demand on a road exceeds the trafﬁc capacity. The causes of
trafﬁc congestion may be an increment in trafﬁc demand due
to commuting to and from work and school or a reduction in
trafﬁc capacity due to trafﬁc accidents or lane restrictions forroad construction. One approach to solving such problems
is encouraging vehicles to detour around congested roads. In
this method, the system detects the occurrence of trafﬁc jams
and suggests the drivers a route to detour around the detectedroad segments. This approach reduces the number of vehicles
entering the congested road, thereby relieving the congestion.
However, this method has problems. This method temporarily
reduces the number of vehicles entering a congestion detectedroad segment. Then, when the congestion eases, the systemterminates the rerouting or reduces the amount of rerouting.

2 RELATED WORK
Urban Transportation Optimization is one of the major interests in the worlds, and thus a large amount of studies exist
in the literature [1]. Two major means to optimize transportation is the trafﬁc signal control [2] and the vehicle route control [3]. The trafﬁc signal control can control all vehicles inthe entire network. However, this approach can not reduce
trafﬁc on congested roads. In contrast, the route control reduces trafﬁc on congested roads by detouring. Therefore, theroute control is more effective in case of congestion because
immediate effect can be expected.
Among several existing route control approaches, the route53

International Workshop on Informatics ( IWIN2023 )guidance system (RGS) is one of the most promissing one [4].
RGS is basically a reactive system that, once congestion isdetected, the system typically computes routes that bypass thecongested roads, and guide vehicles to avoid the congestion
with those routes. As a considerable amount of trafﬁc avoids
the congested routes, the congestion resolves in a short time.
RGS methods are classiﬁed into three types, i.e., centralized, distributed, and hybrid ones. In the centralized approach,trafﬁc information is collected by the central server, and theserver computes the routes to guide. Several centralized approaches have been proposed [5][12], and several current route
map services such as Google Maps [6] and TomTom [7] are
implemented as a centralized service. Several fully distributed
approach exists, in which vehicles monitors states of roads bythemselves, and make their own route decision [8][9]. However, in the distributed approach, it is hard to collect road-state
information so that reliable route computation is not possible.
The hybrid approach assumes an infrastructure to collecttrafﬁc state information, and the collected information is forwarded to each vehicle, and the vehicle computes their own
optimal routes from the obtained information [10][11]. Although those methods sometimes coorporates to distribute trafﬁc to prevent concentration, since essentially the route computation is distributed, the effect inevitably is suboptimal. Fromthe viewpoint of optimization, the central approach achieves
the best performance. Although the computational load could
be one of the weakpoints, it is still worth exploring the potential of it.
Matsui et al. focus on detouring vehicles far from congested roads compute [12]. While DSP, RkSP, and EBkSP
compute detour routes from near the congested road, Matsui
et al. considered the possibility of reducing vehicle arrival
time by detouring vehicles from roads farther away from congested roads. The method developed by Matsui et al. calculates multiple detour routes from nearby and distant intersections from the congested roads and preferentially guides
vehicles to routes with smaller travel-time increases. In addition, because their method calculates detour routes considering the trafﬁc capacity of the roads on the detour route,it prevents secondary congestion from occurring on the detour route. However, this method has the problem that trafﬁc
on congested roads becomes unstable because it rapidly decreases the detouring vehicles when congestion eases.
The above methods fail to stabilize trafﬁc on congested
roads because they do not properly account for the numberof detouring vehicles. The number of detouring vehicles isclosely related to trafﬁc stability. When the number of detouring vehicles is high, trafﬁc congestion rapidly relieves.
However, when the congestion is relieved and the detour guidance is stopped, trafﬁc is disrupted due to the sudden increasein the number of vehicles on the congested road, resulting in
longer vehicle arrival time. On the other hand, if fewer vehicles take detours, trafﬁc is not disrupted by the change in
trafﬁc volume. However, there is a risk of worsening trafﬁccongestion due to less trafﬁc reduction on the road. This results in longer vehicle arrival time. Therefore, it is essential
to properly determine the number of detouring vehicles when
we resolve trafﬁc jams by vehicle detouring.

These methods terminate all detour guidance when there is
no more trafﬁc congestion. As mentioned earlier, this leads
to an unstable and inefﬁcient trafﬁc state. A smooth change
in the number of detouring vehicles would stabilize the trafﬁc condition, but to the best of our knowledge, no method
has been proposed in the literature that controls the volume
of detouring vehicle ﬂows to stabilize the whole congestion
management process. In this regard, this work is new andnovel.

3 PROPOSED METHOD
3.1 Overview of the Proposed Method
We propose a method for calculating the number of vehicles detouring around a congested road segment so that balances the number of vehicles entering and exiting that roadsegment. When trafﬁc congestion is detected, the proposed
method calculates the number of vehicles detouring aroundthe congested road so that the number of vehicles entering
the congested road is lower than the number of vehicles exiting that road. This approach stabilizes ﬂuctuation in trafﬁc on congested roads and resolves trafﬁc congestion. Even
if the detour guidance relieves the trafﬁc congestion and thecongestion is no more detected, our method continues thedetour guidance. If the guidance is terminated in this situation, the increase in trafﬁc volume on the congested roadcauses congestion. Therefore, the detour guidance terminates
when the cause of the trafﬁc congestion, such as lane closures,is removed and the number of detouring vehicles is reduced
to zero. Therefore, the proposed method efﬁciently resolves
trafﬁc congestion and reduces trafﬁc travel time.

3.2 Assumptions
In this study, we envision the situation where a methodof vehicle detour guidance resolves trafﬁc congestion. The
method detects a trafﬁc jam when the number of vehicles ona road segment exceeds the threshold value, and guides vehicles to detour routes that avoid the congested road segment
to relieve the congestion. In this study, we distinguish between the method of computing detour routes and the methodof calculating the number of detouring vehicles, and focus on
the latter to stabilize and improve the efﬁciency of congestion
resolution. In the computing of detour routes, it is assumed
that the method computes more than a single detour route andthat road segment as the starting point of the detour route. It
is envisioned that roadside units are installed at various locations on the target road network and the data obtained fromthese units provide the trafﬁc volume for the entire road network. In addition, we focus on the centralized control system in which a central server computes the detour routes andthe number of detouring vehicles for the entire target road
network, rather than the distributed control system in which
each vehicle computes its detour route. In a centralized control system, roadside units transmit the trafﬁc volume to thecentral server. If the central server detects a trafﬁc jam basedon the received trafﬁc volume, it computes the detour planthat consists of the number of detouring vehicles and their de-54

International Workshop on Informatics ( IWIN2023 )tour routes and transmits the plan to roadside units. Then, theroadside units guide the target vehicles based on the detour
plan.

the adjusted status, we continue to guide vehicles with the appropriate detour volume Drt until it determines that the congestion is resolved and there is no need for detour guidance.

3.3 Notation

3.5 Calculating the Amount of Detour

We deﬁne the inﬂow Irt and the outﬂow Ort as the number of vehicles entering and exiting road segment r at timet, respectively. We also deﬁne the detour volume Drt as thenumber of vehicles that are proposed to change from a route
through road segment r to a route avoiding road segment r
at time t. There is more than one road segment that is thestarting point of the detour route. Then, when U is the set of
those road segments, and dtu for u ∈ U denotes the numberof vehicles detouring to the route that starts∑at road segmentu. Therefore, it can be expressed as Drt = u∈U dtu . In addition, it is assumed that we can predict the time vehicles ittakes to travel each road segment. When a vehicle change tot
the route that detours around the road segment r, travel(u,r)
denotes a predicted travel time of the route from the road segment u, the starting point of the detouring route, to road segment r before the detour. In this case, the effect of the detourt.
volume Drt appears at time t + travel(u,r)
When detouring vehicles resolve trafﬁc congestion by detouring vehicles, even after the congestion is no longer detected, a certain amount of detouring must be continued toprevent the recurrence of the congestion and to maintain stable road trafﬁc. Therefore, in this study, the resolution of
trafﬁc congestion means that the causes of trafﬁc congestion,such as trafﬁc accidents or trafﬁc restrictions, etc., have been
removed, the number of detours has been reduced to zero,and trafﬁc has stabilized, not that the congestion is no longer
detected.

The detour volume Drt in the congested and adjusted states
is computed as shown in Eq. (1).
Drt = Irt − Ort + Rrt(1)

Note that we deﬁne the reduction Rrt as the desired difference
between the inﬂow and the outﬂow at road segment r, and deﬁne the detour reservation Ert as the trafﬁc volume scheduled
to decrease by recent detour guidance. The calculation of the
reduction Rrt is explained in Sec. 3.6.
Irt − Ort + Rrt in Eq. (1) controls the relationship between
the inﬂow and the outﬂow of the road segment r after detour
guidance. When the reduction Rrt is positive and the detour
volume Drt is Irt − Ort + Rrt , the inﬂow of r is reduced lower
than the outﬂow. In addition, the higher the reduction Rrt is,
the larger the difference between the inﬂow and the outﬂow of
the road segment r after the detour guidance is provided. As
this difference between the inﬂow and the outﬂow increases,
there is a tradeoff: while congestion is rapidly eliminated bylowering the inﬂow on congested roads, the overall trafﬁc network becomes unstable due to the large disruption in trafﬁcvolume.
Note that simply setting the detour volume Drt to Irt −Ort +t
Rr is not enough to calculate the appropriate detour volume
Drt , because the effect of detouring takes time to be reﬂectedin the congested road r. For example, if the detour volume
Drt = Irt − Ort + Rrt is made at time t, the time it takes forthe detour effect to be reﬂected in r is equal to the time ittakes to reach road segment r from detour road segment u,
which is a reasonably long time. In other words, there is adelay before the detour effect is reﬂected on the congested
road. This results in a feedback system with a large delay,
which generally causes large oscillations in trafﬁc volume oncongested roads and results in unstable trafﬁc.
Based on the above, the detour volume Drt is calculated in
the congested and adjusted states. To balance the inﬂow Irt
and the outﬂow Ort of a congested road segment r, the detour
volume Drt is calculated by Irt − Ort + Rrt . Trafﬁc control isstabilized by calculating the detour volume Drt by considering
the feedback time delay.

3.4 State Transition
The central server assigns each road segment a state thattransitions according to the road conditions. Figure 1 shows
the state transition diagram. There are three states: normal,
congested, and adjusted. The road segment given the normal state has no congestion detected, and no guidance is provided to detour around that road. On the road segment given
the congested state, trafﬁc congestion is detected. The detour volume Drt is calculated so that the inﬂow Irt is a certain
amount lower than the outﬂow Ort , and vehicles planning aroute through the congested road segment are guided to a detour route based on the detour amount Drt . This approach reduces trafﬁc volume below a certain level without disruption.
The adjusted state is assigned to a road segment when trafﬁc congestion is no longer detected during detouring vehicles
around that road segment. For the road segment given adjustment state, the detour volume Drt is calculated to maintainor gradually reduce the current severity of congestion without
worsening it until the congestion is resolved.
If the number of vehicles on a congested road segment
drops below the congestion detection threshold due to vehicle
detours, terminating detour guidance may cause an increasein trafﬁc volume and congestion again. Thus, even though
there is no congestion detected on a road segment r assigned

3.6 Reduction Rrt
On a road segment r in congested state, the inﬂow Irt must
be lower than the outﬂow Ort to relieve the congestion. The
reduction Rrt affects the difference between the inﬂow and theoutﬂow. Then, the reduction Rrt is equal to α and controls thedifference between the inﬂow and the outﬂow to always keepa constant α.
The state of a road segment changes from congested state
to adjusted state when the congestion eases and the numberof vehicles on that road segment becomes less than the congestion detection threshold. The adjusted state is intended toresolve trafﬁc congestion stably by maintaining or gradually55

International Workshop on Informatics ( IWIN2023 )Figure 1: state change diagram
nario, lane restrictions cause trafﬁc congestion and the proposed method resolves that congestion. By calculating the
average travel time for all vehicles, we evaluate that the proposed method can efﬁciently control not only congested roads
but also the entire road network, which is the objective of thisstudy. We implemented Matsui et al.’s method [12] and the
proposed method as an extension to it, and compared the performance of the proposed method with that of conventional
methods. In the simulation of the proposed method, we use
the proposed method for calculating the detour volume, anduse Matsui et al.’s method [12] for the congestion detection
and the detour route computation. The trafﬁc ﬂow simulator
is SUMO (Simulation for Urban MObility) [13], and TraCI(Trafﬁc Control Interface) [14] acquires trafﬁc informationand suggests routes for vehicles.
The roadside units acquire trafﬁc volumes on each road atregular intervals, and the central server uses the average of the
trafﬁc volumes over the past period of about two trafﬁc-lightcycles to compute detour routes and detect trafﬁc congestion because this study assumes the road network with trafﬁclights. Note that if the central server detects congestion on aroad segment with a red light, the congestion may be falsely
detected due to an accidental increase in trafﬁc volume. In addition, the detour volume calculated by the proposed method
may not be not stable because there is a large difference in
trafﬁc volume between roads with green and red trafﬁc signals. Therefore, to avoid such kind of ﬂuctuation on trafﬁcmeasurement, this simulation calculated the average value fortwo signal cycles.

Figure 2: the reduction Rrt
decreasing the trafﬁc volume so that the trafﬁc volume does
not increase and the congestion is not detected again. For thispurpose, the reduction Rrt is adjusted according to the trafﬁcvolume on the road segment r. If there is a high probability
that congestion is detected again on road segment r, set Rrt toa positive value to reduce the trafﬁc volume on road segmentr by making the inﬂow on road segment r smaller than theoutﬂow. Conversely, if congestion is easing, set Rrt to a negative value to increase trafﬁc on r by making the inﬂow higher
than the outﬂow.
In the adjustment state, the proposed method adjusts the reduction Rrt according to the number of vehicles on the roadsegment r used for congestion detection, as shown in Fig. 2.
Note that the trafﬁc volume when Rrt = 0 is called the stable value γ, which is set as a parameter value smaller thanthe congestion detection threshold. Also, α is the same parameter used to calculate the reduction Rrt in congested. By
controlling the reduction Rrt as shown in Fig. 2, the detour
volume Drt is adjusted so that the number of vehicles on roadsegment r is close to the stable value. Then, when the outﬂow
on road segment r increases due to the removal of congestion
causes, the detour volume Drt decreases to maintain the trafﬁcvolume. Finally, the detour volume Drt becomes zero and thedetour guidance is terminated.

4.2 Scenario
In this simulation, we used the grid road map shown in
Fig. 3. Each road segment on the road map is a right-of-way
with three lanes in each direction, consisting of left-turn andstraight-ahead lane, straight-ahead only lane, and right-turnand straight-ahead lane. An enlarged map of the area near theintersections is shown in Fig. 3. Trafﬁc lights is installed ateach intersection, and the signal cycle is set to 140 seconds.
Vehicles travel from one end of the road map to the others.
For example, the destination of vehicles departing from thebottom of the map is equally assigned to a total of 15 roadsegments toward the left, top, and right ends of each road network. Therefore, the number of vehicles departing from each

4 EVALUATION
4.1 Method
To conﬁrm that the proposed method reduces the averagevehicle travel time, we performed simulations. In our sce-

56

International Workshop on Informatics ( IWIN2023 )Table 1: Simulation setting
Trafﬁc simulator
Simulation start time
Simulation end time
Lane closure time
Vehicle used for evaluation
Trafﬁc volume
Signal period
Congestion detection cycle
Trafﬁc acquisition cycle
Congestion detection threshold
Expansion rate of congestion
Estimated probability offollowing route guidance β
Actual probability offollowing route guidance
αstable value γ

Figure 3: Road maps used in the evaluation

SUMO [13]Time 0s
Time 28800s
Time 3600s to 10800s
Vehicle departingtime 1800s to 14400s
4500/h, 5000/h
140s
20s
20s
0.5
1.3
0.8
0.7
3, 5, 7, 9, 11, 13
0.375

spectively. In this simulation, we detect trafﬁc congestion at
regular intervals. When congestion is detected, a detour guidance plan until the next congestion detection time is calculated. The congestion detection and trafﬁc volume acquisition are performed every 20 seconds, and the period of the
trafﬁc light is 140 seconds. We set the threshold for trafﬁccongestion detection at 0.5 based on the trafﬁc ﬂow model
called Greenshields Model [16]. This means that congestion
is detected when the number of vehicles on a road is morethan half of the maximum number of vehicles on that road.
The expansion rate of congestion, a parameter of the comparison method, is set to 1.3 based on Matsui et al.’s paper [12].
The actual probability of following the route guidance is 0.7,
i.e., 30 % of vehicles does not follow the route guidance ofthe system and do not change to follow their own route, andthe estimated probability of following the route guidance, described in Sec. 3.5, is 0.8. For both the proposed method andthe comparison method, the actual number of guided vehicles is the calculated detour volume divided by this estimatedvalue. As the settings in the proposed method, the stable value
γ is set at 0.375, and we simulated the cases where α is 3, 5,
7, 9, 11, and 13, respectively.

Figure 4: Close-up view of the intersection
road is the same for all roads, and the number of vehicles
arriving at each road is the same for all roads. SUMO automatically computes the shortest route for the vehicle using
Dijkstra’s algorithm [15] based on the origin and destination
roads. The weight of each road is its travel time, which is thequotient of the road length and the maximum speed.
In this simulation, we closed two lanes of the busiest roadsegment to generate trafﬁc congestion. The road segmentwith the lane closures is shown at point X in Fig. 3. We
blocked the two right-hand lanes of the road segment at point
X: the straight-ahead lane and the right-turn and straight-aheadlane. Thus, the bottleneck, which is the remaining left laneof the three lanes, causes congestion downstream of point X.
Also, the lane restriction prevents vehicles from turning rightfrom point X. During the lane closure, the vehicle planning toturn right from point X re-computes its route when it arrives
at the road segment ahead of point X. The start time of thelane closure is set to 3600 seconds after the start of the simulation, and the end time of the lane closure is set to 10800
seconds after the start of the simulation.
The simulation settings are shown in Table 1. The simulation starts at time 0 seconds and ends at time 28800 seconds.
Then, we evaluated the vehicles that departed between 1800
and 14400 seconds. We simulated trafﬁc volumes of 4,500
and 5,000 vehicles per hour for the entire road network, re-

4.3 Results
We calculated the average travel time for all vehicles departing between 1800 and 14400 seconds from the beginningof the simulation. The results for a trafﬁc volume of 4500 vehicles per hour are shown in Fig. 5, and that of 5000 vehicles
per hour in Fig. 6. The parameter α of the proposed method
is on the horizontal axis. Since the conventional method does
not use the parameter α, its average travel time is the samevalue regardless of the horizontal axis.
Figures 5 and 6 show that the average vehicle travel time57

International Workshop on Informatics ( IWIN2023 )gestion was detected for the longest time. The transition ofroad Y for the proposed and comparative methods are shownin Figs. 7 and 8, respectively. These ﬁgures show the roadcondition, the average number of vehicles, and the average
speed of vehicles on road Y. The number of vehicles and thevehicle speeds are averaged over two signal cycles (140 seconds) to eliminate oscillations in values due to trafﬁc signals.
The conventional method does not deﬁne the road state. Thus,
we regard that the congestion state as the condition in which
trafﬁc congestion is detected detour guidance is appled, andthe normal state as the condition in which no trafﬁc congestion is detected, and no guidance appled as well. Then, thedeﬁnition of the congestion state is the same for the proposedand conventional methods since they use the same congestion
detection method. In this simulation, trafﬁc congestion is detected based on the number of vehicles. Therefore, when thecongested state is assigned to a road, the number of vehicles
on that road is exceeding the certain level.

Figure 5: Average travel time for all vehicles(4500 vehicles / hour)

Figures 7 and 8 show that both the proposed and conventional methods detected trafﬁc congestion at around 3600 seconds, which is the start time of lane restrictions, and resolved
it at around 10800 seconds, which is the end time of the lane
restrictions. Due to the closure of the two right lanes of Road
X, a queue of vehicles extends to make the upstram Road
Y congested as well. On the other hand, results during theperiod of lane restrictions showed the differences in performance between the proposed and conventional methods. The
results of the conventional methods in Fig. 8 shows that theroad state and the average number of vehicles on the roadkeeps considerably high value. During lane restrictions, detour guidance must actually be continued because trafﬁc capacity on that road is reduced due to the downstream lane
restriction. However, the conventional method transits to thenormal state and quit detouring when the number of vehicle
reduces, which again increases the number of vehicles, leading to the repeated ﬂuctuation of trafﬁc. Compared to the conventional method, Fig 7 shows that in the proposed method,
the road state is stabilized and the number of vehicles on theroad signiﬁcantly reduced by controlling the detour volume tomaintain a balance between the inﬂow and outﬂow volumes
on the congested roads. It also shows that the speed of thevehicle in the proposed method is higher than that of the conventional method. The 45 % increase in average vehicle speedwith the proposed method reduces the travel time by about 30
seconds per vehicle.

Figure 6: Average travel time for all vehicles(5000 vehicles / hour)of the proposed method is lower than that of the conventional method. The difference in average travel time between
the proposed and the compared methods is about 30 seconds
when the trafﬁc volume is 4500 vehicles per hour and about
70 seconds when the trafﬁc volume is 5000 vehicles per hour.
Clearly, it can be said that the proposed method reduces theaverage vehicle travel time. The proposed method performs
stable trafﬁc control by maintaining a balance between thenumber of vehicles entering and exiting congested roads. The
results showed that this approach reduces the average travel
time of vehicles.
In addition, Fig. 5 shows that the performance is better
when alpha is high, while Fig. 6 shows that performance isbetter when α is low. We suppose that α is related to the ﬂuctuation in trafﬁc volume on congested roads. The larger α is,the larger the difference between the inﬂow and the outﬂow
on a congested road, and the more rapidly congestion is relieved. On the other hand, the greater the difference between
the inﬂow and the outﬂow on a congested road, the more turbulent the trafﬁc on that road is. Therefore, when the trafﬁcvolume of the entire road network is low, the large α rapidly
relieves the trafﬁc congestion and the average travel time becomes low. Conversely, when the trafﬁc volume of the entire
road network is high, the large α disrupts the trafﬁc, and theaverage travel time becomes high.
To conﬁrm that the proposed method improves the efﬁciency of trafﬁc control on congested roads, we consider indetail the road Y in Fig. 3 with a trafﬁc volume of 5000 vehicles per hour and α = 13. Road Y is the road where con-

The above results shows that stabilizing trafﬁc on congested
roads reduces the average travel time of vehicles in the vehicle route guidance method for resolving trafﬁc congestion.
Stabilizing trafﬁc on congested roads prevents trafﬁc disruptions on roads around that congested road. Therefore, the proposed method stabilizes the control of the entire road networkand reduces the average travel time for all vehicles. In addition, since the proposed method stabilizes trafﬁc better thanthe compared method, it suggests that maintaining a balance
between the inﬂow and outﬂow on congested roads is an effective way to stabilize trafﬁc.

58

International Workshop on Informatics ( IWIN2023 )(a) Transition in road state and average number of vehicles(b) Transition in road state and average speed of vehicles

Figure 7: Simulation of proposed method (trafﬁc volume: 4500 vehicles per hour)(a) Transition in road state and average number of vehicles(b) Transition in road state and average speed of vehicles

Figure 8: Simulation of conventional method (trafﬁc volume: 4500 vehicles per hour)59International Workshop on Informatics ( IWIN2023 )5 CONCLUSION

Travel Time,” IEEE Transactions on Vehicular Technology, Vol.62, No.8, 2013.
[6] http:// www.google.com/mobile/[7] http://www.tomtom.com/en gb/products/mobilenavigation/[8] A.M. de Souza and L.A. Villas, “A Fully-distributed
Trafﬁc Management System to Improve the Overall Trafﬁc Efﬁciency,” In Proceedings of the 19th
ACM International Conference on Modeling, Analysis, and Smulatino of Wireless and Mobile Systems(MSWiM2016), pp.19-26.
[9] Z. Khan, P. Fan, S. Fang, and F. Abbas, “An Unsupervised Cluster-based VANET Oriented Evolving
Graph (CVoEG) Model and Associated Reliable Routing Scheme,” in IEEE Transactions on Intelligent Transportation Systems, Vol.20, No.10, pp.3844-3859, 2019.
[10] J. Pan, I.S. Popa and C. Borcea, “DIVERT: A Distributed Vehicular Trafﬁc Re-routing System for Congestion Avoidance,” in IEEE Transactions on Mobile
Computing, Vol.16, No.1, pp.58-72, 2017.
[11] B. Liu, J. Long, M. Deng, X. Yang, Y. Shi, “An Adaptive
Route Planning Method of Connected Vehicles for Improving the Transport Efﬁciency,” ISPRS International
Journal of Geo-Information, 11(1):39, 2022.
[12] Y. Matsui, T. Yoshihiro,: A Route Guidance Method
for Vehicles to Improve Driver s Experienced Delayagainst Trafﬁc Congestion, 18th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, pp 375–389 (2021).
[13] P.A. Lopez, M. Behrisch, L. Bieker, J. Erdmann, Y.
Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, E. Wießner.: Microscopic Trafﬁc Simulation using
SUMO, 2019 IEEE Intelligent Transportation Systems
Conference (ITSC), pp. 2575-2582 (2018).
[14] A. Wegener, M. Piorkowski, M. Raya, H. Hellbruck, S.
Fischer, and J.P. Hubaux.: TraCI: An Interface for Coupling Road Trafﬁc and Network Simulators, Processingof the 11th communications and networking simulation
symposium, pp.155-163 (2008).
[15] E.W. Dijkstra.; A note on two problems in connexion
with graphs, Numerische Mathematik, Vol. 1, pp. 269271 (1959).
[16] B.D. Greenshields.: A Study of Trafﬁc Capacity, Proceedings of the highway research board, Vol. 14, pp.
448-477 (1935).

In this study as an extention to the route guidance method
to resolve congestion, we proposed a method for controllingthe number of detouring vehicles to stabilize the ﬂuctuationof trafﬁc volume on congested roads. By maintaining the balance between the number of vehicles entering and exiting thecongested road, the trafﬁc volume on the congested road isstabilized. This approach stabilizes the trafﬁc control fromthe occurrence of trafﬁc congestion to its resolution, thereby
reducing the average travel time for all vehicles.
We evaluated the proposed method using a trafﬁc ﬂow simulator, SUMO [13]. In our scenario, Lane restrictions on agrid road map leads to trafﬁc congestion, and a route guidance method resolves the trafﬁc congestion. As a route guidance method, we used the method of Matsui et al. [12], andcompared the cases where the proposed method is applied andnot applied. The results showed that the proposed method
reduces the average travel time for all vehicles compared tothe conventional method. The results showed that the proposed method As a future issue, it is necessary to improvethe calculation method of the reduction Rrt in the proposed
method. The proposed method stably resolves trafﬁc congestion by maintaining a balance between the number of vehicles entering and exiting congested roads. However, in the
proposed method for calculating the reduction Rrt , the appropriate value of α depends on the trafﬁc volume of the entire
road network. In addition, the simulation scenarios in thispaper are rather simple scenarios for evaluationg realistic ofroad conditions. Therefore, we will evaluate the proposed
method by preparing scenarios similar to the road network to
be realized in the future.

ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number 23H03385.

REFERENCES[1] C. Lin, K.L. Choy, G.T.S. Ho, S.H. Chung, H.Y.Lam,“Survey of Green Vehicle Routing Problem: Past andFuture Trends,” Expert Systems with Applications,Vol.41, pp.1118-1138, 2014.
[2] I. Tomar, S. Indu, and N. Pandey. “Trafﬁc Signal Control Methods: Current Status, Challenges, and Emerging Trends,” Proceedings of Data Analytics and Management: ICDAM2021, Vol.1, pp.151-163, 2022.
[3] J. Pan,I. Sandu Popa,K. Zeitouni,and C. Borcea.: Proactive vehicular trafﬁc re-routing for lower travel time,IEEE Transactions on Vehicular Technology, Vol.62,
No.8 (2013).
[4] J.M. Sparmann, “Beneﬁts of Dynamic Route Guidance
Systems as Part of a future Oriented City Trafﬁc Management System,” Vehicle Navigation and Information
Systems Conference 1991, pp.839–847, 1991.
[5] J. Pan, I. Sandu Popa, K. Zeitouni, and C. Borcea,“Proactive Vehicular Trafﬁc Re-routing for Lower

60

International Workshop on Informatics ( IWIN2023 )A Transfer Scheduling Method for ride-sharing Services
to Alleviate Traffic Congestion
Shion Suzuki† , Takuya Yoshihiro‡†

Graduate School of Systems Engineering, Wakayama University, Japan‡
Faculty of Systems Engineering, Wakayama University, Japan{s246135, tac}@wakayama-u.ac.jp

Abstract - Traffic congestion causes various problems suchas lost traffic time and increased exhaust emissions. One solution that has gained attention for alleviating traffic congestion is ride-sharing services. By allowing users to share rides
with other users, the cost of transportation per person can be
reduced. Replacing traditional private vehicle transport withride-sharing services can decrease the number of vehicles on
the roads, contributing to the resolution of traffic congestion.
However, as ride-sharing services become more popular, newtransportation demands may arise, leading to changes in traffic patterns and the emergence of congestion due to different factors than those observed at present. One such factor is
the increased traffic around destination areas due to the concentration of ride-share vehicles. The aim of this study is
to mitigate traffic congestion caused by this increased traffic.
The proposed method adds a process to existing ride-sharingscheduling methods, generating transfers in response to congestion. In the case of congestion on a particular road, rideshare vehicles sharing congested routes within the plannedroute are instructed to transfer during specific time intervals.
This transfer allows users already using the ride-sharing service to board vehicles carrying other users, thereby promoting
ride-sharing. By facilitating multiple transfers, the number ofvehicles heading to the destination is reduced, mitigating congestion caused by vehicle concentration.

ownership.
In the near future, it is predicted that the widespread ofride-sharing will result in a majority of people traveling totheir destinations using ride-sharing services without owninga private vehicle. Furthermore, with the widespread of electric vehicles and autonomous driving technology, it becomes
possible to operate ride-sharing services at lower costs due
to reductions in labor and fuel expenses. Ride-sharing services will be used not only for travel transportation but also
for daily commuting. Since the cost of owning and maintaining a private vehicle will be eliminated, people’s expenditures
for transportation will be significantly reduced.
Ride-sharing, which is expected to alleviate traffic congestion by reducing the number of vehicles on the road, is anticipated to bring about changes in the current state of transportation. This leads to congestion caused by factors different
from the current ones.One of the causes is concentration ofride-sharing vehicles. Demand for transportation to a specificlocation in a short period of time is likely to be concentrated at
popular sightseeing spots on holidays, at event venues that attract many visitors, and at stations during daily commuting toschool and work, etc. When ride-sharing services are extensively used, there is a concern that serious traffic congestion
may occur in such a busy destination, leading to increased
crowding on the roads. This study addresses this issue by encouraging sharing rides among users when traffic congestion
occurs. By having more people ride in a single vehicle, thetraffic volume around the destination is reduced, contributing
to the alleviation of congestion. To the best of our knowledge,
method has been proposed that improves traffic congestion inride-sharing services by transferring ride-sharing vehicles.

Keywords: Intelligent Transportation System (ITS), Ridesharing, Transfer, Traffic Congestion, Mobility as a Service(MaaS)

1 INTRODUCTION
Traffic congestion is a significant issue in recent years. Traffic congestion is a pressing issue that leads to problems suchas fuel loss and the generation of emissions. The effectiveness of ride-sharing services in alleviating traffic congestion
has garnered attention. Ride-sharing are transportation services that, like taxis, provide users with transportation fromtheir pickup location to their destination. The main difference
from traditional taxis in ride-sharing services is the option toshare the ride with other users. Compared to traveling by private vehicle, the use of a ride-sharing service can be expected
to reduce traffic volume by allowing multiple people to travelin a single vehicle. In addition to reducing traffic congestion
by decreasing overall traffic volume, ride-sharing services offer various other benefits. For users, benefits include reduced
fuel costs through ride-sharing, as well as savings in taxes, insurance, and maintenance expenses associated with individual

2 RELATED WORK
2.1 General ride-sharing
This section describes related work on general ride-sharing
services. Ma et al. [1] proposed a method for minimizing
the travel distance of vehicles. To increase the profit of theride-sharing service operator, they compute all combinationsof the picking up and dropping off sequences and assign theschedule that minimizes the travel distance to the request. It
is confirmed that the proposed method reduces the total distance traveled by the cab by 11% and the price per user by 7%
compared to the case without ride-sharing. Okoso et al [2]proposed a route optimization method that prioritizes routes
where requests are expected to occur when selecting routes,
while allowing users to travel with fewer detours. In an envi-

61

International Workshop on Informatics ( IWIN2023 )ronment where the number of vehicles is insufficient for thedemand, this method was found to increase the number of acceptable reservations, suppressing the vehicle travel distanceand user waiting time, compared to the conventional methodthat does not consider other requests when selecting routes.

congestion is continuing, the system needs to instruct a large
number of vehicles to transfer.
This study assumes the operation of a ride-sharing service
through a centralized control system. The central server controls the routing of all vehicles. When a user arrives at thepicking-up point, he/she sends a vehicle request to the central server, including the picking-up location and destination.
When the central server receives a request, it selects an assigned vehicle and updates the stop-point so that the vehicletransport the user to the destination. If multiple users are traveling in a single vehicle, the system instructs the vehicle tofollow a route that minimizes the total delay time for eachuser.
Specifically, the proposed method extends the SBI method
described in Sec. 2.2. This system typically uses the SBI
method to determine the vehicle and route for transporting
the user to their destination. The proposed method monitors
congestion on each road, and upon detecting congestion, thesystem transitions to the Transfer Promotion (TP) state. In the
TP state, vehicles traveling on congested roads receive transfer instructions from the system, promoting ride-sharing. The
proposed method constantly monitors traffic congestion oneach road. This control reduces the number of ride-sharing
vehicles traveling on the congested roads, thereby alleviating traffic congestion. When the congestion is cleared by thiscontrol, the system exits the TP state and returns to the normal state. In the normal state, the system works acording to
the SBI method again to compute routes and does not provide
any transfer instrucution.

2.2 Successive Best Insertion (SBI) Method
Noda et al. [3] proposed a vehicle dispatching algorithm
for responding to user’s transportation demand, called the Successive Best Insertion method. The SBI method is a ridesharing scheduling algorithm based on a greedy approach.
For each request, the SBI method calculates the optimal routeand schedules the picking up and dropping off from the current location to the destination. The following shows the operational steps when a user requests a vehicle until a vehicle
is dispatched.
1. When the user gets to the boarding location, he/she submits a dispatch request to the system. Each vehicle
maintains a stop-point list as a list of the picking-up
and dropping-off points for the allocated requests.
2. The system calculates a new stop-point list for all vehicles.These include the new picking-up and droppingoff points specified by the request. The stop-point list
for each vehicle corresponds to the arrival time at theuser’s destination that minimizes the cost function basedon user delay time.
3. The system selects the vehicle with the best schedule,
i.e. the stop-point list that has the lowest cost among
all the vehicle, and assign the request to that vehicle.
Namely we adopt the stop-point list of that vehicle tosatisfy the request.

3.2 System Model
Let G = (I, A) be the road network, where I is a set ofintersections and A is a set of road segments. There are stoppoints on the road network G, and users can board and alighttheir vehicles at the access points. V is defined as a set ofargride-sharing vehicles on road network G. Let Vlen
be Theaverage vehicle length. Let µ be the user capacity of a vehicle(excluding the driver).
A user of the ride-sharing service sends a request q =(oq , dq , tq ) to the system. oq is the ride location, dq is thedestination, and tq is the time when the request is sent. Upon
receiving a request, the system assigns a vehicle to the request and updates the vehicle’s planned route. The vehicle
picks up the user at the picking-up position oq and delivers
the user to the destination dq according to the planned route.
The planned route is determined using the SBI method.
Each vehicle v ∈ V has a planned route Rv = (r1 , r2 ,. . . , rn )and moves along Rv . Let rk = (ak1 , ak2 , . . . , akm )(1 ≤ k ≤ n)
be the route from stop-point pko to pkd , where pko is located on
ak1 and pkd on akm , respectively. For rk , pko is called the starting
point and pkd the ending point of rk .
The list of stop-points on a planned route Rv is denotedby Pv . Specifically, let Pv = (p1 , p2 , . . . , pn+1 ) be the stoppoint list on the planned route Rv = (r1 , r2 , . . . , rn ), wherestarting point of rk (1 ≤ k ≤ n) is pk and the ending pointis pk+1 . The route from the start point p1 to the end point px
included in the planned route Rv is denoted by Rv,px . Thatis, Rv,px refers to the route until reaching the stop-point px

The cost function takes into account the time required to fulfill the request and the delay time incurred by other users.
Minimizing the cost function allows for faster transportationof users to their destinations.
Due to the nature of the algorithm, it is not possible to consider factors such as the user’s boarding location or schedules
involving transfers. If there is a sufficient number of vehicles,
simulations have confirmed that this method allows users toreach their destinations faster than traditional fixed bus services.

3 PROPOSED METHOD
3.1 Overview
When travel demand concentrates on a single location, ridesharing vehicles are also concentrated on around that destination, causing congestion. The proposed method aims to alleviate congestion by controlling the movement of ride-sharevehicles.controlling the movement of ride-share vehicles is
implemented when congestion occurs on a specific road. Transfer promotes to ride a single vehicle with multiple users, andthe traffic volume of ride-sharing around the destination is
reduced. One transfer reduces by one the number of vehicles going to the destination, which means that, while traffic62International Workshop on Informatics ( IWIN2023 )(n+2)(n+3)

Algorithm 1 Incremental Optimal Insertion Method
1: for v in V do
2:
Cqv , Rv′ ← minInsertCost(Pv , q)
3: end for
4: v ← argminv∈V Cqv
5: Rv ← Rv′

andrithm computes planned routes Rv1 , Rv2 , . . . , Rv 2
its cost Cqv for each Pv1 = (oq , dq , p1 , p2 , . . . , pn+1 ), Pv2 =(oq , p1 , dq , p2 , . . . , pn+1 ), . . . , Pvn+3 = (p1 , oq , dq , p2 , . . . ,(n+2)(n+3)

2=
pn+1 ), Pvn+4 = (p1 , oq , p2 , dq , . . . , pn+1 ), . . . , Pv
x(p1 , p2 , . . . , pn+1 , oq , dq ), finds the minuimum-cost Rv at tine
tq , and return Rvx and its cost Cqv After that, in line 4, we selectthe vehicle v with the minimum cost. For the selected vehiclev, in line 5, we assign request q and update the planned routeof v to the new route Rv .
Function 1 shows The procedure of the function minInsert
Cost(·) to find the optimal planned route for vehicle v to process query q. Lines 1-5 creates all combinations to insert
picking-up point oq and destination dq of query q. Line 2
creates a planned route with oq inserted immediately after Pviand dq immediately after Pvj . Here, as we mentioned before,P˙v = (p0 , p1 , . . . , pn+1 ) is the stop-list list of v, with the last
stopped point p0 added as the first element. The line 3 finds
the planned route Rvij corresponding to Pvij . Line 4 computes
the cost of Rvij according to formula (1). In line 6, let x be theone with the smallest cost among all combinations of i and j.
Finally, line 7 returns the cost Cqvx and the planned route Rvx .
Up to this point, it has been the same ride-sharing model
as related work. Based on the SBI method, it is possible totransport multiple individuals using a single vehicle. In the
proposed method, transfer processes are added to reduce thenumber of vehicles traveling on the road. When a traffic congestion is detected on any road segment a ∈ A on the road
network G, the model transitions to the Transfer Promotion(TP) state. Once congestion is relieved, and it is confirmedthat there is no congestion on all roads, the system returns tonormal state. In the TP state, the system alleviates congestion by triggering transfers for vehicles scheduled to travel oncongested roads, thereby reducing the number of ride-sharing
vehicles on congested roads. The transfer instruction method
is described in the next section.

Function 1 minInsertCost(Pv , q = (oq , dq ))1: for all (piv , pjv ) ∈ P˙v × P˙v , i ≤ j do
2:
Pvij ← insertPositon(Pv , oq , dq , piv , pjv )3:
Rvij ← calcualteShortestRoute(Pvij )4:
Cqv ij ← calculateAssignCost(Rvij )5: end forij
6: x ← argminij Cqv
vx
x
7: return Cq , Rv
in the planned route Rv . For simplicity, we sometimes refer
the planned route Rv,dq to the destination dq of a request q as
Rv,q . Rv is sometimes referred as Rv = (a1 , a2 , . . . , an ) asa list of road segments along the route.
Also, let Rv,a be the route to reach road segment a. For
each road segment a ∈ A, let δat be the estimated passing timeat time t. This is the time estimated at time t to take fromentering until exiting
we∑define the estimated∑ a. Similarly,t= r∈R δrt for r and R.
transit time δrt = a∈r δat and δR
If the current time is t, the estimated transit time that vehicle
v travels along the planned route Rv,p and∑arrives at the stop
location p can be denoted as tvp = t + a∈Rv,p δat . For astop-point list Pv = (p1 , p2 , . . . , pn+1 ) of vehicle v, let P˙v =(p0 , p1 , . . . , pn+1 ) denote the stop-point list that includes theprevious stop-point p0 . Here, p0 is the stop-point that thevehicle has just passed, and vehicle v is on the road between
P0 and P1 at that time.
The SBI method assigns a new request q to the vehicle thatminimizes the cost increased by accepting q. The cost is thesum of the delay time when q are allocated to the vehicle.
Specifically, for a vehicle v ∈ V , let Qv be the set of accepted
requests, and the cost is calculated as follows.
t
Cqv = δR+v,q∑q ′ ∈Qv

t
t
(δR
− δR
),
′
v,q ′
v,q ′

3.3 Determination of Transfer Vehicle andTransfer Point
When traffic congestion occurs on a road segment ac , thesystem transitions to a TP state. Let Rv,ac = (a1 , a2 , . . . ,tan ) be the route of vehicle v to the congested road and δR,ac
be the time taken to reach the congested road ac . In the TP
state, At regular intervals, the system instructs the vehicles toinitiate transfers. The transfer schedule is calculated as theplanned route for each vehicle, including transfers. In otherwords, the system modifies the SBI-based planned route ofeach vehicle described above to incorporate a transfer schedule for some vehicles. The transfer schedule calculation first
determines the two vehicles that will transfer, and then calculates the transfer point and the routes so that their users cantransfer between them. The first step is to find transfer vehicle
pairs that can transfer in a low cost.
The transfer vehicle pairs are calculated as follows. Let
ac be the road where congestion is detected when the systemtransition to the TP state. The TP state persists until congestion at ac is resolved. At regular intervals, the system getsa set of vehicles scheduled to pass through ac . The system(1)

where Rv′ denotes the planned route of v before query q is
added. That is, Rv is obtained by merging query q to thetplanned route Rv′ , the first term δRdenote the arrival delayv,qof q to its destination oq , and the second term denotes theincrease in delay time of each accepted query q ′ ∈ Qv byinserting q into the schedule.
Algorithm 1 shows the request assignment algorithm of the
SBI method. Whenever the system receives a new request q,
it executes Algorithm 1 and assigns q to one of the vehiclesv ∈ V . In lines 1-3, the cost of assigning q to each vehicle
v is calculated according to formula (1). minInsertCost(·)
is calculated by adding stop-points for q to the arbitrary locations in the stop-point list Pv = (p1 , p2 , . . . , pn+1 ) for vehiclev, computing the cost of every case based on formula (1), andfinding the minimum among them. In other words, this algo-63

International Workshop on Informatics ( IWIN2023 )Algorithm 2 Transfer Scheduling Algorithm
1: Vs ← emptyList()
2: for v in V do
3:
if ac ∈ Rv then4:
append(Vs , v)t
5:compute δRv,ac
6:
end if
7: end for
8: sortVehicles(Vs )9: for i = 1 to |Vs | do10:
for j = i to |Vs | do
11:
ps ← getTransferPlace(v1 , v2 )12:
Cv1 ,v2 ← calcTransferCost(v1 , v2 , ps )13:
if Cv1 ,v2 < γ then14:
Rv1 ← changeDest(Pv1 , ps )15:
Rv2 ← insertPlace(Pv2 , ps )16:
remove (Vs , v1 , v2 )17:
end if18:
end for
19: end foris notified of the updated planned route and is instructed totransfer.

3.4 Calculation of Transfer Cost
The system calculates the transfer cost, which is the delay
time incurred by the transfer, for each of the transfer vehicle
pair (v1 , v2 ). When the transfer cost is less than threshold γ,
the system conducts transfer. Transfers are made such thatv1 first arrives at the transfer point ps , drops off all users, v2
arrives there, and picks up all of them. Therefore, the transfer
cost is determined by the total increase in arrival time for eachuser.
Let n1 and n2 be the number of boarding users in vehicles
v1 and v2 , respectively. Since the planned routes include ps
for both v1 and v2 , transfers do not alter the route of the vehicle. Let δtran be the time required for users boarding andalighting at ps . If the current time is t, the time when v1 ,tt
v2 arrive at ps be denoted as δR
and δR,respectively.
v1 ,psv2 ,psTherefore, the transfer cost can be calculated as follows.

t, which are their scheduled arrival
sort these vehicles by δRv,actime to ac , and create a list of vehicles Vs = (v1 , v2 , . . . , vn )t
in the ascending order. Let t be the current time, and δR
≤
i ,ac
t
δRj ,ac meets for any i < j. A nested loops are executed on
Vs to retrieve pairs (vi , vj ) in the order of the arrival time toac , and if the transfer condition is satisfied, the transfer point
and transfer schedule are calculated and applied. The transfer
conditions are that, first, the destinations are the same, andsecond, that the transfer cost to the users si lower tha a predetermined threshold. The transfer cost is explained in the next
section.
After determining the transfer route for the vehicle pairs(vi , vj ), it is necessary to calculate where the transfer willtake place. Let Rvi ,ac and Rvj ,ac be the planned routes of vi
and vj , respectively. Both of these planned routes include ac
and therefore two vehicles will eventually converge. Therefore, the vehicle pair shares a common route. Among these,
select the road segment as that is the furthest from ac andon which no congestion is detected, and set the appropriate
transfer place on as as the transfer point. If there is no suitable transfer point that meets the conditions, the vehicle pair(vi , vj ) is not instructed to transfer.
Algorithm 2 shows the algorithm for computing the transfer schedules. In lines 1-7, we create a list Vs of all vehicles
whose planned route includes the congested road segment ac ,and compute the arrival time to ac of each vehicle v ∈ Vs . Inline 5, t is the time when the Algorithm 2 is executed. In line
8, Vs is sorted based on arrival time to ac , and in lines 9-10,
all transfer vehicle pairs in Vs are generated with the priority
given based on the arrival time. In line 11, the transfer pointis calculated, and in line 15, caluculated the transfer cost. Inlines 12-16, when the transfer cost is less than the thresholdvalue γ, a transfer schedule is applied. In lines 14-15, a newplanned route is created by inserting a transfer point into eachof the planned routes of v1 and v2 . In line 16, v1 and v2 areremoved from Vs so that they will not appear in the transfer
vehicle calculation. After Algorithm 2 finishes, each vehiclett
Cv1 ,v2 = (n1 + n2 )δtran + n2 (δR− δR).
v2 ,psv1 ,ps

(2)

4 CONCLUSION
Compared to traveling by private vehicle, using ride-sharing
services allows multiple people to travel in a single vehicle,
leading to an anticipated reduction in traffic volume. In thenear future, in a society where ride-sharing services are widelyadopted, the proportion of people using ride-sharing services
without owning private cars is expected to significantly increase In this study, we propose a method to mitigate congestion by reducing the number of ride-share vehicles on roads
near the destination and issuing transfer instructions whencongestion is detected. This approach aims to address congestion issues in a society where ride-sharing services are prevalent.

REFERENCES[1] Ma Shuo, Zheng Yu, and Wolfson Ouri: Real-time CityScaleTaxi ride-sharing, IEEE Transactions on Knowledge and Data Engineering VOL27, NO.7, pp. 17821795 (2015)．
[2] Ayano Okoso, Keisueke Otaki, Satoshi Koide, andTomoki Nishi: Path Optimization with Future Demandfor Ride-sharing(in Japan), Transactions of Information
Processing Society of Japan Vol.60 No.10, pp. 16531661 (2019)．
[3] Noda Itsuki, Ohta Masayuki, Shinoda Kousuke, Kumada Yoichiro, and Nakashima Hideyuki: Evaluationof usability of dial-a-ride systems by social simulation, Lecture Notes in Computer Science, pp. 167-181(2003).

64

International Workshop on Informatics ( IWIN2023 )Development of a Framework for Jaywalking Risk Map
to Reduce Pedestrian-to-Vehicle Accidents
Yuichi TOKUNAGA*, Atsushi Yamamoto**, and Masashi SAITO**

Faculty of Graduate School of Engineering, Kanazawa Institute of Technology, Japan**
Department of Management Systems, Kanazawa Institute of Technology, Japan{y.tokunaga, msaito}@neptune.kanazawa-it.ac.jp

Abstract - Crossing outside of a crosswalk or ignoring a
traffic signal is called jaywalking. Jaywalking is a major roadsafety issue because even vehicles with safety features havedifficulty predicting jaywalking suddenly crossing the roadfrom the shadow of an obstacle. We hypothesize that thepossibility of jaywalking is related to the road and its
surrounding environment and propose a method to quantify
the risk using a statistical method and convey it to travelingvehicles. We have developed a framework for generating risk
potentials using Bayesian networks to realize this proposal. If
our hypothesis is correct, we should be able to estimate therisk potentials of jaywalking using our generated modeloutside of the observed location. We plan to verify theaccuracy of the risk maps generated by this framework bycomparing them with actual measurements.

prevent jaywalking. Regulatory signs can be installed invarious locations due to their ease of installation, but they are
not very effective, and neither of them is a fundamental
solution to the problem.
Therefore, instead of eliminating jaywalking, we should
consider foreseeing pedestrian’s jaywalking and avoidingthem on the vehicle side. As mentioned above, it is difficultfor vehicles to be prepared for jaywalking from all points, but
by identifying points with a high probability of jaywalking, it
is possible to reduce speed only there, thereby achieving both
safety and avoidance of traffic congestion. To achieve this,
this paper proposes to statistically determine the risk of
jaywalking on a road.
Section 2 introduces related works on jaywalking
countermeasures, Section 3 describes the proposed
framework based on the hypotheses derived from thesestudies, Section 4 describes its implementation in detail.
Finally, in Section 6, we discuss the results and future works.

Keywords: Jaywalking, Risk map, Bayesian network.

1

INTRODUCTION

2

In recent years, the number of accidents involving
automobiles has been on a downward trend due to theimprovement of the road traffic environment, the spread oftraffic safety awareness, and improvements in safety
performance of vehicles. However, the number of accidents
involving people versus vehicles has remained flat in recent
years [1]. In particular, the number of people killed in
accidents while walking continues to be the leading cause ofdeath by condition and has begun to increase slightly. In
addition, crossing violations and disregard for traffic signals
account for more than 45% of the causes [2]. These dangerous
behaviors of pedestrians are called "jaywalking" andcountermeasures against them have become an issue in roadtraffic, including overseas [3]. It is assumed that elderly
pedestrians jaywalk due to the overconfidence brought on by
years of experience despite their declining physical ability
and sensory perception. Although automobile pedestrian
detection has improved over the years, the braking distance isinsufficient for an emergency stop against a Jaywalker
emerging from between vehicles in the oncoming lane.
Prioritizing safety requires a significant reduction in speed,
resulting in traffic congestion. As a result, it is difficult to
avoid jaywalkers who may cross from anywhere at any time.
Common countermeasures against jaywalking include theinstallation of anti-crossing barriers (guardrails) andregulatory signs. Anti-crossing barriers are installed between
the roadway and sidewalk to physically deter people fromcrossing, and are considered to be an effective measure to

RELATED WORK

More than 70% of those killed in walking accidents are
elderly with older than 65 years old [2]. According to an
analysis of accidents involving elderly people crossing theroad by the Traffic Accident Analysis Center [4], 51% of
elderly people who had accidents while crossing the road did
not notice the approach of a car because they did not checkfor safety or check insufficiently. A typical accident scenario
is shown in Figure 1. The pedestrian starts crossing
immediately after A1's car passes, continues crossing without
checking left and right directions, and crosses without
noticing B1's car approaching from the opposite direction.
Technology to detect and avoid invisible pedestrians isbeing considered as the features of Advanced Driver
Assistance System (ADAS) for automobiles [5]. For example,
research is being conducted on technologies to recognize the

Figure 1: Mechanism of accident while crossing

65International Workshop on Informatics ( IWIN2023 )3

movement of invisible objects from images reflected incurved mirrors and reflective objects [6][7], and to sharesensing information of other vehicles, GPS information ofpedestrians, and image or sensing information from
infrastructure facilities through V2V, V2P, and V2I
communications respectively and to detect the danger range[8]. However, all these technologies are dependent on
outdoor reflective objects, sensors and cameras, andpedestrian communication devices, and thus lack versatility
and do not provide fundamental solutions to the problem.
The use of risk potential maps has been proposed to reduce
accidents caused by unseen hazards [9]. A risk potential map
shows not only the hazards physically present in the driving
path, but also the degree of danger in a space based on pastcases and probability, using numerical values and colors like
contour lines. For example, an intersection without a signal,
which is a blind spot at a wall, has a higher danger level
because of the risk of ejection. The risk map generated in thisway is used to realize safe driving by presenting the driverusing Head-Up-Display (HUD) and Augmented Reality (AR)or by referring in the automatic driving control to plan the
low-risk routes or to control speed. Several studies anddemonstrations have been conducted, but the challenge is
how to narrow down the risk; if the risk map is applied to
jaywalk, generalized to roads without pedestrian crossings,
the jaywalk risk is high, and cars are always forced to drive
at low speeds. On the other hand, if only points where
jaywalking has occurred in the past are considered high risk,
accidents caused by jaywalking at other points cannot be
prevented. It is necessary to narrow down the risk value withmore accurate information.
On the other hand, there have been many studies onpedestrian crossing behavior from the viewpoint of roaddesign in the field of civil engineering planning [10][11].
Hamamoto et al. studied pedestrian behavior at a nationalroad intersection leading to a station connection [12], andexplained that pedestrians want to take the shortest routepossible, and if there is a possible crossing on the route, they
will cross even if it is outside the pedestrian crossing area.
The study notes that jaywalking is more likely to occur at
locations where crossing cannot be physically deterred, suchas street intersections, entrances to roadside facilities, andopenings in anti-crossing barriers. Takehira also studied therelationship between roadway facilities and jaywalking on
several roads in densely populated urban areas [13]. Unlike
national roads, anti-crossing barriers on urban streets areoften characterized by disconnection at the approaches tostores and parking. In all the study locations observed, morethan 90% of the disorderly crossings occurred at locations
where there were openings at both the start and end points.
Thus, it is shown that the potential risk of jaywalking may
be inferred from the road environment, and anti-crossing
barriers are an important clue in estimating jaywalk risk. On
the other hand, many community streets and suburban roads
are not equipped with anti-crossing barriers, and the presenceor absence of such barriers alone cannot narrow down thepotential risk area of jaywalking. It is essential to discoverenvironmental features other than anti-crossing barriers forrisk mapping that aims at both safety and traffic efficiency.

PROPOSAL OF THE FRAMEWORK
3.1 Target Setting

Related works have shown that the opening of an anticrossing fence affects the probability of jaywalking occurring.
So, if we create a model to estimate the probability of
jaywalking based on these road environment characteristics
in a certain limited area and expand it to the whole country
using map information and this estimation model, we cancreate a national jaywalking risk map. However, to narrow
down the risk area, it is necessary to discover features otherthan anti-crossing barriers.
Therefore, we formulate a hypothesis based on use cases.
Facilities where people gather, such as train stations andschools, are often located on main streets, and pedestrians
targeting the facility flow into the main street from side streetsnear the facility, bypassing nearby pedestrian crossings tofollow them to the facility. If there are no crosswalks nearby,
the behavioral psychology of choosing the shortest route is atwork, and pedestrians are more likely to cross at the point
where they exit the side street onto the main street. Based on
this use case, it is highly likely that the mutual location of the
facility where people congregate, the influx of side streets inthe vicinity, and the crosswalks affect the potential risk
probability of jaywalking. The goal of this study is tosubstantiate this hypothesis.
To achieve this goal, we will generate a jaywalking
estimation model based on the location of facilities and otherfeatures and crosswalks and evaluate whether the risk map
derived from the model is valid outside of the observation
points.

3.2 Framework Construction
Creating a jaywalking estimation model requires thedevelopment of factors such as facility type and feature
extraction, as well as observational data. On the other hand,
whether an appropriate risk map can be obtained depends on
the hypotheses based on use cases, and there is no guaranteethat the results will be worth the effort of creating the model.
Therefore, the initial stage is to generate a model from aminimal dataset to prove the hypothesis, while building aframework to extend the missing variables and data.
An overview of the framework is shown in Figure 2. First,
location information of features in the road environment thatinduce jaywalking, such as facilities and pedestrian crossings,
are automatically obtained using open data. Next, location
dataset is created, and an estimation model is generated.
Utilizing the created estimation model, the probability ofoccurrence of jaywalk is estimated. Finally, a heat map issuperimposed on the map using the occurrence probability as
the risk value to create a jaywalking risk map. Each of thesefunctions is described below.

3.2.1. Extraction of location information on theroad environment
Google Maps is used to automatically acquire location
information of features on the road environment that trigger

66

International Workshop on Informatics ( IWIN2023 )From the hypotheses set up in this paper, 𝑥𝑖 indicates therelationship between the location of the facility at theobservation point, the road structure, and the location of thepedestrian crossing. The model is generated by data on the
jaywalking situation at each observation point, analyzing thedependencies between each causal event and determining theconditional probability of each relationship.
Bayesian estimation was used to create the estimation
model. Details of the location dataset and estimation modelare described in 4.2.2 and 4.2.3.

3.2.3. Generating a Jaywalking Risk Map
Finally, a jaywalking risk map is created by overlaying aheat map on a map with the probability of jaywalking as therisk value. The colors displayed on the heatmap should be
separated according to the risk values so that the information
can be obtained visually.

4

Figure 2: Framework overview

The proposed framework automatically obtains the latitude
and longitude of structures from Google Map searches anddetects pedestrian crossings from aerial images to obtainlocation information. Based on these location relationships
and the observed information on the availability of
jaywalking, a Bayesian estimation model is created, and a
jaywalking risk map is generated. This section describes theacquisition of crosswalk location information, creation of theBayesian estimation model, and generation of the risk map.

jaywalk. Facilities such as stations and schools are listed by
the search function and their latitude and longitude
information is obtained. On the other hand, pedestriancrossings cannot be extracted by the search function. As amethod to address this problem, a study has been reported thatanalyzes aerial photo images to detect pedestrian crossings[14][15], and we will use the same method. The method of
extracting crosswalks is explained in 4.1.2.

4.1 Extracting Crosswalk Locations

3.2.2. Generating an Estimation Model
Next, a location dataset is created from the calculated
latitude and longitude information. The presence or absenceof jaywalking is observed in advance from visual surveys,
and the data is created by calculating the distance anddirection for each latitude and longitude of the observation
location. The created data is compiled as location data, andan estimation model is created using a Bayesian network.
Bayesian network is a probabilistic inference model thatshows multiple causal relationships in a weighted graphstructure (network) and indicates each causal relationship by
probability [16]. Even when the full picture of causal
relationships is not known, the estimation accuracy can be
improved by accumulating findings obtained throughobservation as partial causal relationships. We believe that
this is the best method for this project to clarify the causal
relationship of the jaywalking outbreak through repeatedhypotheses and observations.
Bayesian network can be expressed by the following
equation :4.1.1. Image Collection
Using the location data (latitude and longitude) of the
facility retrieved by Google Maps, aerial images centered on
the facility (or its center of gravity in the case of multiple
facilities) are collected from the Google Maps Platform(hereinafter referred to as "GMP"). GMP outputs a 70-metersquare aerial image by specifying the location data of thecenter of the image to be acquired. Assuming a 1 km squarerisk map, the location data sets of 15 x 15 = 225 images mustbe specified to GMP. To minimize gaps and overlaps at image
boundaries, the location data of the center point of each image
was determined using the Vincent method from the latitude
and longitude of the center point of the target area, and the
distance and direction of each image. These data sets were
input into GMP to obtain the images.

4.1.2. Extraction of crosswalks

𝑁

𝑃(𝑥1 , … , 𝑥𝑁 ) = ∏ 𝑃(𝑥𝑖 |𝑝𝑎𝑟𝑒𝑛𝑡(𝑥𝑖 ))IMPLEMENTATION

Although several methods have been proposed forextracting pedestrian crossings from aerial images, we apply
YOLO v5, which is widely used for general object detection
applications. We trained on 300 crosswalk images extracted
from aerial photographs and obtained a high detection
accuracy with an F value of 0.95 by setting confidence>= 0.8.
Using the acquired crosswalk coordinate information andthe latitude and longitude information of the images, the

(1)

𝑖=1

where 𝑥1 denotes each event and 𝑝𝑎𝑟𝑒𝑛𝑡(𝑥𝑖 ) denotes theupper event in the relationship. The probability of occurrenceof a given combination of events is expressed as the productof the conditional probabilities of each causal relationship.

67

International Workshop on Informatics ( IWIN2023 )Map data (c) OpenStreetMap contributors, CC-BY-SA

Figure 3: Observation points of jaywalking
latitude and longitude of the crosswalk are calculated YOLO
outputs the coordinate information of the bounding box of thedetected object. Since the latitude and longitude informationof the center of the original image is known, the latitude andlongitude information of the center of the bounding box was
calculated from this information, and this was used as thelocation of the crosswalk.

4.2 Creating a Bayesian network Model
4.2.1. Collection of training data
Figure 4: Trends in occurrence of Jaywalking

To obtain the training data, a visual survey of jaywalking
was conducted. Observations were made for in an area
centered in front of the main gate of the university, mainly
during the hours when the school attendance wasconcentrated, and the starting point of the jaywalking wasrecorded. Although the percentage of pedestrians killed in
accidents while walking is higher among the elderly, weobserved jaywalking among all age groups of pedestrians, as
jaywalking is done regardless of age. The overview of thestudy area is shown in Figure 3 and the time trends ofobserved jaywalking are shown in Figure 4. Red boxes in
Figure 3 indicate locations where jaywalks were observed,
and black boxes indicate locations where a particularly large
number of jaywalks were observed. Many jaywalkers wereobserved at the intersection of side streets to the main street,and many pedestrians were observed jaywalking not only toand from the university but also to and from the supermarket.

4.2.2. Creation of Dataset
The dataset consists of five items: the number of jaywalkers,
the distance and direction to the nearest facility, and the
distance and direction to the nearest pedestrian crossing.
Road structure was excluded from the item because thecriteria for categorization had not been clarified. This is oneof the issues to be addressed in the future.
The visual survey results are entered into the number of
jaywalking occurrences in this dataset. The survey in thisstudy covered an area of 80 meters from north to south andFigure5: Created Bayesian network
800 meters from east to west. This corresponds to 80 mesh x
800 mesh = 68,000 records.

4.2.3. Estimation Model Generation
BayoLinkS [17], software for building Bayesian network,
was used for model generation. This software can learn

68International Workshop on Informatics ( IWIN2023 )5structures from data, extract dependencies (Bayesian
networks), and generate estimation models. Since the dataneeded to be discretized, the distances in the dataset wereequally divided every 50 meters, and the directions werelabeled in eight levels: east, west, south, north, and south. The
number of divisions for discretization has a significant impact
on model performance, so the appropriate number ofdivisions is an issue to be considered in the future.
The Bayesian network constructed by structural learning isshown in Figure 5. The solid line shows the dependencies,
and the table below the items shows the probability ofoccurrence of each state relative to jaywalking probability. In
the current study, the distance to the facility and the crosswalk
and the direction of the crosswalk are directly causally related
to the occurrence of jaywalking, while the direction of the
facility is causally related to the direction of the crosswalk.
This is considered to be a result of the influence of the learned
terrain and is a challenge when generalization, i.e., when themodel is used to estimate other areas.

CONCLUSION

This study developed a jaywalking risk map that visualizes
the probability of jaywalking occurrence as a potential risk
with the aim of preventing jaywalking, one of the causes ofhuman-vehicle accidents. Based on related works, wehypothesized that the mutual location of facilities where
people gather, the influx of side streets around them, andpedestrian crossings affect the occurrence of jaywalking, andaimed to create an environment that can be used as aframework to materialize this hypothesis. We extracted thelocation of keyword facilities and crosswalks from maps andaerial imagery, generated a Bayesian network risk estimation
model along with observation data, and displayed the risk of
jaywalking on the road in a QGIS heat map.
We generated an estimation model using this framework
based on observations of signal neglect in the vicinity of auniversity facility, and confirmed that it is possible togenerate a Jaywalking Risk Map for an area that is larger thanthe observed area. Areas without pedestrian crossings inresidential neighborhoods show a high level of danger, whileareas near pedestrian crossings show a low level of danger, as
expected. In the future, additional jaywalking observations
will be conducted for the risk map areas, and the accuracy
will be verified by comparing the generated maps with actual
measurements.

4.3 Risk Map Generation
The risk maps are generated in QGIS, using the estimation
model generated by BayoLinkS to obtain the probability of
jaywalking occurrence for each mesh location, which is thenconverted into a heat map. Since we want to show a risk map
for a road, we obtain road information from OpenStreetMap
and display the probability of occurrence only for the meshlocation that overlaps the road. The created risk map is shownin Figure 6. The circled dots represent pedestrian crossings.
The green boxes indicate visual survey locations, i.e., areasof correct data. The upper left corner of the map indicates ahigh-risk area. This area is considered high-risk because it isa residential area and there are no pedestrian crossings. On
the other hand, there are no areas near pedestrian crossingsthat are indicated as high risk, indicating that the distancefrom the pedestrian crossing has a significant impact.

ACKNOWLEDGEMENT
This research was supported by a Grants-in-Aid for
Scientific Research(C) numbered 21K11856.

REFERENCES[1] Cabinet Office, FY2022 Status of Traffic Accident andCurrent State of Traffic Safety Measures(in Japanese),
WHITE PAPER ON TRAFFIC SAFETY IN JAPAN2023, pp. 50-70 (2023).
[2] National Police Agency, Traffic Accidents in 2022 andOther Matters(in Japanese), https://www.npa.go.jp/
news/release/2023/20230302jiko.html (19 July 2023)[3] Millington, G., Jaywalking, In Shades of Deviance,
Routledge, pp. 17-20 (2014).
[4] H. Shibasaki, Accidents Involving Elderly People
Crossing the Street (in Japanese), Institute for Traffic
Accident
Researchand
Data
Analysis,
https://www.itarda.or.jp/presentation/19 (31, Jan 2023)[5] L Liu, S. Lu, R. Zhong, B. Wu, Y. Yao, Q. Zhang, W.
Shi, Computing Systems for Autonomous Driving :State of the Art and Challenges, IEEE Internet of Things
Journal, vol. 8, no. 8, pp. 6469-6486 (2021)[6] Y. Shi, X. Ying, and H. Zha, Unsupervised Domain
Adaptation for Semantic Segmentation of Urban Street
Scenes Reflected by Convex Mirrors, IEEE
Transactions on Intelligent Transportation Systems, vol.
23, no. 12, pp. 24276-24289 (2022)[7] R. Sato , and K. Onoguchi, Scene Recognition for Blind
Spot via Road Safety Mirror and In-Vehicle Camera,
26th International Conference on Pattern Recognition(ICPR), pp. 3714-3720 (2022)

Map data (c) OpenStreetMap contributors, CC-BY-SA

Figure 6: Jaywalking risk map

69

International Workshop on Informatics ( IWIN2023 )[8] A. Kabil, K. Rabieh, F. Kaleem, and M. A. Azer,
Vehicle to Pedestrian Systems: Survey, Challenges andRecent Trends, IEEE Access, vol 10, pp. 123981123994 (2022)[9] P. Raksincharoensak, T. Hasegawa, and M. Nagai,
Motion Planning and Control of Autonomous Driving
Intelligence System Based on Risk Potential
Optimization Framework, International Journal ofAutomotive Engineering, vol. 7, no. 1, pp.53-60 (2016)[10] Y. Nagawaki, and K. Suzuki, Empirical Analysis of
Factors for Jaywalking and Its Preventive Measures on
Urban Road, JSTE Journal of Traffic Engineering, vol.
8, no. 2, pp. A_149-A_158 (2022).
[11] K. Adachi, K. Suzuki, Y. Okada, and H. Ito, Empirical
Analysis of Jaywalking Behavior on Two-stage
Crosswalk, JSTE Journal of Traffic Engineering, vol. 7,
no. 2, pp. A_102-A_109 (2021).
[12] K. Hamamoto, and T. Murayama, The Safety
Evaluation Analysis of the Walker in the Crosswalk andStation Junction, JSCE, vol. 39, No. 402, pp. 1-4 (2009)[13] S. Takehira, Study on Design and Operation of Urban
Arterials Based on Nature of the Pedestrian Behavior,
University of Tokyo (2014)[14] Y. Ishino, and H. Saji, Extraction of Road Markingsfrom Aerial Images, 2008 SICE Annual Conference, pp.
2180-2183 (2008)[15] A. Maeda, Automatic Generation of a Pedestrial Road
Network Using Aerial Photographs and Maps, Journalof Information Processing, vol.61, no.2, pp. 262-273(2020)[16] D. Heckerman, A Tutorial on Learning with Bayesian
Networks, Innovations in Bayesian Networks: Theory
and applications, pp. 33-82 (2008)[17] NTT DATA Mathematical Systems Inc., "BayoLinkS",
https://www.msi.co.jp/english/index.html (11 June
2023)

70

International Workshop on Informatics ( IWIN2023 )Obstacle avoidance with extended velocity obstacles algorithm
Takumi Seita*, Shunsuke Michita*, Seiji Komiya* and Toshihiro Wakita**

Graduate School of Engineering, Kanagawa Institute of Technology, Japans2384004@cco.kanagawa-it.ac.jp

Abstract - Many traffic accidents occur on community
roads where pedestrians and vehicles coexist, therefore safe
and smooth autonomous driving is expected. Based on thissituation, this paper proposes a path planning method forautonomous vehicles by extending the Velocity Obstacles
algorithm (VO). VO is commonly used in mobile robotics.
Three points were extended to apply the algorithm.
The first point was to use 2D TTC to reduce unnecessary
avoidance. It was decided that avoidance would not be
performed if there is room for it based on the 2D TTC
calculation of collision time. The second point establishedmotion constraints by assuming avoidance is performed withsteady-state circular motion because vehicles have moremotion constraints than robots. The third point was to change
the collision detection method from circle approximation.
Considering the size and shape of the vehicle, it was decided
to draw tangents from the four corners of the vehicle to the
obstacle and combine them.
Using the above algorithm, avoidance paths were generated
and better performance than VO method.

DWA [2] is an approach that generates trajectory
candidates by combining translational and rotational
velocities. It evaluates each generated trajectory using anevaluation function. Following that, the trajectory with thehighest evaluation value is selected, and the robot follows that
trajectory. One of the problems of this method is that it does
not consider dynamic obstacles, which may result ininsufficient avoidance of dynamic obstacles such aspedestrians.
The risk potential method [3] defines repulsive potentials
and an attractive potential to the target reaching position to
obstacles. It generates target velocities and target paths basedon the calculated potential gradients. One of the problems ofthis method is that the generated potential field is a virtualquantity without physical meaning. Due to the lack ofphysical significance, parameter adjustments are required,making it difficult to achieve proper control of the trajectory,
velocity, and adaptation to various traffic environments.
Velocity Obstacle (VO) [4] is a method for motion
planning of a robot in a dynamic environment with moving
obstacles. This method defines a "velocity obstacle" as thevelocity space within which a robot may have potential
collisions based on the current positions and velocities of therobot and obstacles. By calculating velocity obstacles for all
obstacles and selecting a velocity that does not belong to any
velocity obstacle, this method generates obstacle-avoiding
trajectories. As the latest algorithm in VO, Adaptive Velocity
Obstacle (AVO) [5] has been proposed. This method is
proposed to solve that if the robot chooses the critical velocityof the collision cone to minimize the time consumption, theerror of the sensor will lead to the collision. This approachshows that the expansion radius of the moving obstacle isadjusted adaptively according to the relative velocity anddistance between the robot and the obstacle. After, the DWA
is used to get the local path through optimizing the velocity
outside the collision cone. Regarding this method’s issues,
one could point out the limited range of selectable speeds
when the robot approaches obstacles and the expansion radiusincreases. Another algorithm Ellipse-based Velocity Obstacle(EBVO) [6] has been proposed. This approach first selected
the linear velocity with the EBVO, and then calculated theangular velocity proportional to the change rate of theboundary line of the EBVO with respect to the orientation of
the robot. However, since the robot’s angular velocity wasdepending on the user-defined parameter, the robot oscillated
or collided with obstacles at worst if the parameter was set in
the wrong way. And there is an improved version addressing
the issues of this EBVO [7]. A common issue among thesealgorithms is the lack of consideration for the kinematicconstraints of non-holonomic robots.
Among these methods, it was considered that VO could beapplicable to autonomous driving on diverse community

Keywords: Autonomous Mobility, Pedestrian prediction

1

INTRODUCTION

In recent years, although the reduction of traffic accident
fatalities has been decreasing, the decreasing trend of thenumber of traffic accidents on community roads has remained
relatively low. According to data from the Ministry of Land,
Infrastructure, Transport and Tourism, in 2021, there wereapproximately 80,000 traffic accidents on community roads,
compared to approximately 220,000 accidents on arterial
roads [1]. In terms of the reduction rate since 2004, arterial
road accidents have decreased by about 70%, while
community road accidents have decreased by about 60%.
Therefore, in order to reduce traffic accidents on community
roads as well as arterial roads, the realization of safe andefficient autonomous driving is desired.
However, community roads have distinct characteristics
such as the lack of white lanes, pedestrians freely walking
along various paths, and narrow road widths, making asignificantly different traffic environment compared toarterial roads. Because there are numerous problems thatdiffer from traditional autonomous driving, it difficult toachieve autonomous driving on community roads.
Autonomous navigation in complex environments like
community roads has been extensively studied in the context
of mobile robots.
Some representative methods that have been traditionallystudied include the Dynamic Window Approach (DWA), therisk potential method, and the Velocity Obstacle (VO)method.

71

International Workshop on Informatics ( IWIN2023 )The collision cone 𝑪𝑪𝑨,𝑩 is a planar region bounded by
two tangents, 𝜆𝑓 and 𝜆𝑟 , from 𝐀 to the collision circle. In thisroads where numerous pedestrians act as moving obstacles
and there is a wide range of traffic environment variations.
However, applying VO to autonomous driving of vehicles oncommunity roads presents several problems.
The first problem is the existence of numerous pedestrians.
On community roads, there are often many pedestrians
walking in various directions along narrow roadways. When
applying VO to such situations, the velocity space may
become entirely covered by velocity obstacle regions, leaving
no viable velocity options to choose from.
The second problem is related to achieving the selectedvelocity. In VO, the velocity pair 𝑣𝑥 , 𝑣𝑦 that can reach thegoal fastest among velocities not belonging to the velocity
obstacle region is chosen. Many mobile robots are equippedwith differential drives or omnidirectional wheels, making it
relatively easy to achieve the selected velocity. However, incase of automobiles, which mostly adopt front-wheel steeringand have longer wheelbases, there are significant constraints
on achievable velocities. As a result, there can be a significant
difference between the selected velocity and the realized
velocity, potentially leading to inappropriate trajectory
generation.
The third problem relates to road width. Generally,
community roads have narrow roadways, which limits the
freedom of path selection. On the other hand, in VO, there arecases where a viable velocity is not selected despite being
capable of driving on narrow roads. One reason is that VO
approximates both pedestrians and the ego vehicle withcircles for collision detection. Since the shape of automobiles
often differs significantly in terms of width and length, there
can be a significant difference between the actual collision
detection and the circle approximation. Additionally, thecircular approximation tends to generate avoidance paths thattake unnecessarily large avoidance distances from obstacles.
As a result, on narrow community roads, there can besituations where no velocity candidates are available.
To address these problems, this study attempts to achieveautonomous driving on community roads by extending thecapabilities of VO. The extended method is referred to as VOdrive in this research.

2
2.1

region, the relative velocity 𝐯𝑨,𝑩 between the robot and the
obstacle would result in a collision. Thus, the region ofrelative velocities 𝐯𝑨,𝑩 that lead to a collision is determined.

Figure 1 Relative distance 𝐯𝑨,𝑩 and collision cone 𝑪𝑪𝑨,𝑩
Next, the region of robot velocities 𝐯𝐀 that would result in acollision is calculated. The obstacle velocity 𝐯𝐁 is added toeach velocity in the collision cone 𝑪𝑪𝑨,𝑩 . This is equivalent
to translating 𝑪𝑪𝑨,𝑩 by the velocity 𝐯𝐁 (Figure 2). If therobot velocity 𝐯𝐀 lies within this translated collision cone, acollision would occur. Therefore, this translated collisioncone is referred as the "velocity obstacle."

Figure 2 velocity obstacle

METHODS

The selection of velocity for robot motion is performed asfollows. First, an initial velocity is calculated based on apredetermined magnitude of velocity, assuming no obstacles.
If the initial velocity lies outside the velocity obstacle, it isdirectly used for motion. If the initial velocity lies within thevelocity obstacle, the velocity closest to the initial velocity on
the outside of the velocity obstacle is selected and used formotion.

VO

VO is a method for motion planning of a robot in a dynamic
environment with multiple moving obstacles. In thisalgorithm, based on the current positions and velocities of therobot and obstacles, the velocity space of the robot that may
lead to future collisions is defined as the "velocity obstacle."
The positions of the robot and obstacle are presented as 𝐀 and𝐁, respectively, and their velocities as 𝐯𝐀 and 𝐯𝐁 . The size ofthe robot and obstacles are approximated by circles
circumscribing them. Each radius is presented 𝑅𝐴 and 𝑅𝐵 ,respectively. The circle centered at 𝐁 with a radius of 𝑅𝐴 +𝑅𝐵 is defined as the collision circle. In this case, the set of
relative velocities 𝐯𝑨,𝑩 that would result in a collision
between 𝐀 and 𝐁 forms a collision cone 𝑪𝑪𝑨,𝑩 (Figure 1).

2.2

VO-drive

To adapt VO to community roads and autonomous driving,
three extensions were made.

2.2.1. Limitation of the target object
The first extension addresses the issue of handling a large
number of pedestrians. In community roads, there may be
many pedestrians walking in various directions along narrow72

International Workshop on Informatics ( IWIN2023 )roadways. When applying VO to such a situation, the velocity
space is covered by the velocity obstacle region, and noavailable velocities become exist. To resolve this, a method
can be considered to limit the pedestrians (obstacles) thatneed to be avoided using some criteria. Possible criteria
include distance, direction, etc. In this study, a twoDimensional Time To Collision (2D TTC) based on the time
margin until collision is used [5]. It is known that 2D TTC is
related to pedestrians' sense of safety in collision avoidance
for mobile robots. Therefore, pedestrians with a 2D TTC
value above a certain threshold can be considered to have alower risk of collision and can be excluded from collision
detection.
Time To Collision (TTC) is a physical indicator used inAutonomous Emergency Braking (AEB) systems in vehicles.
This indicator represents the time until collision with aleading vehicle if the current relative velocity between theego vehicle and the leading vehicle is maintained. In theworld coordinate system, the ego vehicle's front-end positionand velocity are presented as 𝑥𝑒 and 𝑣𝑒 , respectively, while
the leading vehicle's rear-end position and velocity arepresented as 𝑥𝑙 and 𝑣𝑙 , respectively. In this case, the relative
distance, 𝑑𝑥 , and relative velocity, 𝑣𝑥 , between the hostvehicle and the leading vehicle can be calculated as 𝑥𝑙 − 𝑥𝑒and 𝑣𝑙 − 𝑣𝑒 , respectively (Figure 3).

Figure 4 2D TTC outline figure
Therefore, the value of the 2D TTC is given by the following
equation (2).
√(𝑥𝑏 − 𝑥𝑎 )2 + (𝑦𝑏 − 𝑦𝑎 )2 − (𝑟𝑎 + 𝑟𝑏 )(2)
𝑇𝑇𝐶2𝑑 =
2
2√(𝑣𝑏𝑥 − 𝑣𝑎𝑥 ) + (𝑣𝑏𝑦 − 𝑣𝑎𝑦 )Pedestrians with a 2D TTC value equal to or greater than acertain threshold are considered not requiring avoidance andare therefore excluded from the velocity obstacle calculations.

2.2.2. Velocity selection based on vehicle
motion constraints
The second point is addressing vehicle motion constraints.
Unlike mobile robots, automobiles have significant motion
constraints, so an extension was made to select velocities
achievable by car steering.
First, assuming the vehicle is either moving straight or
performing a steady-state circular motion, the trajectory is
calculated [9]. The illustration of this is shown in the figure
below (Figure 5). Let 𝜃 be the angle between the X-axis andthe vehicle's longitudinal direction (yaw angle), 𝛽 be theangle between the vehicle's direction of travel and its
longitudinal direction (slip angle), 𝑟 be the angle between the
X-axis and the vehicle's direction of travel, and 𝑉 be thevehicle's velocity. The slip angle is determined by Equation(3), and the angle between the X-axis and the vehicle's
direction of travel is given by Equation (4).

Figure 3 TTC outline figure
Therefore, the value of TTC, presented as 𝑡𝑥 , can be
expressed by the following equation (1).
𝑡𝑥 = −

𝑑𝑥
𝑥𝑙 − 𝑥𝑒=−𝑣𝑥
𝑣𝑙 − 𝑣𝑒(1)

In this study, the 2D extension of TTC, referred to as 2D
TTC [8] was used. This index takes into account thepositional relationship in a 2D space considering the vehicle's
longitudinal direction (x-axis in the vehicle coordinate
system) and lateral direction (y-axis in the vehicle coordinate
system), whereas the previous TTC was calculated based on
the positional relationship along the vehicle's longitudinal
direction (x-axis) only. In the world coordinate system, theposition of the ego vehicle is represented as (𝑥𝑎 , 𝑦𝑎 ), theposition of the obstacle is represented as (𝑥𝑏 , 𝑦𝑏 ), the velocityof the ego vehicle is presented as (𝑣𝑎𝑥 , 𝑣𝑎𝑦 ), the velocity ofthe obstacle is presented as (𝑣𝑏𝑥 , 𝑣𝑏𝑦 ), the size of the ego
vehicle is presented as 𝑟𝑎 , and the size of the obstacle ispresented as 𝑟𝑏 . Therefore, the relative distance is calculated
as √(𝑥𝑏 − 𝑥𝑎 )2 + (𝑦𝑏 − 𝑦𝑎 )2 , and the relative velocity is

Figure 5 Vehicle trajectory prediction

2

calculated as √(𝑣𝑏𝑥 − 𝑣𝑎𝑥 )2 + (𝑣𝑏𝑦 − 𝑣𝑎𝑦 ) (Figure 4).

73International Workshop on Informatics ( IWIN2023 )𝑚 𝑙𝑓 2𝑉𝑙𝑟
2𝑙 𝑙𝑟 𝐾𝑟
𝛽=
𝛿
𝑙
𝐾
𝑙
𝐾
𝑚 𝑓 𝑓𝑟 𝑟 2 𝑙
1− 2𝑉2𝑙 𝐾𝑓 𝐾𝑟()

Step7 Repeat Step 1 to 6.
Here is the illustration of these steps (Figure 7).

1−

𝑟=

1
𝑉
𝛿
𝑚 𝑙𝑓 𝐾𝑓 − 𝑙𝑟 𝐾𝑟 2 𝑙
1− 2
𝑉
𝐾𝑓 𝐾𝑟
2𝑙
(
)

(3)

(4)

Figure 7 Outline of trajectory prediction

Therefore, the trajectory of the moving vehicle's center ofgravity is considered. The positions and velocities of thevehicle and the obstacle in the world coordinate system areshown in the figure below (Figure 6). Let (𝑥𝑒 , 𝑦𝑒 ) and(𝑣𝑒𝑥 , 𝑣𝑒𝑦 ) represent the position and velocity of the vehicle,
respectively, and (𝑥𝑝 , 𝑦𝑝 ) and (𝑣𝑝𝑥 , 𝑣𝑝𝑦 ) represent theposition and velocity of the obstacle. The obstacle is assumed
to move in a straight line at a constant speed, and its trajectoryis predicted.

However, there is a difference when combining the VO
method with trajectory prediction. VO assumes both the ego
vehicle and the obstacles move in constant linear motion andgenerate avoidance paths based on the assumption that thecurrent velocity will continue. On the other hand, trajectory
prediction for constant circular motion involves nonlinear
motion with constant translational and rotational velocities,
making it incompatible to directly apply the VO method.
Therefore, in this case, an attainment point based on thecurrent velocity that would result in a constant circularmotion was assumed. The line connecting the starting point
and the attainment point was approximated.
The generated trajectory prediction points for the ego
vehicle are shown in the figure below (Figure 8).

Figure 6 Each variable in the world coordinate system
Given the initial position (𝑋0 , 𝑌0 ) and the initial yaw angle 𝜃0 ,the position (𝑋, 𝑌) and the yaw angle 𝜃 at any given time 𝑡
can be calculated using equations (5), (6), and (7).
𝑡

𝑋 = 𝑋0 + 𝑉 ∫ cos(𝛽 + 𝜃) 𝑑𝑡(5)Figure 8 Example of trajectory prediction(6)

2.2.3. Collision detection based on the vehicle
contour0𝑡

𝑌 = 𝑌0 + 𝑉 ∫ sin(𝛽 + 𝜃) 𝑑𝑡
0

The third point addresses the issue of narrow road widths. In
the VO approach, both pedestrians and ego vehicles areapproximated as circles for collision detection. However, thisapproximation can result in significant differences between
the actual collision detection and the circle approximation.
Using circular approximation often leads to unnecessarily
large avoidance routes.
According to the Japanese Cabinet Office, a "community
road width" is defined as a road with a carriageway width ofless than 5.5 meters in urban areas. If the avoidance route is
too large, there is a risk of veering off the road or even thepossibility of being unable to avoid the obstacle altogether.
To achieve a compact and safe avoidance route, a collision
detection method that takes into account the vehicle's contour
was adopted (Figure 9).

𝑡𝜃 = 𝜃0 + ∫ 𝑟𝑑𝑡(7)

0

The actual calculation process is outlined below.
Step1 Provide steering angle for ego vehicle.
Step2 Calculate 𝛽 and 𝑟 using the steering angle from Step1
according to equations (3) and (4).
Step3 Using equations (5) and (6) from Step2, calculate thereachable points.
Step4 Calculate the velocity required to reach the speed in
Step3.
Step5 Convert the velocity in Step4 to the velocity within theVOmap.
Step6 Prepare a map similar to VOmap, and substitute thevalue of the steering angle at the velocity position in
Step5.

74

International Workshop on Informatics ( IWIN2023 )Figure 9 Comparison between circle approximation andcontour-based methods

Figure 12 Example of avoidance using vehicle contour

3 SIMULATION EXPERIMENT AND RESULTS

By utilizing the vehicle's contour, performing VO
calculations from the four corners of the vehicle to the
obstacles. Figure 10 shows velocity obstacle area. The
original VO area is shown in gray and extended method area
shown in black. This shows that it can be observed that
extended method results in a smaller area compared to theoriginal method. This approach allows for more precise
collision detection and avoidance planning, taking into
account the actual shape and size of the vehicle.

3.1

Using the VO-drive method proposed in this study, asimulation experiment was conducted to compare it with theoriginal VO (Velocity Obstacle) method.
The simulation was performed using MATLAB. A time
step of 0.1 seconds was used, and the initial value of thevehicle speed was set to 15 km/h. When the path generation
method instructed a specific speed, it followed the generated
speed by each route generation accordingly.
For VO-drive, the threshold for 2D TTC was set to 6
seconds as a limitation condition for the target obstacle. Only
obstacles with TTC values below this threshold areconsidered for avoidance. And, the attainment point was
defined as the point reachable 0.1 seconds later. For obstacle
behavior prediction, the walking speed of the tracked obstacle
was calculated using a Kalman filter, and the predicted
position was calculated based on the calculated walking
speed [10].

Figure 10 Caomparison of calculation methods between
original VO and extended method

3.2

When actually performing circular approximation in a
scenario simulating a community road environment, asshown in Figure 11, attempting to avoid obstacles leads todeviating from the lane due to the constraints of the roadwidth. Therefore, by performing calculations from the four
corners of the vehicle, it becomes possible to avoid obstacles
while staying within the lane width, as shown in Figure 12.
This approach ensures that the vehicle navigates safely within
the available space and minimizes the risk of deviating off theroad.

Figure 11 Example
approximationof

avoidance

using

Experimental conditions

Comparison between VO-drive and VO

First, a comparison was conducted between the original
VO and the extended VO-drive to confirm the effectivenessof the extension.
A comparative experiment was conducted on the limitationof the target obstacle. The results are shown in Figure 13. The
ego vehicle starts from the position (-15,0) in the world
coordinate system and moves straight at a speed of 15 km/h.
An experimental scenario was designed where the vehicle
needs to avoid a stationary obstacle located at (15,0). From
Figure 13, it can be observed that the extension leads to adelay in the timing of avoidance initiation.

circleFigure 13 Comparison of target obstacle limitation75International Workshop on Informatics ( IWIN2023 )5
Second, a comparative experiment was conducted on velocity
selection based on vehicle motion constraints. The results areshown in Figure 14. The ego vehicle starts from the position(-12,0) in the world coordinate system and moves straight at
a speed of 15 km/h. Two obstacles are introduced, eachmoving diagonally at a speed of 0.5 m/s, creating a crossing
scenario. From Figure 14, it can be observed that the proposed
method achieves smoother and more efficient avoidance,
while the original VO approach exhibits sharp turns.

FUTURE WORKS

Currently, the real vehicle experiments are still ongoing. In
the future, it is plan to conduct real vehicle experiments,
increasing the vehicle's speed and introducing moving
obstacles to create an environment closer to real-life urban
road conditions. In addition, by having pedestrians actuallywalk, it is aim to realize avoidance that makes them feel safe.

REFERENCES
Ministry of Land, Infrastructure, Transport and Tourism,
Community road traffic safety measures portal,
https://www.mlit.go.jp/road/road/traffic/sesaku/anzen.ht
ml.
[2] D. Fox, W. Burgard, and S. Thrun, “The dynamic window
approach to collision avoidance,” IEEE Robot. Autom.
Mag., vol. 4, no. 1, pp. 23-33(1997).
[3] Michael T. Wolf, Joel W. Burdick, “Artificial Potential
Functions for Highway Driving with Collision
Avoidance”, IEEE International Conference on Robotics
and Automation (2008).
[4] Fiorini P, Shiller Z, “Motion Planning in Dynamic
Environments Using Velocity Obstacles”, The
International Journal of Robotics Research, vol.17, no.7,
pp.760-772 (1998).
[5] Mengyuan Gu, Yunzhi Huang, “Dynamic Obstacle
Avoidance of Mobile Robot Based on Adaptive Velocity
Obstacle”, 36th Youth Academic Annual Conference ofChinese Association of Automation (YAC), IEEE (2021).
[6] Jeon, J. D., Lee, B. H. “Ellipse-based velocity obstacles
for local navigation of holonomic mobile robot”,
Electronics Letters, 50(18), 1279-1281 (2014).
[7] B. Lee, J. Jeon, J. Oh, “Velocity Obstacle based local
collision avoidance for a holonomic elliptic robot”,
Autonomous Robots, vol.41, no.6, pp.1347-1363 (2017).
[8] Yuto Yada, Shunsuke Michita, Seiji Komiya andToshihiro Wakita, “Path Planning considering PedestrianCharacteristics”, International Workshop on Informatics(IWIN 2022).
[9] Masato Abe, Automotive Vehicle Dynamics Trajectoryand Applications, Tokyo Denki University Press (2012).
[10] Takumi Goto, “Pedestrian behavior prediction andforecast circle generation using Kalman filter”,
Kanagawa Institute of Technology, Undergraduate Thesis(2019).
[1]

Figure 14 Comparison of vehicle motion constraints
Lastly, a comparative experiment was then conducted oncollision detection based on the vehicle contour. The results
are shown in Figure 15. The ego vehicle starts from theposition (-12,0) in the world coordinate system and moves
straight at a speed of 15 km/h, aiming to compare whether it
can navigate between the obstacles. From Figure 15, it can beobserved that the conventional circle approximation method
fails to navigate between the obstacles and takes asignificantly larger avoidance path, while the extension
allows the vehicle to pass between the obstacles successfully.

Title: Autonomous driving on community roads using small
mobility -Route generation using trajectory predictionand 2D TTCAuthor: Takumi Seita, Shunsuke Michita, Seiji Komiya,Toshihiro Wakita
Corresponding Author: Takumi Seita
Tel: 046-291-3091
Fax:046-242-6806
E-mail: s2384004@cco.kanagawa-it.ac.jp

Figure 15 Comparison of collision detection

4

CONCLUSION

By conducting a comparison of methods using simulations,
it was possible to demonstrate the usefulness of the proposed
VO-drive approach for autonomous driving on urban roads.
After conducting simulations, actual experiments werecarried out using a vehicle, and successful avoidance routes
were performed.

76International Workshop on Informatics ( IWIN2023 )ALiDARPointCloudDataFusionfromConnectedAutonomousVehiclesto
ReduceNetworkTrafﬁcCongestionatServer
PasupuletiHaripriya,YoshinaoWada,KenjiroYamada,TakuroSone,andTomoyaKitani
GraduateSchoolofIntegratedScienceandTechnology,ShizuokaUniversity,Japan{haripriya-p, y-wada, k-yamada, t-sone, t-kitani}@kitanilab.org
Abstract - In a road trafﬁc setting, autonomous vehicles
continuously generate LiDAR point cloud data, which is useful in generating and maintaining dynamic map of on-road
objects. This data can be used to form a Static Object Map(SOM) that identiﬁes static objects that are useful in identifying irregular objects like road defects, road construction andso on. However, transmitting large raw LiDAR data poseschallenges due to server-level network congestion. This paper
proposes a solution by developing communication and upload
protocols, along with an aggregation algorithm, to remove redundant data before transmission. Simulations demonstratesigniﬁcant reductions in data size, alleviating network congestion at the server, and improving efﬁciency compared toconventional methods.

Figure 1: Dynamic map layers and contents of SOM

2 Related work

In our research on road trafﬁc networks, we aim to alleviate congestion and reduce server load through communication
protocols and aggregation algorithm.
Shagufta Ali et al. [2] discuss wireless communication technologies in V2V communication, including DSRC, 4G-LTE,and 5G. DSRC excels in mobility but has scalability issues,
1 Introduction
while 4G-LTE faces challenges in high trafﬁc conditions. 5GV2X, leveraging ﬁfth-generation technology, offers enhanced
This paper explores key components in developing autonomous communication and coverage.
driving applications, which includes dynamic maps, high preChen Qi et al. [3] proposed a LiDAR-based 3D object decision 3D maps, LiDAR sensor technology, and efforts to retection method with data fusion for cooperative perception.
duce data size through V2V communication, difference exHongyu Li et al. [4] introduced a two-phase point cloud regtraction and aggregation. It emphasizes the importance of
istration mechanism, and Anand et al. [5] have researched
Static Object Map (SOM) for not only identifying shape of
the issue of the massive size of point cloud data produced bystatic objects like trafﬁc poles, road shape and so on, but also
LiDAR and have suggested Octree-based compression techhelpful in recognizing irregular objects in the road environnique. In point cloud registration, the Iterative Closest Point
ment like road construction, road defects and so on. The fo(ICP)Algorithm [6] and Normal Distribution Transform (NDT)
cus is to collect the LiDAR point cloud data acquired by auAlgorithm [7] are widely used, but challenges persist in accutonomous vehicles at server by updating SOM in autonomous
rately registering point clouds from vehicles due to high outvehicles based on the frequency of applications, such as static,lier ratios caused by different views and obstructions [4].
semi-static, and semi-dynamic information layers of dynamic
map[1], as shown in Fig. 1. The paper delves into LiDAR
3 Target problem
data, particularly its 3D perception capabilities, and the challenges possessed by the large size of LiDAR data collected by
3.1 Target system architecture
autonomous vehicles. To address this, we proposed a method
In this section, we describe the target VANET model. Ourfor V2V communication of LiDAR point cloud data, utiliztarget system architecture consists of a road environment sceing a voxel-based map of SOM and acquired LiDAR point
nario with cars, buildings, roads, and other infrastructures.
cloud data by autonomous vehicles to reduce the data size.
We consider each car is an autonomous driving vehicle with a
Evaluations demonstrate a signiﬁcant reduction in unstrucLiDAR sensor that can communicate with other vehicles andtured LiDAR data, alleviating network trafﬁc congestion at
infrastructure, and we also assume that every car has samethe server with the help of differece extraction, aggregation
version of SOM (in voxel form) stored in it’s database. The
and shared LiDAR point cloud data among neighbouring vetarget system architecture is illustrated in Fig. 2. Initially, auhicles. This approach proves more efﬁcient compared to contonomous vehicles acquire LiDAR point cloud data. Subseventional methods of individually sending pre-processed Liquently, this newly acquired LiDAR point cloud data is conDAR data to the server.
Keywords: LiDAR point cloud data, dynamic map, network trafﬁc congestion, communication protocol, aggregation algorithm

77International Workshop on Informatics ( IWIN2023 )verted into voxels and compared to the stored Static Object
Map (SOM) in the vehicle to identify differences, termed as”Difference Extraction.” All other autonomous vehicles undergo a similar process. The LiDAR point cloud data extracted during difference extraction is exchanged between vehicles for aggregation, aiming to eliminate redundant data.
The ﬁnal aggregated data is then uploaded to the server.

3.2.2 Communication technologies
Various communication technologies are utilized in vehicularcommunication networks. For the communication between
vehicles, DSRC technology can be used as its range is limited, and for the communication between vehicles and servers,
cellular technology is capable of transferring data to remoteareas such as servers.

3.3 Target object model of urban space
3.3.1 Data structure of target LiDAR device
In this experiment, we used LIVOX AVIA as LiDAR sensor1 .
The speciﬁcations of the LIVOX AVIA LiDAR sensor are described in the Table. 1 as shown below. LIVOX AVIA can
Table 1: LiDAR sensor speciﬁcations
Speciﬁcation
Details
Field of View(FOV) Non-repetitive 70.4◦ × 77.2◦
Point Rate
240KpsRange Accuracy
2cm
Angular Precision< 0.05ºReturn ModeFirst and Strongest

Figure 2: Target System Architecture

3.2 Assumptions
3.2.1 Road network
We assume the following road trafﬁc network as shown in
Fig. 3.
Vehicle ID’s: sender vehicle (V1 ), receiver vehicle (V2 )Intersection ID’s: {J1, J2, J3, J4}
Edge ID’s: {E1, -E1, E2, -E2, E3, -E3, E4, -E4, E5, -E5, E6,-E6, E7, -E7}
V1 ’s route path (R1): {-E0, -E1, -E2, -E3, E5}
V2 ’s route path (R2): {-E0, E4, E5}
Overlapped route path (R3): {-E0, E5}

measure 240,000 points per second, and the bit rate is calculated from this value. To consider the bit rate of speciﬁc data,
we will refer to the LVX format of point clouds that can beobtained from LIVOX AVIA.
According to “LVX Speciﬁcations” 2 , a document publishedby LIVOX, the LVX format stores the values of the Cartesian
coordinates of the data in 4-byte increments. Since there arethree items (X, Y, Z), this requires a total of 12 bytes of data.
In addition, in LIVOX AVIA, a Time Stamp value is assigned
to each of the 96 points, requiring a data volume of 8 bytes.
Therefore, there are 240,000 points with 12 bytes of coordinate value (X, Y, Z) data and 25,000 points with 8 bytes oftime information (TimeStamp) data, resulting in a total bit
rate of approximately 2.94MB/S.
The fact that the coordinate values (X, Y, Z) are expressedin 4 bytes each and the time information in 8 bytes is a generalvalue, not limited to the LVX format of point clouds that is acquired by LIVOX AVIA. This data volume is also used in the
LAS format [9], which is one of the point cloud ﬁle formats.
The LAS format is also used in “VIRTUAL SHIZUOKA”[10], which is an open data point cloud database publishedby Shizuoka Prefecture and is a common point cloud format.
If the ﬁle format obtained from LiDAR was in LAS format, there would be 240,000 points with a total of 20 bytes of
X, Y, Z, and T imeStamp, resulting in a bit rate of approximately 4.58MB/s.
1 Avia LiDAR sensor, https://www.livoxtech.com/avia

Figure 3: Road Network Scenario

2 LVX Speciﬁcations V1.1.0.0, https://www.livoxtech.com/
3296f540ecf5458a8829e01cf429798e/downloads/Livox\%20Viewer/LVX\%20Specifications\%20EN_20190924.pdf

78

International Workshop on Informatics ( IWIN2023 )3.3.2 Voxelization
Voxelization refers to the process of converting 3D objects
or scenes into a structured grid-like representation into cubic volumetric elements called as voxels. Voxels are similar
to pixels in images, but voxels deal with 3D representationand visualization of objects or scenes, whereas pixels deal
with 2D images. According to [8], LiDAR point cloud data
have the most accurate 3D description, but its unstructured
data organization can cause data density variation, noise &outliers, occlusions & shadows, misalignment & registration,and segmentation & object extraction. However, voxel gridsof LiDAR point cloud data can represents not only organizedspatial information from point clouds but also provides more
efﬁcient storage, processing, and analysis. An example ofvoxelized urban area is shown in Fig. 4

Figure 5: An example of voxelized SOM

4.2 Sensing and Data Compression phase
We assume that all the autonomous vehicles and server
has the same version of SOM in voxel form. In sensing anddata compression phase, all the vehicles acquire LiDAR point
cloud data along it’s travelled route path and convert into voxels through voxelization process. Then each vehicle performs
difference extraction. In which, the vehicle extract differenceswith the stored SOM and acquired LiDAR point cloud data.
The detailed explanation of voxelization process and difference extraction is explained in section 4.5. If there is any difference extraction, then the vehicle triggers control message
exchange protocol. Otherwise, there will be no communication established between vehicles.

Figure 4: An example of voxelized urban area

3.3.3 SOM (Static Object Map)
A High-precision 3D map typically contains static information such as road link information, lanes, guardrails, roadsigns, and pedestrian crossings at more accurate locations. It
gives the position of the road equipment and infrastructure,but it lacks the shape information of the static objects. So,
we proposed a target object model of urban space called as a
Static Object Map (SOM) that provides the structure of static
objects in the static information layer. We extended the SOM
mechanism to semi-static and semi-dynamic information applications in identifying irregular objects such as road construction, ﬂooded areas, road defects, and so on. In this paper,
we represented the static object map (SOM) as a set of structured voxel grids with static, semi-static and semi-dynamic
information, which refers to parked cars, that are indicated by
red circle as shown in Fig. 5.

4 Proposed method
4.1 Overview
Figure 6 shows the overall system ﬂow of the proposed method
to maintain and update SOM by acquiring LiDAR point clouddata from autonomous vehicles with less communication trafﬁc at the server.

4.3 Design of Control Message Exchange
protocolThe vehicles in the road trafﬁc environment communicate witheach other within the IEEE 802.11p DSRC protocol. Afterthe sensing and data compression phase, if there is any difference extraction available at the sender vehicle, then the control message exchange protocol is performed. The sender vehicle broadcasts the difference extraction LiDAR point clouddata Edge ID’s of road id list. The vehicle ID’s, edge ID’s
and road id list of route paths are derived in section 3.2.1.
The ﬂow of the control message exchange protocol is shownin Fig. 7.

4.4 Design of a LiDAR point cloud dataexchange decision algorithm
During the sensing and data compression phase, the receivervehicle is also able to extract the difference in SOM and acquired LiDAR point cloud data. The receiver vehicle receives
and checks the control message. The receiver vehicle compares own road id list with sender vehicle’s road id list to
identify any overlapping road id’s. Then the receiver vehicle requests the sender vehicle for LiDAR point cloud dataof the overlapping road id list as shown in Fig. 8. Next, thesender vehicle sends only the difference extraction LiDAR
point cloud data of overlapping road id list to receiver vehicle. We consider IEEE 802.11p communication technology
to exchange LiDAR point cloud data between vehicles.

79International Workshop on Informatics ( IWIN2023 )Figure 7: Control Message Exchange Protocol Flow

Figure 6: A ﬂowchart representing the overview of the proposed method

4.5 Design of LiDAR point cloud data
aggregation algorithm
To aggregate point clouds measured from multiple vehicles,
data reduction and fusion must be performed due to limited
line speeds. Algorithms for data reduction include voxelization and comparison with SOM, while point cloud synthesis
is an algorithm for fusing multiple point clouds.
In this study, voxelization is used to reduce the amount ofdata in a point cloud and makes it easier to exchange point
clouds. Voxelization reduces the coordinate resolution of apoint cloud. By reducing the resolution, the absolute number of points in a point cloud can be reduced by combining
neighboring points into a single point cloud. It is possible
to eliminate duplicate point clouds. However, doing so may
result in a decrease in accuracy.
Let us consider the speciﬁc amount of point cloud deletionon the surface area. Suppose that LiDAR, which can measure
240,000 points per second, evenly measures a surface withan area of 24/m2 . The original point density was one point
per cm2 , but the data size of the point cloud was reduced byconstructing voxels with a side of 2.5 cm. This means thatfor 1/m2 , there are 1600 points, which means that for 24/m2 ,there are 38,400 points. In other words, for a surface ﬁlledwith a point cloud of cm2 , the point density can be reduced
to about 1/6 with a side of 2.5cm. We regard this process asvoxelization.
To give you an idea of the voxelization process, we prepared a point cloud of the area near the fountain on the Hamamatsu campus of Shizuoka University. The original point
cloud data visualized by Houdini which is a 3D computer

Figure 8: LiDAR Point Cloud Data Exchange Decision Algorithm Flowgraphics (CG) software is shown in Fig. 9.
The point cloud in Fig. 10 is voxelized every 25 cm by
Houdini.
In this case, voxelization reduces the amount of data since
any number of points in a 25 cm square can be represented bya single cube.
In addition to voxelization, difference extraction from the
SOM is used as an algorithm for data reduction. The original
point cloud is as shown in Fig. 11 (a). Figure 11 (b) shows theextracted differences from the SOM. By extracting the differences, point clouds that overlap with the SOM can be found.
Therefore, point cloud data can be reduced.
Point cloud merging is a method of aligning and merging
point clouds measured from multiple vehicles. When point
clouds are merged, they combine into one single point cloud.
This allows multiple point clouds to be brought together andconsolidated into a master vehicle. Speciﬁc methods for merging point clouds include the ICP algorithm, which matches
the nearest points of two point clouds.
To compare the SOM and the synthesized point cloud withminimal error, we did not use any synthesis algorithm thatis dependent on initial values or introduces errors but simply
captured the two point clouds from the same location and superimposed them to reproduce the synthesized point cloud.
Since the ICP algorithm and other positioning methods do

80

International Workshop on Informatics ( IWIN2023 )(a) before extracting thedifferences(b) after extracting thedifferences

Figure 11: Extracting the differences between two point cloud
data
Figure 9: Raw LiDAR point data

Figure 10: 25cm-cube Voxelized data of Fig. 9
not remove duplicate point clouds, the point cloud synthesis
by superposition in this experiment does not differ from general registration methods, such as the ICP algorithm in termsof the amount of data.

4.6 Design of LiDAR point cloud data upload
timing decision algorithm
At this stage, the receiver vehicle contains the ﬁnal aggregateddata of difference extraction LiDAR point cloud data of bothsender vehicle and receiver vehicle. Finally, the aggregateddata is sent within a timeout of (t0 ) to the server as shownin Fig. 12, via a cellular communication technology due to
its large bandwidth and wide coverage area. The reason fortimeout (t0 ) is because the maximum update frequency varies
for static, semi-static and semi-dynamic applications. If theaggregated data is not uploaded to the server within the designated timeout (t0 ), it will be considered as expired and eventually be discarded. So, we set a timeout (t0 ) to upload theaggregated data to server.

5 Evaluation
To evaluate the proposed method, we have conducted an experiment that involves the test data collected at Shizuoka University, Hamamatsu campus. The collected test data is processed through computer simulations. A detailed explanationof the experimental environment and conﬁguration is given in
the following subsections.

Figure 12: LiDAR Point Cloud Data Upload Timing Decision
Algorithm Flow

5.1 LiDAR point cloud data aggregation
5.1.1 Experimental environment and conﬁguration
We conducted experiments on voxelization and difference extraction from the SOM to decrease the data volume of thepoint cloud.
The scanning time was set to 10.6 seconds and incremented
every 0.1 seconds for a total of 106-point cloud samples. The
reason for using 0.1 seconds as the unit is that the FOV coverage of a typical LiDAR reaches its upper limit after 0.1 seconds. According to Fig. 13 in the LIVOX AVIA User Manual published by LIVOX 3 , the scan pattern of LiDAR withthe conventional mechanism shows that the FOV coveragereaches its upper limit at 0.1 seconds.
In this study, we utilized the LIVOX AVIA scan pattern,which differs from conventional patterns. To adapt to theconditions, we divided the point clouds into 0.1-second increments. Although LIVOX AVIA has the same pattern as theconventional scan pattern, it is difﬁcult to adjust the angle ofview and include the entire object because of the narrow FOV.
Therefore, a non-repeating scanning method with a wide FOV
coverage was adopted.
For the experiment, LiDAR was initially positioned in astationary location, and point cloud V0 , which solely contains stationary objects, was created as a SOM. Additionally,
3 LIVOX AVIA User Manual, https://terra-1-g.djicdn.com/65c028cd298f4669a7f0e40e50ba1131/Download/Avia/Livox%20Avia%20User%20Manual%20202204.pdf

81International Workshop on Informatics ( IWIN2023 )Figure 13: Scaninng time comparison (from LIVOX AVIA
User Manual, p. 5)
LiDAR point clouds of V1 and V2 were prepared to include
stationary objects that were not present in the SOM V0 . LiDAR point cloud V2 includes irregular objects in addition to
the stationary objects found in V1 . And we estimate that V1
is a semi-static and V2 is a semi-dynamic data. The environment of the experiment is as follows: an original LiDAR
point cloud as SOM, LiDAR point cloud with a new box, and
LiDAR point cloud with a new chair is shown in Fig. 14.

ing vehicle with LiDAR sensor to accurately measure the point
clouds, which are closely linked to real-world trafﬁc conditions. However, the point cloud difference extraction method
used in this experiment is based on the SOM point cloud anduses a simple method of deleting the point clouds measured
from vehicles that are within the threshold value. The point
cloud gathered by LiDAR while in motion may contain inaccuracies from the IMU, GPS, and other sources, in addition
to errors from the LiDAR sensor itself. Therefore, if a point
cloud is extracted using a threshold value of 2cm measurement error of LiDAR along with the errors of IMU and GPS,
it is likely that a considerable number of point clouds will be
excluded.
Hence, for this instance, we immobilized both the LiDAR
and the object, thereby constraining the error factor exclusively to the LiDAR itself. This allows us to estimate theamount of data to be deleted from the point cloud measured
from the vehicle.
5.1.2 Acquired LiDAR point cloud data and aggregatedresults
First, each point cloud was voxelized. After performing voxelization, the average value of the total data count excluding
point group V0 (which represents the data for SOM) and the
reduction ratio can be summarized as follows. To calculate
the reduction ratio, we divided the number of points left after
voxelization by 24,000, which represents the number of samplings done in a 0.1-second time frame. As shown in Table 2,voxelization reduced the point cloud by more than 27.0%.
Table 2: Results after conducting voxelization
remaining points reduced ratio
voxelization V1
17333
27.8%
voxelization V2
17169
28.5%

(a) V0 : original as
SOM(b) V1 : new box
adding to V0(c) V2 : new chairadding to V1

Figure 14: Example measurement environment
The voxelization process utilized a LiDAR measurement
error margin of 2.0 cm, with each voxel having a side lengthof 2.5 cm. The center of gravity of the point cloud existing
within the 2.5 cm unit delimitation was used as the representative value of a single voxel.
The comparison was performed using an algorithm that removes all point clouds within 2.0 cm of the SOM (V0 ) from
V1 and V2 .
The LiDAR has a measurement error of 2.0 cm. Any points
beyond this distance can be considered non-existent in the
SOM (V0 ). In other words, there is a high possibility that
objects that do not exist in the SOM are being measured, anddifferential detection of point clouds is possible.
Ideally, the experiment should be conducted while in a mov-

Next, a comparison was made between the point groups V1
and V2 with V0 , which is the SOM, to extract the differences.
The average of the total number of point clouds and the reduction ratio was as shown in the following Table 3. By comparing the voxelized point cloud with the SOM, we reduced thedata amount by more than 95.0%. The reduction was calculated by dividing the number of voxelized points by the number of points remaining after comparing with the SOM. The
results of the difference extraction between V1 and V2 using
SOM can be seen in Fig. 15. To enhance a clear visible distinction, both V1 and V2 point clouds showcase a 10.6-second
duration instead of the previous 0.1-second representation.
Table 3: Results after differences extraction
remaining points reduced ratiodiff extraction V1
241
98.6%
diff extraction V2
755
95.6%
From here, both point clouds(voxel) V1 and V2 are further
compared and the remaining point clouds are extracted. The
average of the total data amount and the reduction ratio was82International Workshop on Informatics ( IWIN2023 )believe that in future work, SOM can be further extended tothe dynamic information layer of a dynamic map. In our proposed method, we only considered two vehicles data, but infuture work, this scope can be expanded to identify the rightand wrong information which is an error data by judging thecollected data from more than two vehicles. Moreover, wecollected the test data from the University campus. As a future work, We plan to expand this model by collecting the test
data from the real road scenario with  n number of vehicles.

REFERENCES(a) V1 result(b) V2 result

Figure 15: difference extraction result
as shown in Table 4 below. The number of point clouds that
could be eliminated by performing the difference extraction
between V1 and V2 exceeded 84.0%. The reduction ratio was
calculated by dividing the number of points remaining after
the comparison of V1 and V2 by the total number of points in
V1 and V2 after the comparison with the SOM.
Table 4: Overlapped points of V1 and V2
remaining points reduced ratio
V1 and V2
154
84.6%
As a result, the total number of original point clouds, V1
and V2 , consisting of 48,000 points, were reduced to 154
points through voxelization, SOM, and comparison with theaggregated point cloud. In other words, the data size could be
reduced to about 0.3% of the original data size.
Note that voxelization and difference extraction depends
on the size and shape of the object, so it is not possible to saythat they reduce the speciﬁc data size. In this experimental
data, the object volume occupied by the SOM is small, andpoints within 2.0 cm of the object being compared were removed during difference extraction. Therefore, the data size
may tend to be smaller than the actual voxelization and difference extraction performed.
In fact, the remaining points should be 241 as a result ofTable 4, but are smaller than 241 for the aforementioned reasons.
In addition, the point cloud aggregation algorithm used inthis experiment is a simple superposition. Therefore, in a realenvironment, there is a possibility that errors may occur inthe superimposed point cloud, and the detection distance andother factors must be adjusted when performing difference
extraction.

6 Conclusion
Our proposed method not only reduces the network trafﬁccongestion, that is the communication load at the server but
also reduces the computational load on the server parallelly.
We fuse LiDAR point cloud data to not only remove the duplicate, redundant data but also increase the accuracy of thecollected data from the neighboring vehicles and piece themissing data by exchanging data between other vehicles. We[1] Autononous Driving Lab’s Editors, “What is Dynamic map?
It is useful for Autonomous Driving (2023),” https://jidounten-lab.com/y_6059 (in Japanese)[2] Ali, Shagufta, “Vehicle to Vehicle communication,”10.13140/RG.2.2.24951.88487, 2019[3] Chen, Q., Tang, S., Yang, Q., & Fu, S, “Cooper: Cooperative perception for connected autonomous vehicles
based on 3d point clouds,” Proc. in 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), (pp. 514-524). IEEE, July 2019.
[4] H. Li, H. Liu, H. Lu, B. Cheng, M. Gruteser and T.
Shimizu “Bridging the Gap Between Point Cloud Registration and Connected Vehicles,” Proc. in IEEE Open
Journal of Vehicular Technology, vol. 3, pp. 178-192,
doi: 10.1109/OJVT.2022.3165930, 2022.
[5] B. Anand, V. Barsaiyan, M. Senapati and P. Rajalakshmi, “Real Time LiDAR Point Cloud Compression
and Transmission for Intelligent Transportation System,” 2019 IEEE 89th Vehicular Technology Conference(VTC2019-Spring), Kuala Lumpur, Malaysia, pp. 1-5,
doi: 10.1109/VTCSpring.2019.8746417, 2019.
[6] P. J. Besl and N. D. McKay “Method for registrationof 3D shapes,” Proc. in Proc. Sensor Fusion IV: Control Paradigms Data Structures, International Society
for Optics and Photonics, vol. 1611, pp. 586 ‒ 606,
1992.
[7] M. Magnusson, A. Lilienthal, and T. Duckett, “Scanregistration for autonomous mining vehicles using 3DNDT,” Journal of Field Robotics,, 24(10), 803-827,
2007.
[8] Xu, Yusheng, Xiaohua Tong, and Uwe Stilla, “Voxelbased representation of 3D point clouds: Methods,
applications, and its potential use in the constructionindustry,” Automation in Construction,, 126 (2021):
103675, 2021.
[9] LAS Speciﬁcation 1.4 - R15, LAS Speciﬁcation
https://www.asprs.org/wp-content/
uploads/2019/07/LAS\_1\_4\_r15.pdf[10] “VIRTUAL SHIZUOKA,” Shizuoka Prefectural Government, Japan, https://www.pref.shizuoka.
jp/machizukuri/1049255/1052183.html (inJapanese)[11] Bentley. J. L , ”Multidimensional binary search trees
used for associative searching” , Commun. ACM, vol.
18, number. 9, pp. 509 ‒ 517, 1975

83

Session岱
Ne(Chair: Tomoya Kitani)

International Workshop on Informatics ( IWIN2023 )A Scheduling Method for LoRaWAN
Assuming a Mixture of Incompatible Devices
Tabasa Ishibashi† and Takuya Yoshihiro‡†

Graduate school of Systems Engineering, Wakayama University, Japan‡
Faculty of Systems Engineering, Wakayama University, Japan†e-mail : s246015@wakayama-u.ac.jp‡
e-mail : tac@wakayama-u.ac.jp

Abstract IoT(Internet of Things) is being used as a system to collectand utilize the vast amount of data from numbers of things.
LPWA (Low Power Wide Area) is payed attention as a communication technology for IoT and amang them, we use LoRaWAN. LoRaWAN has a problem of increasing frame collisions as number of devices in the network increase. Frame
collisions happen when several devices transmit messeages
to base station at the same time and are more likely to occur as the number of devices increases. Existing studies haveproposed a method for scheduling the transmission timing ofdevices to solve the problem. However, it is difﬁcult to transit to a new method because existing studies require all devices in the network to support the new protocol. Therefore,
in this paper, we present a scheduling method for LoRaWAN
to support a mixture of compatible and incompatible devices
by extending the MAC protocol of LoRaWAN. The mechanism estimates the time of incompatible devices transmitting
messages and schedule transmissions sent by compatible devices avoiding the time. The proposed method thus aims to
improve the scalability of the whole network.

ALOHA-based MAC protocol, the scalability of LoRaWAN
is limited [4].
Therefore, several methods have been proposed for LoRaWAN to optimize communication networks by schedulingdata frames. In these methods, all devices in the network arerequired to implement the new protocol, so that it is hard totransit from the current protocol to the new protocols.
In this paper, we propose a scheduling method for LoRaWAN to support a mixture of compatible and incompatible
devices by extending the MAC protocol of LoRaWAN. The
system ﬁrst estimates the time when incompatible devices
send their data frames based on the records of past data frame
receptions. Next, the timing and the channel of transmissionsof compatible devices are scheduled by systems and are notiﬁed to the devices with ACK frames. The proposed methodthus aims to improve the scalability of the whole network bycontrolling the transmission timings of devices to avoid collisions.
The remainder of this paper is organized as follows. In
Section 2, we give an introduction to LoRaWAN networks.
Section 3 describes the details of our scheduling scheme. Finally, Section 4 concludes the paper.

Keywords: IoT, LPWA, LoRaWAN, scheduling, communication protocol

2 LORAWAN

1 INTRODUCTION

In the LoRaWAN protocol, many end devices communicate with NS (Network Server) through GW (Gateway). GW
and end devices in a network of LoRaWAN make the star
topology of the network. The devices in a network seen arelisted as follows. End devices are placed in the network to
send data frames to GW. GW receives data frames from end
devices, then forwards them to NS as IP packets. NS manages end devices in the network according to the LoRaWAN
protocol.
In the LoRaWAN network, end devices communicate withNS through GW. The physical layer on which end devices and
GW communicate is LoRa modulation, and the physical layer
on which GW and NS communicate is Ethernet. All communications can be bi-directional, but mainly most of them areuplink transmissions sent from end devices to NS. The timing
at which end devices listen to the channel is deﬁned, at which
GW transmits downlink messages to end devices.
In the LoRaWAN protocol, the operation of end devices
is classiﬁed into three types, class-A, class-B, and class-C.
Class-A is the standard class, in which all end devices must
operate. End devices in class-A communicate with GW in-

Recently, IoT(Internet of Things) is being called as a system to collect and utilize a vast amount of data from many
devices. IoT devices require low energy consumption to run
on battery power and long-distance wireless communication
to perform small and infrequent packet exchanges. This typeof wireless communication technology is generally called as
LPWA (Low Power Wide Area), which includes several speciﬁc standards such as NB-IoT [1], SIGFOX [2], andLoRaWAN [3], etc. Among them, LoRaWAN is suitable tostudy new protocols because of its open deployment policy.
LoRaWAN is a communication protocol established by LoRa
Alliance, has an open speciﬁcation, and uses unlicensed bands.
In the current protocol, LoRaWAN uses an ALOHA-based
protocol, which does not schedule transmissions. Class-A devices send a data frame immediately without carrier sensing
when a data frame to send is ready. If there are a small numberof devices in a network, the probability of a frame collision
is low. However, if the number of devices in a network increases, the probability of frame collision increases becauseof the high density of transmissions. Due to the nature of the87International Workshop on Informatics ( IWIN2023 )teractively in which each end device’s uplink transmission isfollowed by two downlink receive windows. At each of thedownlink receive windows, GW sends an ACK frame when
conﬁrmation is required. NS is allowed to send a MAC command with the ACK frames to manage end devices. ClassA operation has the lowest power consumption among allclasses, in which end devices of class-A run on battery powerso that they sleep most of the time. Class-B is the operation that extends class-A, and end devices of class-B sleep
most of the time similarly to class-A. In addition to ClassA, Class-B operation forces end devices to receive downlink
communications at regular intervals, which allows NS to periodically control end devices, thereby the power consumption
with class-B operation is higher than class-A operation. Thus,
class-B operation seems to assume that the end devices areequipped with a solar panel at least. Class-C is the operationthat further extends class-B. End devices of class-C always
listen to the channel, thereby class-C operation requires thehighest power consumption. Thus end devices of class-C are
assumed to have a power source. In this paper, we suppose
the most basic deployment scenario of LoRaWAN, and all
end devices are assumed to implement class-A operation.

Figure 1: Scheguling ﬂow of compatible end devices overtime.
schedule computed by GW, they transmit uplink messages according to the schedule. Then, the probability of frame collision reduce because the schedule enables compatible end devices to avoid frame collisions.

3.3 Alteration of Communication Proceduresand Format
In the proposed method, the downlink messages from GW
to compatible devices are extended to support scheduling functions. Therefore, GW must distinguish compatible end devices from incompatible devices. A schedule request as the
MAC command from an end device indicates it is compatible,
whereas no schedule request from the end device indicatesthat it is incompatible with the proposed method. Sending
the MAC command to GW, the compatible end devices inform their transmission interval and request to schedule theirtransmissions.
Figure 1 shows the process in which compatible end devices inform the transmission interval to GW and obtain theirschedule in return. Note that the end devices are assumed
to have completed the join procedure before this trasnaction.
A compatible end device i transmits a data frame attachedwith the MAC command, which is newly deﬁned in the proposed method, to inform its transmission interval. When GW
receives the data frame, GW schedules the transmissions ofdevice i, and replies the transmission schedule expressed as
the offset, where offset is the value to shift the next transmission timing. The MAC command, ScheduleRequest, is newly
deﬁned, by which the compatible devices inform their transmission intervals. Figure 2 shows the frame format of ScheduleRequest command. The transmission intervals of the compatible end devices are assumed 10 minutes, 15 minutes, 20
minutes, and 30 minutes, and those four kinds of transmissionintervals are denoted using 2-bit. According to the transmission interval of the device, GW determines the transmission
schedule.
A compatible end device which completed the join procedure to the network sends data frames with ScheduleRequest.
If this is the ﬁrst time transmission for the device, it sends
the ScheduleRequest at an arbitrary time according to the LoRaWAN protocol. After GW receives the ScheduleRequest,
the transmission schedule is determined according to the method presented in Sec. 3.5. After the transmission schedule isdetermined, GW supplies the schedule for the compatible end
device using the MAC command called InformOffset.

3 SCHEDULING TRANSMISSIONS
3.1 Premise of The Method
In the proposed method, we assume that the LoRaWAN
network consists of end devices that are incompatible or compatible with the proposed method, and a single GW which canreceive uplink messages in multiple channels. In this paper,for the sake of conciseness, we assume that all end devices
and GW use the same Spreading Factor (SF) value, and thatthe transmission channels are limited to two of them, which
all LoRaWAN end devices must implement.

3.2 Overview of Sheduling Transmission
MethodThe proposed method aims at reducing the number of frame
collisions with time-slotted scheduling. Time slots are deﬁned by dividing time per a certain length. In this paper, thelength of the time slot is determined according to the timeof transmitting an uplink message, i.e., end devices send one
uplink message at one time slot. Transmissions of compatible end devices are scheduled to avoid collision with those ofother end devices in the network.
The ﬁrst step, which is shown in Sec. 3.4 in detail, is to observe incompatible end devices transmitting the uplink messages and to estimate the timing at which they will send uplinkmessages. Incompatible device is any device that is not compatible with the proposed method. With a certain amount oftime drift, incompatible end devices transmit uplink messages
periodically. Thereby, it is possible to estimate the timing at
which incompatible end devices send the uplink messages,
according to the records when GW receives the frames. The
second step, which is shown in Sec. 3.5 in detail, is to determine when and on which channel, compatible end devices
transmit their uplink messages. After being informed of the88International Workshop on Informatics ( IWIN2023 )performs the following procedure to estimate the time at which
the incompatible end devices transmit as accurate as possible.
When GW receives the uplink messages from the incompatible end devices, GW stores the reception time on the reception record table. The number of reception time records
per device is N , and so if the records exceeds N , the earliest record is deleted. If incompatible end device’s reception
records are more than k, the average interval of the reception
time from the device is computed. This is to reduce the affection of the time error.
The average interval of reception time from the device i isdeﬁned as the estimated transmission interval and expressedas Ii . With this value, we estimate the transmission timings
until T minutes away fram the current time. Given the latest
reception time being tr , GW estimates the device’s transmission schedule k times as tr + Ii , tr + 2Ii , ......, tr + kIi , where
k is the quotient of T divided by Ii .
In the proposed method, the schedule array is used to manage the above transmission schedule of all the end devices.
Let the length of a time slot be ts , then the number of slots M
in T is expressed as M = tTs . In the schedule calculation, we
assume that s0 corresponds to the latest reception time, andthat the slots are given as S = {s0 , s1 , . . . , sM −1 }. In theschedule array, each slot sj holds the list of the device IDs atwhich the device is estimated for transmission. If there is nodevice to be transmitted in slot sj , the corresponding list isnull. After estimating the transmission slots of the incompatible end device,GW uodates the transmission schedule array,by ﬁrst deleting all the past estimation and then adding the
newly estimated slots to the array. In addition, GW sends the
ACK message one or two seconds later after receiving the uplink message. If the data frame requires GW to send the ACK
frame to the end device, the slots at which GW must transmit
the downlink messages are scheduled, too.

Figure 2: Frame format of ScheduleRequest

Figure 3: Frame format of InformOffset
The InformOffset is also a newly deﬁned MAC command
to inform the offset and the frequency channel of the compatible end devices. The offset is the value to force the end
devices to change the next transmission timing. The offset, o,
is determined as follows.
o = tnext − (tr + Ii ),(1)

where tr is the last receive time from the compatible end device, tnext is the estimated next transmission time and Ii isthe transmission interval of the device i, respectively.
Figure 3 shows the frame format of InformOffset command.
In the InformOffset command, the pair of the OffsetSeconds
and the OffsetFractionalSecond ﬁelds stand for o by 3 bytes.
The OffsetSeconds ﬁeld represents the integer part of o rounded
down to the nearest whole number. The OffsetSeconds is apositive number when o is a positive number, and a negative
number when o is a negative number. The OffsetFractionalSecond ﬁeld denotes the decimal point of o in 256 steps.
On the other hand in the LoRaWAN protocol, the frequency
channels are denoted by 3 bytes. Using this representation,
the Freq ﬁeld denotes the frequency channel by 3 bytes inwhich the end device transmits the uplink messages. Whenthe compatible end devices receive the InformOffset, they set
the channel and the time of the next transmission according
to the InformOffset. The time interval until the next transmission is Ii +o and after that the intervals are o, i.e., o is effective
until the device receives another InformOffset. The devices
do not change the transmission channel until the device receives another InformOffset, too. The compatible end devices
act like incompatible end devices if the devices do not receive the InformOffset in responce for their ScheduleRequest
commands consecutively, regarding the scheduling process aswrong. Even if the compatible end devices transmit when thetime is not in the schedule, GW computes their new schedules
based on the latest receive time and replies the InformOffset.

3.5 Schedule The Transmission of The
Compatible Devices
After receiving an uplink message from a compatible end
device, GW calclates the transmission schedule of that device, and sends an ACK frame with the offset, which is thevalue indicating how much time that device must shift thenext transmission. Receiving the offset with the ACK frame,that device changes the scheduled time of the next transmission accordingly.
On calculating the transmission schedule for the end device, GW selects the best offset having the lowest cost among
all candidate offsets. We let the required transmission intervalof compatible end device i is Ii , the time duration to calculate schedule is T , the length of the slot is ts , the number ofslots in Ii is Ji = tIsi and that in T is M = tTs . The offset candidate sets Ocand = {⌈− J2i ⌉, . . . , −1, 0, 1, . . . , ⌊ J2i ⌋}represents the starting time of slots in a single transmission interval Ii from half before to halfvafter one cycle with respect
to the ﬁrst slot s0 , i.e., the current time. The schedule So is
the set of the transmission slots at which the device transmits
the uplink messages every JI after the current time, deﬁned
for each offset candidate o. Since the current time expresses

3.4 Estimate Transmission Timing of The
Incompatible Devices
To avoid collisions between transmissions of compatible
and incompatible end devices, it is necessary to estimate thetime at which the incompatible end devices transmit. Generally speaking, end devices get a large time error while they
sleep. This time error causes the end devices to shift theirtransmission time. Under the effect of the time errror, GW

89

International Workshop on Informatics ( IWIN2023 )varies for each device, this cost function takes into accountthe difference of transmission intervals among end devices.
Finally, the offset o adopted as the transmission schedule forthe device is deﬁned as follows.
oad = argmin C(o).

and slateFigure 4: The example for searlyttAfter determining the optimal offset oad , GW reﬂects thescheduling result on the schedule array. First, the device ID
i are all removed from the array and write the transmission
schedule on the array, where the schedule oad is
Soad = {soad +kJi }, k = 1, 2, 3, ... Second, it is necessary totake into account the timings at which GW sends the downlink messages, which are one and two seconds later. Thereby,for each st ∈ Soad added into the schedule array, we add thedevice ID i to slots st1 and st2 , which are corresponding to
Rx1 and Rx2 respectively. The schedule is computed in thismanner and the offset o for the device is determined. Thisvalue of o is sent to device i in the InformOffset command
attached to the ACK frame.

slot s0 , it is deﬁned as So = {so+kJi }, where k = 1, 2, 3, . . .
and 0 ≤ o + kJi ≤ M .
The cost for the offset candidate o ∈ Ocand has two indicators, one evaluates the risk of collisions with other transmissions for each transmission, and the other evaluates thedegree of time difference the device is forced to change its
transmission time. The candidate offset with the lowest costis adopted for the transmission schedule for the device. The
cost C(o) for the offset candidate, o, is deﬁned as follows.
C(o) = αCcol (o) + (1 − α)Cint (o),(6)

o∈Ocand(2)

where Ccol (o) is the collision cost and Cint (o) is the intervalchange cost, and α is the constant given in advance, whichtakes balance between those two costs.
The collision cost Ccol (·) is aimed at preventing to allocate
slots whose nearby slots have already been used by anotherdevice. In the LPWA network, most end devices may transmit
uplink messages out of the allocated slots because of their low
accuracy clock. Thereby, it is desirable that the slots scheduled for the devices are far from each other to decrease theprobability of frame collisions. Terefore, Ccol (·) is deﬁned
to take value if the allocated slots are closer to other occustpied slots. The collision cost Ccolof the slot st is deﬁned asfollows.
()
T − kIi
11
st
Ccol =+,T− st + 1
st − searly+ 1 slatett(3)
where searlyistheclosestoccupiedslot
earlier
than
s
(=
t
t
so+kJi ), slate
is that later than st . Figure 4 shows the ext. There are four occupied slots, andample searly
and
slate
t
t
early
late
st
and st are the closest ones earlier and later than st .
By summing the collision costs of the slots corresponding
to an offset, the collision cost for the offset candidate o isdeﬁned as follows.
∑
st
Ccol
.
(4)
Ccol (o) =4 CONCLUSION
In this paper, we proposed a scheduling method for the LoRaWAN network which supports a mixture of incompatible
and compatible devices with the proposed method. In thecurrent protocol, since the ALOHA-style protocol does notcontrol end devices to avoid collision, the devices transmit
the uplink messages at arbitrary timings.
However, frame collisions are more likely to occur as thenumber of the end devices increased. The proposed method
is aimed at improving the scalability of the LoRaWAN network by using the schedule array with time slots and by supplying the transmission schedule to compatible devices with
the MAC command. First, the network server estimates thetimings at which each incompatible device transmits and GW
receives the uplink messages from the incompatible device inthe future. Next, GW determines the transmission scheduleof each compatible device when GW receives the uplink messages from the compatible device. When GW computes thetransmission schedule for each compatible device, it considers the ﬂuctuation of the transmission interval and the risk offrame collisions. The proposed method thus aims to improve
the scalability of the whole network by controlling the transmission timings of devices to avoid collisions.

st ∈SoAcknowledgement

On the other hand, Cint (·) is the interval change cost aiming at avoiding large data interval alteration for each device
in case of resheduling. It is desirable that the transmissioninterval of end devices does not change largely because IoT
devices are required to transmissions regularly. Since larger
offset means larger change in interval, the interval change cost
Cint (·) should also be larger. The interval change cost is deﬁned as follws.
oCint (o) = | |.
(5)Ji
The cost is deﬁned as the ratio of the offset value o to thenumber of the slots in the transmission interval Ji . Since JiThis work is partially supported by the Cooperative Research Project Program of the Research Institute of ElectricalCommunication,Tohoku University.

REFERENCES[1] 3GPPHP, https://www.3gpp.org/news-events/
3gpp-news/nb-iot-complete,Last visited February 2023.
[2] sigfoxHP, https://www.sigfox.com, Last visited February 2023.

90

International Workshop on Informatics ( IWIN2023 )[3] LoRaAllianceHP, https://lora-alliance.org, Last visited
February 2023.
[4] Jetmir Haxhibeqiri, Eli De Poorter, Ingrid Moerman,
Jeroen Hoebeke: A survey of LoRaWAN for IoT: From
Technology to Application, Sensors 2018, 2018.

91

International Workshop on Informatics ( IWIN2023 )Comparison on function and performance between MQTT and DDS for IoT DEP
Masaki Mitsuuchi *, Kazuhiro Kosaka *, Tetsuya Yokotani**, and Koichi Ishibashi***
Graduate School of Engineering, Kanazawa Institute of Technology, Japan{c6301158, c6201098}@st.kanazawa-it.ac.jp**
College of Engineering, Kanazawa Institute of Technology, Japan{yokotani, k_ishibashi}@neptune.kanazawa-it.ac.jp
Abstract - Internet of Things (IoT) services are currently
being introduced in various fields. This has led to a marked
increase in the number of connected IoT devices. To
accommodate them efficiently, a platform that lightens
processing is considered necessary. One such platform is the
IoT Data Exchange Platform (IoT DEP) standardized by
ISO/IEC JTC1/SC41, which is a platform architecture that
actively utilizes Pub/Sub type communication. This paper
compares Message Queuing Telemetry Transport (MQTT)
and Data Distribution Service (DDS), which are
representative of Pub/Sub type communication, for therealization of IoT DEP. Comparison items are shown
qualitatively in terms of protocol positioning (scope of
specification), popularity, components, performance, quality
control, security, etc. The results of the performance
evaluation when discards were given showed that MQTT
showed superior values in terms of both traffic volume andtraffic stability compared to DDS. In terms of quality
control, DDS is superior to MQTT because, in addition toguaranteed delivery, it is possible to set the timing oftransmission and the deadline for transmission. Similarly,
for security, DDS, which is configured with common access
control settings, is considered superior to MQTT. Based onthese results, MQTT is applied to End Point connections,prioritizing dissemination and lightweight. For interworking
points, it is appropriate to apply DDS, which does not
require a broker, and to operate as a distributed broker for
MQTT.

end device sends data. In addition, the operational burden ofmanaging communication routes from a huge number of end
devices also increases. From these perspectives, it can besaid that the conventional Internet is not necessarily suitablefor IoT communication. In addition, if minute data sent from
IoT devices is transmitted by conventional methods, thecontrol data becomes large and inefficient. To accommodate
these efficiently, a platform that is lightweight in processing
is considered necessary. To support this, topic-based
publication/subscription (aka pub/sub model) is considered
the most widely used communication method, and several
implementations are available for field applications [2]. One
such implementation is the IoT Data Exchange Platform(IoT DEP), which is being standardized at ISO/IEC
JTC1/SC41 under the initiative of this university. This study
compares Message Queuing Telemetry Transport (MQTT)
and Data Distribution Service (DDS), which are
representative of Pub/Sub type communication, aiming torealize IoT DEP.

Keywords: IoT DEP, IoT Platform, message typecommunication, MQTT, DDS

Fig.1 Typical architecture model

1. Introduction

2. Overview ofrelated areas

In recent years, Internet of Things (IoT) services havebeen introduced in various fields. This has led to asignificant increase in the number of connected IoT devices,and with the increase in IoT devices comes the need for theability to synchronize state and data among multiple IoT
devices [1]. For example, in a typical IoT architecture, it iscommonly considered to apply an Internet to the Network
layer. A typical IoT architecture model is shown in Fig. 1. In
conventional Internets, TCP/IP is used for network access,and HTTP or HTTPS is generally applied to the application
layer protocol. In this case, TCP/IP, which is a connectionbased communication, requires the execution of 3-way
handler education, which complicates the sequence andthereby significantly increases the required bandwidth. In
addition,theconventional
Internetrequiresacommunication sequence that includes data and address
resolution by the Domain Name System (DNS) each time an

IoT

communicationandIn this section, we will organize the situation surrounding
IoT communications and provide an overview of theconditions and requirements for IoT communication
applications and the relevance of the IoT reference modeland platforms.

2.1. IoT Reference Model and Platforms
Various IoT reference models have been discussed invarious countries, including RAMI4.0 (German: Reference
Architecture Model for Industrie 4.0), IIRA (U.S.: Industrial
Internet Reference Architecture), and IVI (Japan: Industrial
Value chain Initiative). These overviews are publicized in[4] and elsewhere. In this context, the IoT reference model
was specified by JEITA in 2019 for a wide range of
purposes, including the dissemination of IoT systems,

93International Workshop on Informatics ( IWIN2023 )standardization of IoT architectures, and dissemination of
IoT platforms.
Cite [3]. The model consists of a Value Chain axis (Xaxis) indicating the purpose and business flow of the entire
IoT service, an Interoperability axis (Y-axis) indicating theconnection of information. The model consists of theInteroperability axis (Y-axis), which indicates integration,and the Zoon axis (Z-axis), which indicates the spread ofservices. Among these, the Y-axis and Z-axis, which are the
key axes for a bird's-eye view of the IoT platform, are used
to classify the IoT platform as shown in Fig.2.

IoT applications require high-speed processing of tiny data
generated by a huge number of devices. This means that alightweight protocol must be used to transfer those data. In
addition, IoT applications may require real-time rather thanreliable transfers. As with lightweight protocol headers, a
simplified communication sequence is desirable. Next is the
QoS control function for each IoT application. Since each
IoT application may require different QoS, it is necessary toprovide a flexible QoS control function. When multiple
applications are multiplexed, the current Internet provides
these functions to each IP flow. Finally, reliabletransmission independent of TCP connections. The current
Internet can reliably transmit data even when some of theapplications multiplexed onto the connection require realtime transmission rather than reliable transmission. One of
the factors for this is the establishment of TCP connections.
However, when TCP connections are established, large
delays occur when real-time transmission is required. By
making it selectable for each application, highly reliabletransmission independent of TCP connections can berealized.

3. Overview of IoT DEP
This section provides an overview of the IoT DEP and
describes the relationship between this standardization
initiative led by the University and the verification ofprinciple that is underway in parallel. We also identify
issues for IoT DEP implementation and select topic-based
Pub/Sub type communication to be addressed in this report.

Fig.2 Positioning of IoT Platforms
Fig.2 shows the part of [3] that is necessary to discuss IoT
platforms. As shown in [3], IoT platforms exist for Site(within a company), Enterprise (between sites), and Interenterprise (between organizations), and can be divided into
Communication (network: communication protocol),Information (information processing: data model), andService (service: data distribution/utilization). Information
processing: data model), and Service (provisioning service:data distribution/utilization). In this report, Platform 1,
Platform 2, and Platform 3 will be referred to as Platform 1,
Platform 2, and Platform 3, respectively, with Platform 1
being the main focus.

3.1.

Standardization on IoT-DEP

The following is an overview of the IoT data exchange
platform (IoT DEP) that the author and others are promoting
as an example of Platform 1. For more information on IoT
DEP, please refer to [5]-[7], etc.
The IoT, in which all things are connected to the Internet,
is a technology that is currently attracting a lot of attention,
but as IoT devices become more widespread, the current
Internet communication standards will become inconvenient.
However, as IoT devices become more widespread, thecurrent communication standards for the Internet will
become inconvenient. In addition, an analysis of IoT use
cases has been conducted, and the results indicate the needfor a platform that is lightweight in its communication
processing and abstracts the physical network. Based on the
results of this study, the IoT DEP is the specification of anew platform that can be commonly applied to applications.
IoT DEP is a platform architecture that actively utilizes
Pub/Sub communication, an IoT-specific communication
method that enables more efficient data transfer thanconventional communication methods using IP addresses
and DNS servers. To achieve lightweight communication,
Information Centric Network (ICN) technology, which is
not bound to IP addresses, is applied to end devices andservers, while dedicated paths connect multiple Nodal
Points in the IoT DEP network. The IoT DEP network
consists of dedicated paths connecting multiple Nodal Points.
IoT DEP functions are implemented in Nodal Points. This
enables the IoT DEP to operate on the existing Internet andto be compatible with existing protocols such as Hypertext

2.2. Requirements for IoT communication
The current Internet mechanism incurs overhead in datatransfer. When the frequency of data transfer is high, itcauses significant problems for the communication network.
Lightweight communication methods need to be introduced
to solve this problem. Many IoT applications require public
networks, and these applications need to coexist withtraditional applications such as e-mail, video distribution,and IP telephony. However, IoT applications can benefit
from lightweight communication architectures and protocols.
Therefore, approaches and solutions for traditional and IoTapplications should be specified. Many IoT applications
require the efficient transfer of a huge number of small
packets with small data blocks over the network.
Furthermore, the required communication quality also
depends on the type of application: application layer
protocols such as HTTP require a TCP 3-way handshakeand IP-based routing. Therefore, these protocols are notsuitable for the widespread use of IoT.
There are requirements for the protocols to be used fordata communication. One is lightweight protocol overhead:94

International Workshop on Informatics ( IWIN2023 )Transfer Protocol (HTTP) and File Transfer Protocol (FTP),
and to realize the transfer of a huge amount of data.
To briefly summarize the advantages of IoT DEP, accessto a DNS server is not required, and because ICN
technology is used, less header information is required,resulting in smaller communication data volume. As a result,
low latency can be achieved. It is also possible to operate on
the existing Internet.
IoT DEP is being standardized by ISO/IEC JTC/SC41 as
ISO/IEC30161 series. Table.1 summarizes the relationshipbetween the promotion of this standardization and theverification of principle that is being conducted in parallelwith it in Fig.3.

Part
Part1

Part2

Table.2 Classification of Nodal Point-to-Nodal Point
Control
Dynamic
Static(iv)
Approach1
Approach4(v)
DirectApproach2
Approach5
Indirect
Approach3
Approach6
In Table.2, (iv) is a method that transfers the information
request (equivalent to "Subscription" in the case of MQTT),and (v) is a method that transfers the information itself(equivalent to "Publication" in the case of MQTT). In thecase of response-driven, the method in which theinformation itself is transferred directly into the network iscalled Direct, while the method in which only the ID of theinformation is transferred and the information itself is
offloaded by a network other than the IoT DEP (e.g.,existing Internet) to reduce the amount of traffic is called
Indirect. In Table 2, Approach 4 and 5 are shown in [9],Approach 6 in [10], and Approach 2 in [11].

Table.1 Scope of ISO/IEC30161 series
Contents
・Requirement
・Architecture
・Functional block definition
・Functional definition of Nodal Point
・Coordinated control between Nodal Points

3.2. Future challenges for IoT DEP
As described above, the IoT DEP so far has applied
MQTT, a typical Pub/Sub type communication between
ICNs, to achieve higher speed and lower latency comparedto the existing Internets. However, there are still some issues
to be solved in the coordination and control between Nodal
points. For example, the connection between Nodal Pointsof the IoT DEP in [9] is realized by connecting distributed
brokers of MQTT with a virtual ring. However, this method
increases the delay time on the virtual ring, and the problem
between Nodal Points needs to be solved to implement IoT
DEP. Therefore, this report presents a proposal for practicalapplication of IoT DEP by qualitatively comparing MQTT,
which has already been applied as a connection method
between Nodal Points in IoT DEP, and DDS, acommunication protocol that does not require a Broker.

Fig.3 International standardization and proof of principleof IoT DEP
In Table.1, these standards provide requirements andarchitecture. Detailed protocols are positioned as
implementation issues. However, we are currently verifying
the principles within the framework specified in thesestandards.
The following three points are the key points of the proofof-principle. In the verification of principle, priority is given
to the application of MQTT to the ICN portion of Fig.3.
(i) Service specification and interface verification:
Information from various sensors, etc., is analyzed andtransferred to the payload of MQTT's Publish message. The
format of information from sensors, collection period, etc.
are parameters.
(ii) Connections between ICN and DEP networks:
Connections between MQTT and the edge of the DEP
network (implementation of gateway functionality).
(iii) Interface verification between Nodal Points: The
connection between Nodal Points is between the IoT DEP
middleware, which applies a simpler scheme than IP-based
routing in the Internet. The schemes are summarized in [8]and can be categorized as shown in Table.2.

4. Summary of MQTT and DDSThis section provides an overview of Pub/Sub typecommunication, and outlines MQTT and DDS.

4.1 Publish/Subscribe type communication
Publish/Subscribe type communication refers tocommunication for sending and receiving messages with thesender as the Publisher and the receiver as the Subscriber.
Conventionally, to send and receive messages, it isnecessary to prepare a Broker to mediate messages between
the Publisher and Subscriber, and MQTT uses this method.
Fig.4 and Fig.5 show the configuration diagrams of MQTT
and DDS.
Fig.6 shows a configuration diagram of thecommunication method using HTTP. In the conventional
communication method, it is necessary to convert a URL to
IP, go through a DNS server, and then connect to a dataserver. In MQTT, however, a Broker is placed between the
Publisher and the Subscriber, and connection control
between the Publisher and the Subscriber can be simplified
through the Broker. DDS does not require a Broker, butinstead allows data servers to communicate directly with95

International Workshop on Informatics ( IWIN2023 )each other. This eliminates the need to access a DNS server,which is necessary with conventional communication
methods, thereby preventing delays and congestion. Thismeans that Pub/Sub communication is lighter in weight thanconventional communication methods.

"+" symbols. The following is an example of a topic to be
specified.
Exact match：KIT/GSE/LAB
Left-hand match：KIT/GSE/＃
Partial match：KIT/＋/Floor
The system is designed to receive only those items
specified in this way.
MQTT implements three guaranteed delivery functions.
QoS0 (non-guaranteed delivery), QoS1 (guaranteed
delivery), and QoS2 (non-overlapping guaranteed delivery)
can be set, respectively. Fig.7 shows the communication
sequence for each QoS function.

Fig.4 MQTT communication

Fig.5 DDS communication

Fig.7 Communication sequence of MQTT QoS function
QoS0 has no delivery guarantee and only sends a Publish
message once, which is suitable for frequent transmission of
sensor data over time. QoS1 provides more reliablecommunication than QoS0 by sending a delivery
confirmation message called PUBACK to the Publish
message and guaranteeing delivery once. QoS2 provides
more reliable communication than QoS1 and is the mostreliable in MQTT. Compared to QoS0 and QoS1, QoS2 isused when the number of communications is higher thanthat of QoS0 and QoS1, but when data is to be sent reliably.
MQTT is used when you want to send data reliably. In thisway, MQTT provides delivery assurance with three QoS
functions.

Fig.6 Configuration diagram for HTTP communication

4.2.

MQTT

The Message Queuing Telemetry Transport (MQTT)
protocol [12], developed by IBM and now an OASIS
standard, is a lightweight message-oriented protocol [13]and a topic-based Pub/Sub type communication. MQTT can
operate over TCP/IP and requires a Broker to exchange data.
In practice, the Broker is a server to which clients canpublish/subscribe topics, and message traffic goes through it.
Clients can also authenticate to the Broker by accessing witha user name and password. SSL/TLS [14] encrypts the
security of messages [15].
MQTT communicates by specifying the unique name of
the information to be acquired. This unique name is ahierarchical name address called a Topic, and MQTT
communicates by specifying a unique name called a Topic.
Topics have a hierarchical structure, which can be createdby separating each Topic with "/" when describing it. For
example, KIT/GSE/LAB can be used to create a hierarchicalstructure as shown in the figure. In this case, use the "#" and

4.3.

DDS

Data Distribution Service (DDS) is data-centric
publication and subscription middleware for highly dynamic
distributed systems [19], standardized by OMG (Object
Management Group) [16]. Data is published to a DDS
domain, and subscribers can subscribe to data from thatdomain without knowing the source or structure of theinformation. DDS offers a wide range of QoS parameterssuch as durability, lifetime, presentation, reliability, anddelivery time [17]. According to the OMG website, DDS is96

International Workshop on Informatics ( IWIN2023 )one of many protocols used in industrial sectors such as
railway networks, air traffic control, smart energy, medical
services, military and aerospace, and industrial automation[18].
Like MQTT, DDS is a topic-based Pub/Sub typecommunication, and has in common that the quality control
function called QoS can be used to set the guaranteed
delivery level and that it is implemented by middleware [21].
One difference is that MQTT operates over TCP/IP, while
DDS can operate over UDP and TCP. Another difference indesign is that MQTT requires a Broker, whereas DDS does
not require a Broker and allows direct communication
between Publishers and Subscribers. DDS can also provide
real-time, many-to-many managed connections [20].

specified topic. Although some MQTT protocols, such as
Mosquitto, implement access control by user/password or
RSA authentication for the Subscriber and Publisher of agiven topic, the protocol itself does not specify access
control, and is considered less secure than DDS [23].

6. Performance evaluation
In this section, performance evaluation is conductedunder the following conditions in order to evaluate the"performance" item shown in Table.4 on actual equipment.
The evaluation results are also discussed and explained.

6.1.

Configuration

Next, performance evaluation is performed under thefollowing conditions: MQTT is based on TCP, DDS isbased on UDP and TCP, and QoS is set to QoS0 (nonguaranteed delivery) for MQTT, Besteffort (non-guaranteed
delivery) for TCP-based DDS, and Reliable for UDP-based
DDS. The Reliable setting refers to the state in which the
ACK return function that is originally performed over TCP
is added. In addition, experiments will be conducted using
two Raspberry Pi, with the same Raspberry Pi used as the
Broker and Subscriber for MQTT. In order to clearly show
the difference in performance between the two, thePublisher side will be given 1% and 10% discards forverification, and the data will be observed using Wireshark.
The protocol stacks of MQTT and DDS are shown in Fig. 8,
and the configuration diagram for performance evaluation isshown in Fig. 9.

5. Comparison of MQTT and DDS forconnection between Nodal Points
In this section, we will qualitatively compare MQTT and
DDS and describe the comparison results with references
and other information. Based on the comparison results,
advantages and disadvantages in MQTT and DDS will beidentified.

5.1 Comparative verification
Comparison of MQTT and DDS. In Table.3, we show thecomparison results in MQTT and DDS based on theliterature in [24] and the results of the performance
evaluation conducted in this study. The number of +indicates a superior characteristic.
Table.3 Comparison MQTT and DDS
MQTT
DDS
Layer
Application
Application
Layer
Layer, Data
Model
Popularity＋＋＋＋
Architecture
Pub/Sub
Pub/Sub/Broker
Performance
＋＋
＋
QoS
＋＋
＋＋
Security＋＋＋

Fig.8 Protocol StackIn Table.3, with respect to penetration, MQTT is one of
the most used messaging protocols by developers, whereas
DDS is not supported on existing IoT Platform platforms[21]. A clear difference between DDS and MQTT in termsof components is that MQTT has a Broker and DDS does
not; MQTT ensures security by requiring a Broker, whereas
DDS is configured with common access control settings,
The absence of a Broker does not undermine security [22].
In terms of quality control, DDS is superior to MQTT in thatall data requires a QoS contract, and in addition toguaranteed delivery, the timing of transmission and thedeadline for transmission can also be configured. One of thefactors that make DDS superior in terms of security is that it
is equipped with access control plug-ins, such as Mosquitto,
which implements access control by user/password or RSA
authentication for the Subscriber and Publisher of a

Fig.9 Configuration diagram

6.2. Verification Results
6.2.1. Measurement results in Wireshark
As a representative example of Wireshark results measuring
message transmission, Figs. 10 through 12 show the resultsof MQTT and DDS measurements at 1% waste. In theseresults, RTPS refers to DDS. These data were taken from
Wireshark, which was used for the observation. The trafficfor MQTT is the smallest for message transmission, while
the traffic for DDS is almost the same, but the traffic for
ACK is higher for UDP than for TCP, resulting in adifference in the traffic for message transmission. The
difference in the traffic for message transmission is due tothe higher ACK traffic for UDP compared to TCP.

97

International Workshop on Informatics ( IWIN2023 )Fig.10 Packet capture with 1% DDS UDP loss

Fig.11 Packet capture with 1% DDS TCP loss

Fig.12 Packet capture with 1% MQTT loss

6.2.2. Experimental results at 1% disposal
The performance results at 1% waste are shown in Fig.13.
MQTT showed almost the same amount of traffic for eachmessage sent, and the shape of the graph was stable. The
graph shape was also stable. Finally, the DDS traffic for
TCP was smaller than the DDS Reliable traffic for sending
messages, but the graph shape was disturbed by theregularity of the traffic for sharing information with eachother.

Fig.13 Result in loss 1%

98

International Workshop on Informatics ( IWIN2023 )6.2.3. Experimental results at 10% waste
Next, the performance results at 10% disposal are shownin Fig.14. The graphs for both DDS and MQTT are different
in terms of their characteristics at disposal, since they bothshow similar graph shape disorder.

Fig.15

Proposal for IoT DEP

8. Summary
In this paper, we took up MQTT and DDS, which are
representative examples of pub/sub communication, andconducted qualitative comparisons and performance
evaluations using actual equipment. We also proposed an
application method using both MQTT and DDS for the
standardization of IoT DEP. In the future, we plan toimplement and evaluate the proposed application in Fig. 13
in a limited environment using Raspberry Pi, and work
toward practical application.

Fig.14 Result in loss 10%

6.2.4. Summary of experimental results
The results of the average traffic per second are shown in
Table.4. It can be seen that in all conditions, the traffic at
10% discard is higher than that at 1% discard, and for DDS,
TCP has higher traffic than UDP. The comparison of MQTT
and DDS shows that MQTT has lower traffic than DDS
UDP/TCP. The following results may be due to the fact that
TCP has a higher load than UDP and UDP has a smaller
header size than TCP.

Traffic（Byte）
1%
10%

Table.4 Average traffic per second
DDS UDPDDS TCP
361.667
395.007374.167401.447

REFERENCES[1] A. Shaout, and B. Crispin, Using the MQTT Protocol
in Real Time for Synchronizing IoT Device State, The
International Arab Journal of Information Technology,
Vol. 15, No. 3A (2018)[2] W. Y. Liang, Y. Yuan, and H. J. Lin, A Performance
Study on the Throughput and Latency of Zenoh,
MQTT, Kafka, and DDS, Distributed, Parallel, andCluster Computing (cs.DC), arXiv:2303.09419[cs.DC][3] https://home.jeita.or.jp/cgi-bin/page/detail.cgi?n=1171(2022-6-19 Access)[4] M. Weyrich, and C. Ebert, Reference Architectures for
the Internet of Things, IEEE Software, Vol. 33, No. 1,
pp. 112 – 116 (2016)[5] T. Yokotani, and K. Kawai, Concepts and requirementsof IoT networks using IoT Data Exchange Platform
toward International standards, 2019 IEEE Conferenceon Standards for Communications and Networking(CSCN), pp. 1-6(2019)[6] T. Yokotani, A proposal of IoT DEP as a Platform for
Efficient IoT Service Deployment on Wide Area
Networks, IEEE 2019 International Japan – Africa
conference on Electronics, Communications andComputers (IEEE JAC-ECC 2019), IEICE Invited
session -1 (2019)[7] T. Yokotani, and K. Kawai, Survey on standardization
activities of IoT and proposal of the IoT data exchange
platform, IEICE 2020 International Conference on
Emerging Technologies for Communications (ICETC
2020), IB3-3 (2020)[8] T. Yokotani, S. Yamamoto, S. Ohno, K. Sasabayashi,
and Koichi Ishibashi, Survey and comparison ofInterworking point routing mechanisms for IoT
services in wide area ICNs, IEICE 2020 International

MQTT
154.293
162.593

7. Proposed configuration for IoT DEP
In previous IoT DEPs, MQTT has been applied toconnect EPs, and Nodal Points have been realized byconnecting distributed MQTT brokers in a virtual ring.
However, there is concern that this method increases thedelay time on the virtual ring. Therefore, based on theseresults, a proposal for application to IoT DEP is shown in
Fig.15. For endpoint connections, MQTT is applied,prioritizing dissemination and lightweighting. It is also
appropriate to apply DDS, which does not require a broker,
between Nodal Points to act as a distributed broker for
MQTT. Optimization of communication paths can beachieved.

99

International Workshop on Informatics ( IWIN2023 )ConferenceonEmerging
Technologies
forCommunications (ICETC 2020), D2-4 (2020)[9] S. Ohno, K. Terada, T. Yokotani, and K. Ishibashi,
Distributed MQTT broker architecture using ring
topology and its prototype, IEICE Communication
Express, Volume 10 Issue 8, pp.582-586 (2021)[10] S. Ohno, K. Terada, T. Yokotani, and K. Ishibashi, The
control mechanism of distributed MQTT brokers forlarge volume data transfer and its prototype system,
IEICE Communication Express, Volume 11 Issue 3,
pp.160-164 (2022)[11] Y. Noda, S. Ohno, K. Ishibashi, and T. Yokotani, A
new routing mechanism based on layer 2 control in
MQTT networks with multiple brokers, IEICE
Communication Express, Volume 11 Issue 6, pp. 307312 (2022)[12] Sebastian Raff. The MQTT Community. Available
online:https://github.com/mqtt/mqtt.github.io/wiki(accessed on 10 June 2019)[13] D. Silva, L. I. Carvalho, J. Soares, and R. C. Sofia, A
Performance Analysis of Internet of Things
Networking Protocols: Evaluating MQTT, CoAP, OPC
UA, Appl. Sci. (2021)[14] B. H. Çorak, F. Y. Okay, M. Güzel, Ş. Murt and S.
Ozdemir,
Comparative
Analysisof
IoT
Communication
Protocols,
2018
International
Symposiumon
Networks,
ComputersandCommunications (ISNCC), Rome, Italy, pp. 1-6, doi:
10.1109/ISNCC.2018.8530963. (2018)[15] S. Lee, H. Kim, D. K. Hong and H. Ju, Correlation
analysis of MQTT loss and delay according to QoSlevel, 2013 IEEE International Conference on
Information Networking (ICOIN), pp. 714-717 (2013)[16] https://portals.omg.org/dds/[17] http://download.prismtech.com/docs/Vortex/html/ospl/
DDSTutorial/qos.html[18] http://portals.omg.org/dds/who-is-using-dds-2/[19] S. Profanter, A. Tekat, K. Dorofeev, M. Rickert and A.
Knoll, OPC UA versus ROS, DDS, and MQTT:
Performance Evaluation of Industry 4.0 Protocols,
2019 IEEE International Conference on Industrial
Technology (ICIT), Melbourne, VIC, Australia, pp.
955-962, doi: 10.1109/ICIT.2019.8755050. (2019)[20] S. Seleznev, and V. Yakovlev, Industrial Application
Architecture IoT and protocols AMQP, MQTT, JMS,
REST, CoAP, XMPP, DDS, International Journal ofOpen Information Technologies ISSN: 2307-8162 vol.
7, no.5 (2019)[21] E. Al-Masri, K. Raj Kalyanam, J. Ratts, J. Kim, S.
Singh, T. Vo, and C. Yan, Investigating Messaging
Protocols for the Internet of Things (IoT), in IEEE
Access, vol. 8, pp. 94880-94911 (2020)[22] eProsima Fast DDS Documentation,
https://fast-dds.docs.eprosima.com/en/latest/index.html[23] G. Aures, and C. Lübben, DDS vs. MQTT vs. VSL
for IoT, Seminar IITM SS 19, Network Architectures
and Services (2019)100

International Workshop on Informatics ( IWIN2023 )Study of an implementation method of point-to-multipoint communication for IoT
data exchange to reduce traffic on an IoT network
Takahiro Shiohara*, Koichi Ishibashi** , and Tetsuya Yokotani***

Information and Computer Engineering, Graduate School of Engineering, Kanazawa Institute ofTechnology, Japan
c6201940@st.kanazawa-it.ac.jp**
College of Engineering, Kanazawa Institute of Technology, Japan{k_ishibashi, yokotani}@neptune.kanazawa-it.ac.jp

Abstract - Due to the growing interest in the Internet ofThings (IoT) in recent years and toward the further
development of the IoT, platforms for efficiently exchanging
IoT data generated by IoT devices such as sensors andactuators among IoT users are being discussed in various
areas. In this platform, the interworking among nodal points,
which IoT devices and IoT users connect to, is essential tosupport wide-area and large-scale IoT systems. In
exchanging and sharing IoT data, requirements such asreliability and low latency depend on the IoT service. In
addition, it is desirable to share IoT data generated by IoT
devices among multiple IoT devices. In address to the above
requirements, this paper proposes an implementation methodof point-to-multipoint communication to efficiently
exchange IoT data among users according to the quality ofservice required by the IoT data. The proposed method ischaracterized by the coordination with multicast control at
the application level in order to accommodate various
quality of services with IoT data while utilizing Data
Distribution Service (DDS), which is one of the candidate
communication protocols among nodal points. By
comparison with the method of using only DDS, we
confirmed the proposed method reduces the traffic volumes
among nodal points.

In addition, platforms for efficiently sharing data generated
by IoT devices (IoT data) among users (IoT users) are also
discussed in various places to cope with the growing scale of
IoT systems and the increasing number of IoT devices. For
example, there is research on providing power-saving IoT
services by applying mobile edge computing technology tounmanned aerial vehicles equipped with IoT devices[4].
We are considering a wide-area and large-scale IoT systemfor efficient exchanging IoT data among users. Our targeted
IoT system consists of IoT access networks and an IoT core
network. The IoT access network is an access network toaccommodate IoT devices and IoT users, and is connected to
the IoT core network via a gateway device called a nodal
point. In the IoT access network, Message Queuing
Telemetry Transport (MQTT)[5] is deployed as acommunication protocol for transmitting/receiving IoT data
to/from IoT devices and IoT users. MQTT is a candidate
communication protocol for IoT systems. The IoT core
network is a network for development of a wide-are IoT
system, and multiple MQTT broker, which are implemented
on nodal points, are interconnected with each other. Theinterworking among nodal points is deployed by the Data
Distribution Service (DDS)[6].
In addition, the targeted IoT system aims to provide IoT
services with various service requirements in terms of
reliability and low latency. If the reliable mode provided by
the DDS is only utilized to achieve interworking among
nodal points, overheads of control packet by the DDS
increases. On the other hand, interworking based on the besteffort mode by the DDS cannot provide the desired QoSlevel in the exchange of IoT data with reliability and/or low
latency requirements. Therefore, we propose an
implementation method which is characterized by thecoordination with multicast control at the application level inorder to accommodate various quality of services with IoT
data while utilizing the DDS. We also show the effectivenessof the proposed method.
The remainder of the paper is organized as follows:
Section 2 explains the targeted IoT system in the paper. In
section 3, related works related to the paper are presented.
And, section 4 proposes the implementation method ofpoint-to-multipoint communication at the application level,
section 5 describes the results of evaluation. Finally, section
6 concludes the work.

Keywords: Internet of Things, DDS, point-to-multipoint
communication.

1

INTRODUCTION

In recent years, there has been growing interest in theInternet of Things (IoT) technology, in which all things areconnected to a network. With the expansion anddevelopment of IoT-based services and businesses, thenumber of connected IoT devices is increasing year by year,
and IoT devices are expected to exceed 29 billion by 2024[1].
Due to the increasing number of IoT devices such as sensors
and actuators connected to IoT systems, the interoperabilityof the network devices, called nodal points that
accommodate the IoT devices and IoT user’s assets isincreasingly being discussed. Therefore, there are manyefforts towards developing IoT systems and proposals toefficiently exchange IoT data under wide-are and large-scale
IoT networks. For example, there are proposals formonitoring urban transportation systems[2] and research on
efficient data collection systems from sensors[3].

101International Workshop on Informatics ( IWIN2023 )2

TARGETED IOT SYSTEM

applications to achieve efficient data communication
between DDS functions and IoT data.
ServerIn this section, we explain the targeted IoT system and its
challenges. Figure 1 shows the targeted IoT system. IoT
devices such as sensors and actuators, and IoT users such asservers are connected to a nodal point via MQTT protocol.
Here, the nodal point acts as the MQTT broker. Interworking
among multiple MQTT brokers, i.e., nodal points, arecoordinated by DDS. That is, nodal points also act as DDS
nodes, and implements a software function in order tointerwork between the MQTT client and the DDS function.
IoT data generated by an IoT device is published to a nodal
point, which is an MQTT broker. The published IoT data isshared among multiple MQTT brokers because it is utilizedby multiple IoT users. In other words, IoT data is forwarded
by multicast manner from the nodal point, which the IoT
device connects to, to multiple nodal points. IoT data
generated by the IoT device require various QoS
requirements, such as best effort, reliable, and low latency.
In addition to IoT data generated by the IoT devices, thenodal point generates novel IoT data based on edge
processing of IoT data published by IoT devices.
Such an IoT system has the following issues. Firstly, theamount of traffic forwarded among nodal points increases as
the number of connected IoT devices increases. Secondly,
due to interference of IoT data with best effort service on thecoexistence of IoT data with various QoS requirements,
service level for IoT data with reliable and/or low latency
requirements degrade. Lastly, since IoT data published from
IoT devices is expected to be subscribed in multiple IoT
users, the nodal point that receives IoT data from IoT
devices is required to forward IoT data with multiple nodal
points, and efficient data sharing in point-to-multipoint
communication among nodal points must also be considered.
And, connections among the nodal points are configured via
communication paths with various communication
characteristics. That is, it is necessary to consider the
existence of nodal points connected via a communication
path with large packet loss rates or large latency. In addition,
the nodal points accommodate with multiple IoT devices. So,
it is necessary to consider that the nodal point receives IoT
data with reliable and/or low latency requirements and IoT
data with best effort service from multiple IoT devices at thesame time.
When DDS is applied in communication among the nodal
points, communication services among nodal points depends
on DDS functionality. That is, reliability support on DDS isensured by acknowledgement control from the destination
nodes. And, in point-to-multipoint communication on DDS,transmitting of a packet is completed after confirmation of
acknowledgements from all destination nodes. Therefore, if
there is a nodal point connected via a communication pathwith large packet loss rate or large latency, point-tomultipoint communication is influenced from the path withlarge packet loss or large latency. It is necessary to consider
a method for ensuring communication methods that does notdepend on the DDS functionality and bad communication
paths. Based on the above, we propose a multicast control
method that considers coordination with DDS by upper-level

ServerSubscribe Server
Subscribe
Subscribe

MQTT Broker+
DDS Node

DDS Network
DDS Node
MQTT Broker + DDS NodePublishPublishIoT device

Figure 1: Example of IoT systems.

3

RELATED WORKS

This section describes the research works related to efforts
using MQTT and DDS, and multicast control at theapplication level.
In [7], several MQTT protocols including the open source“Mosquitto” are evaluated in terms of resource consumption
and latency, and the results are shown. In [8], the authors
proposed a communication scheme for IoT devices and built
a platform for evaluating the system. Evaluations wereconducted on a data-by-data basis, showing that the systemis efficient in distributing data over a network. In [9],performance evaluation and comparison of communication
protocols for IoT such as MQTT and DDS are conducted,
and its evaluation shows that MQTT significantly reduces
round trip time (RTT) for servers and DDS has highperformance in protocol implementation. In [10], datatransfer using DDS is implemented, showing that it provides
low latency and high throughput, and is an effective
communication protocol for communication systems such as
those in smart cities.
Next, multicast control at the application level is proposed
to improve resource consumption, such as throughput, in thetarget network to address the issues introduced byconventional communication protocols. For example,
latency recovery and fault recovery characteristics have beenachieved by drastically reducing control traffic in the
bandwidth of stagnation[11]. In [12], a protocol for
multicasting at the low-bandwidth application layer is
proposed to reduce overhead. Simulation evaluations of the
protocols implemented in applications show that the
proposed protocols can significantly reduce control traffic.
In [12], an algorithm is proposed and evaluated to improve
end-to-end throughput at the application level. The
evaluation results show that the proposed protocol cansignificantly improve the throughput.

4

PROPOSALIn this section, we describe a proposed method among
DDS nodes to efficiently exchange IoT data in the targeted102

International Workshop on Informatics ( IWIN2023 )IoT system. Firstly, an explanation of DDS is given.
Secondly, proposed method is explained.

are received on an application at application level andforwarded to other nodal points via DDS protocol. The
application also generates novel IoT data according to anedge process of received IoT data. In the proposed method,
IoT data are forwarded under DDS best-effort service.
Therefore, IoT data with reliable and/or low latency
requirements are provided a required QoS at applicationlevel. That is, the application for forwarding IoT data, called
forwarding application, support to control for reliable pointto-multipoint communication. The forwarding application
firstly decides whether IoT data require to reliable and/or
low latency services according to requirements with IoT data.
If IoT data requires reliable and/or low latency service, IoT
data is transmitted by way of the control process. The
control process in the forwarding application achieves
reliable data transfer instead of DDS reliability service.
That is, when it transmits/forwards IoT data with reliable
and/or low latency requirements, firstly transmitted packets
are copied to a buffer for retransmission control. And an
entry, that manages reliability to destinated nodes, is created.
Lastly, the packets are transmitted by way of the DDS
function for data transmission with best effort manner. A
node that receives packets responds with theacknowledgement packets if received packets require
acknowledgement for reliable communication. The
transmitting node updates the entry for managing
retransmissions upon receipt of an acknowledgment packet.

4.1 DDS
The protocol stack on DDS is shown in Figure 2. DDS is
implemented on UDP/IP, and supports various Quality ofServices (QoS) for applications that exchange IoT data via
DDS. A software compliant with the DDS provides
application interfaces for transmission and receive of data,such as a “DataWriter” and a “DataReader”. The DataWriteris the application interface and provides a function totransmit data to other nodes. The DataReader is theapplication interface and provides a function to receive datafrom other nodes. In addition, the DataWriter transmits
HEARTBEAT packets to support reliable data exchange.
That is, The DataWriter transmits a HEARTBEAT to thedestination as a packet to confirm the reachability of datawith reliable requirements. The DataReader that receives the
HEARTBEAT responds with the sequence numbers of thepackets it received before receiving the HEARTBEAT. The
sender confirms that the packet has reached the destination
by receiving the response to the HEARTBEAT (Figure 3).

Application
Durability

Ownership

Content
Subscription

Minimum Profile
Real-Time

DDS

Start of process for
IoT data transmission

Publish/Subscribe
Best effort type

UDP/IP
Figure 2: Protocol stack of DDS.
PublisherApp

Copy IoT data to buffer

Buffer:?
Empty

Subscriber
DataWriter

QoS of IoT
data:
Reliable and/or low latency

DataReader

AppNo empty
Discard a previous
requested data

write(1)
No

1

mes

A

Create an entry for management ofmulticast and acknowledgement

check

X

message(1, A)
cache
(A,1)
Ack(1)

No

mes

check

1

A

X

Transmission request to DDS
by best effort manner

write(2)

End
No

mes

check

1

A

X

2

B

X

X

:cathe：Waiting(B,2)

Figure 4: Flowchart on nodal point for forwarding IoT data.

cache(A,1)：HEARTBEAT

Figure 3: Message sequence of DDS Reliability mode.

4.2 Proposed Method
Figure 4 shows the flowchart on the nodal point of the
proposed method when forwarding IoT data from the IoT
device to other nodal points. IoT data from the IoT device

Figure 5: Evaluation systems.

103

International Workshop on Informatics ( IWIN2023 )5

EVALUATION

Figure 9. In this case, since data queue size forretransmission is set to 1, due to the packet loss ofacknowledgement from Node 2, the data queue is filled withdata to be multicast to Node 2 and Node 3. And the queueingof the next data is blocked. That is, in communications that
require an acknowledgement to provide reliability, the delay
or loss of response from the destination nodes connected via
the unstable communication paths affects the transmission ofsubsequent packets. This affects QoS support for data withlow latency requirements when data with low latency
requirements conflicts with data with reliable requirements.
1500000

Traffic volume(bytes)

To verify the effectiveness of the proposed method, weconstructed an experimental network with multiple DDS
nodes which is located on nodal points (Figure 5) andevaluated the traffic volume between DDS nodes, that istraffic volume on the IoT core network. In the experimental
network, we evaluated the traffic volume when transferring
IoT data via the proposed method and via DDS reliability
service, called “using only DDS”. We also compared thetraffic volume according to the generation probability of IoT
data with reliable and/or low latency requirements. In
evaluation, Node#1 transmits IoT data to other DDS nodes at
1 packet/second in 1000 seconds. And the data transmitted
and received on Node#1 are counted. Ratio of IoT data withreliable and/or low latency requirements varied from 10% to
0.5 %. In addition, the flow of messages in point-tomultipoint communication with QoS that provides reliability
service in DDS was verified.
Figure 6 shows the traffic volume when IoT data isforwarded by using the proposed method and by DDS
reliability service, called “using only DDS”. In Figure 6, thenumber of receiving DDS nodes varies from 1 to 5, and ratioof number of IoT data with reliable and/or low latency
requirements is 10%. When “Using only DDS” is applied,the overall traffic volume increased because the traffic ofacknowledgment control for confirmation to guarantee thereliability is required for forwarding all IoT data. In the
proposed method, the traffic volume is suppressed because
acknowledgement is performed at the application level onlyfor IoT data with reliable and/or low latency requirements.
Figure 7 shows the traffic volume depending on the ratioof IoT data with reliable and/or low latency requirements.
The traffic volume of the proposed method increases theincrease in acknowledgement messages, as the ratio of IoT
data with reliable and/or low latency requirements in thesending node decreases. Because in the proposal method,
IoT with reliable and/or low latency requirements datarequire only acknowledgement message. On the other hand,when “using only DDS” is applied, the traffic volume at thesending node remains constant regardless of the percentage
of IoT data with reliable and/or low latency requirements
because reliability service in DDS always checks fortransmission acknowledgement. Therefore, using on DDS
traffic volumes do not change regardless ratio of IoT datawith reliable and/or low latency.
In order to confirm the results of Figure 6 and Figure 7,
we explain behavior of message sequence we clarify tobehavior of message sequences on point-to-multipoint
communication under DDS reliability mode in this part.
Figure 8 shows the captured message sequences of point-tomultipoint communication from Node#l (IP address:
192.168.1.1) to Node#2 (IP address: 192.168.1.1) andNode#3 (IP address: 192.168.1.1). It is noted that queue size
is set 1 of queuing delay because we assume real-time
application. In
Figure 8, due to the loss ofacknowledgements from Node#2, the message requesting an
acknowledgement
from
Node#1,
described
as“HEARTBEAT” in Figure 8, continues to be retransmitted.
In addition, the results in Figure 8, in DDS reliable node in

1300000
1100000

900000700000500000300000100000

1

2

3

4

5

Number of receiving DDS nodes
Using only DDS

Proposal method

Figure 6: Comparison of traffic on IoT core network.

Traffic volume(bytes)

1400000
1300000
1200000
700000680000660000640000620000600000
0.5

1

2

5

10

Ratio of IoT data with reliable and/or low latency …
Proposal method

Figure 7: Traffic volume according to ratio of IoT data withreliability and low latency.

Figure 8: Packet sequence on DDS reliability mode.

104

International Workshop on Informatics ( IWIN2023 )IP address:192.168.1.1
DataWriterApp

IP address：192.168.1.2

IP address:192.168.1.3

DataReader

DataReader

App

App

write()
No

Data

check

1

A

X

DATA(1, A)

Block

DATA (1, A)

Ack(1)

check(1)write()

No

Data

checkAck(1)

No

Data

check

1

A

✓

1

A

X

No Data

check

1

✓

A

check(1)：HEARTBEAT

Figure 9: Acknowledgment retransmission sequence inpoint-to-multipoint communication.

6

CONCLUSION

In this paper, we propose the implementation method for a
multicast control coordination scheme in conjunction with
DDS as an efficient method of transferring IoT data in IoT
systems. We also compared with the data transfer capabilityof the proposed method and the data transfer using only the
DDS function. As results of the evaluation, we confirmedthat the proposed method is effective in reducing the traffic
volume compared to the communication method using only
the DDS function. In addition, we verified the operation in
point-to-multipoint in DDS reliability mode. We confirmedthat in DDS reliability mode, retransmission for
acknowledgement affects transmission of subsequent
packets.
In an actual IoT system, various communication
characteristics such as packet loss and latency on thecommunication paths between nodal points are assumed.
Therefore, it is necessary to further evaluate the proposed
method by considering the communication characteristics ofeach path in the target IoT system, such as packet loss rateand latency.

MQTT in M2M and IoT Systems: A Survey,” IEEE
Access, vol. 8, pp.201071-201086, November 2020.
[6] P. Bellavista, A.Corradi, L.Foschini, and A.Pernafini, "Data Distribution Service (DDS): A performancecomparison of OpenSplice and RTI implementations,"
2013
IEEE
symposiumoncomputers
andcommunications, IEEE, Vol.2, pp.377-383, July 2013.
[7] S.Profanter, A.Tekat, K.Dorofeev, M.Rickert, A.Knoll,"OPC UA versus ROS, DDS, and MQTT: performance
evaluation of industry 4.0 protocols," IEEE International
Conference on Industrial Technology,
pp.955-962,
February 2019.
[8] B.Melvin, K.Erkin, P.Marc-oliver, and C.Georg. "Opensource mqtt evaluation," 2021 IEEE 18th Annual
Consumer Communications & Networking Conference.
IEEE, pp.1-4, January 2021.
[9] L.Xiangtao, H.Ning, Z.Peng and Z.Yu,"The method ofInternet of Things access and network communication
based on MQTT." Computer Communications Vol.153,
pp.169-176, March 2020.
[10] A.Tanushree, N.Payam, M.R.Barzegarn, and V.Luigi.
"Multi-level time-sensitive networking (TSN) using thedata distribution services (DDS) for synchronized threephase measurement data transfer," IEEE Access Vol. 7,
pp.131407- 131417, September 2019.
[11] L.Jie, Y.Zhou, and H.Chen. "Age of information for
multicast transmission with fixed and random deadlines
in IoT systems," IEEE Internet of Things Journal, vol.7,
no.9, pp.8178-8191, March 2020.
[12] B.Suman, B.Bhattacharjee, and C.Kommareddy.
"Scalable application layer multicast," Proceedings of the2002 conference on Applications, technologies,
architectures,
andprotocolsforcomputercommunications, August 2002.
[13] Z.Ying, B.Li,B, and J.Guo. "Multicast with network
coding in application-layer overlay networks," IEEE
Journal on Selected Areas in Communications, vol.22,
no.1, pp.107-120, January 2004.

REFERENCES[1] Ministry of Internal Affairs and Communications,”2022
Summary of the State-of-the-Art Report on Information
andTelecommunications(inJapanese),“https://www.soumu.go.jp/johotsusintokei/whitepaper/ja/
r04/html/nf3r1000.html,Accessed, May 2023.
[2] P, Arman Syah, and H, Leslie Hendric Spits Warnars,"Intelligent traffic monitoring system (ITMS) for smart
city based on IoT monitoring," 2018 Indonesian
Association for Pattern Recognition International
Conference. IEEE, pp.161-165, September 2018.
[3] A.P.Plageras, K.E.Stergiou, C.Wang, and H.Gupta,"Efficient IoT-based sensor BIG Data collection–
processing and analysis in smart buildings," Future
Generation Computer Systems Vol.82, pp.349-357, May
2018.
[4] M.N.Hossein, M.Bagaa, and T.Taleb. "UAV-based IoT
platform: A crowd surveillance use case," IEEE
Communications Magazine, vol.55, no.2, pp.128-134,
November 2017.
[5] Biswajeeban Mishra and Attila Kertesz, “The Use of105

International Workshop on Informatics ( IWIN2023 )107

International Workshop on Informatics ( IWIN2023 )108

International Workshop on Informatics ( IWIN2023 )109

International Workshop on Informatics ( IWIN2023 )110

International Workshop on Informatics ( IWIN2023 )111

International Workshop on Informatics ( IWIN2023 )112

International Workshop on Informatics ( IWIN2023 )113

International Workshop on Informatics ( IWIN2023 )114

International Workshop on Informatics ( IWIN2023 )115

International Workshop on Informatics ( IWIN2023 )116

International Workshop on Informatics ( IWIN2023 )117

International Workshop on Informatics ( IWIN2023 )118

Session 4:Se
sis(Chair: Takuya Yoshihiro)

International Workshop on Informatics ( IWIN2023 )121

International Workshop on Informatics ( IWIN2023 )122International Workshop on Informatics ( IWIN2023 )123

International Workshop on Informatics ( IWIN2023 )124

International Workshop on Informatics ( IWIN2023 )125

International Workshop on Informatics ( IWIN2023 )126

International Workshop on Informatics ( IWIN2023 )127

International Workshop on Informatics ( IWIN2023 )128

International Workshop on Informatics ( IWIN2023 )129

International Workshop on Informatics ( IWIN2023 )131

International Workshop on Informatics ( IWIN2023 )132

International Workshop on Informatics ( IWIN2023 )133

International Workshop on Informatics ( IWIN2023 )134

International Workshop on Informatics ( IWIN2023 )Evaluation of Human Recognition Method by 2D-LiDARs
Haruki Mochizuki† Takuya Watanabe* Yuya Sawano*
Yoshiaki Terashima**, Ryozo Kiyohara††Kanagawa Institute of Technplogy, Japan
Graduate School of Kanagawa Institute of Technology, Japan**
Soka University, Japan*

Abstract -Automated vehicle technologies which includes

millimeter-wave radar, and technologies which has been
established for autonomous vehicles for human recognition.
However, it is not realistic to have enough such expensive
sensors in a robot used in a campus because of its cost.
Moreover, since these technologies for autonomous driving
are aimed at collision prevention with humans, it isimportant to reliably recognize humans rather thanerroneously recognizing non-humans as humans.
Furthermore, since these recognition technologies are
executed while driving at a constant speed and there arevarious moving objects in the surroundings, it is important
to detect obstacles and predict their movements.
On the other hand, On the other hand, these technologies
for security have the same purpose of preventing collisions,but the accuracy of distinguishing between people andobjects is important. It is also possible to take a method ofsensing while repeatedly stopping the vehicle andapproaching it if there is a possibility of a person to further
improve the recognition accuracy.
In many cases, robots move using SLAM (Simultaneous
Localization and Mapping) technology with 3D LiDAR. It
runs while estimating its own position using SLAM. In
SLAM, which processes 3D point cloud information, thecomputational complexity is very large. Therefore, thecomputational complexity of processing other thanautonomous driving should be minimized.
Therefore, we think that it should be clarified whether one
or more 2DLiDARs can identify people with the sameaccuracy as 3DLiDAR. The number of point clouds that canbe acquired by 3DLiDAR and 2DLiDAR is clearly different
as shown in Fig.1 and Fig.2.

human detection technologies are developed by many
manufacturers. Moreover, these technologies are adopted tomany automated robots or carts. On the other hand, once or
twice a year, an elderly person wanders into a university
campus. Then human detection technologies Therefore, it isimportant to find human with robot carts by utilizing humanrecognition technology. In many cases, stationary humanrecognition is realized by 3D LiDAR (3 Dimensional Light
Detection and Ranging) device. However it requires many
computer resources. In this paper, we propose a new humanrecognition method by 2D-LiDARs and evaluate ourmethod..

Keywords: LiDAR, Human Detection, Autonomous Cart,
Machine Learning. PointCloud.

1

INTRODUCTION

In developed countries, the birthrate is declining and the
population is aging. As a result, labor shortage has become asocial problem. On the other hand, expectations for the
security industry are rising and demand is increasing.
However, in the current security industry, there arephysical problems, etc., and there is a shortage of personnel,
and the introduction of robots that can replace humans isessential.
Since robots can operate 24 hours a day, they can be usedefficiently on campuses such as universities as delivery
robots and guidance robots during the day, cleaning robots
in the early morning, and security robots at night.
Since people may accidentally enter the university campus,
nighttime security is important. Elderly people in particular
are not malicious and need to be protected by security
guards. Such cases occur once a year or less, and personnel
expenses should not be spent on security for such cases.
Therefore, it is important for robots to detect suspicious
persons. Technology for robots to judge a person andtechnology to judge that the person is a suspicious person
are important.
If a suspicious person is someone other than a university
faculty member or student, the suspicious person can be
judged by whether or not the robot behaves like showing an
ID when the robot approaches. Therefore, a method for
robots to detect the presence of people is important.
Expensive robots are equipped with many sensors such as
LiDAR (Light Detection and Ranging), cameras, and

2

PROPOSED METHODFirst, the recognition system processes the difference withthe pre-trained background in SLAM and recognizes theunusual. Next, we judge whether the extracted data arehuman or not. For that purpose, we need to learn humans inadvance. Humans learnimg from the front as shown in Fig. 3,
and Fig. 4, but recognition system do not know which angle
the data can be acquired. Therefore, it is necessary toconfirm whether it is possible to determine whether a person
is human even with data from various angles. In this paper,
we evaluated data from various angles.

135International Workshop on Informatics ( IWIN2023 )REFERENCES[1] Velodyne LiDAR https://velodynelidar.com/products/puck/（2023/6/3）

Fig.1 Example of pointcloud from 3D-LiDAR

Fig.2 Example pf pointcloud from 2D-LiDARs

Fig.3 Learning of humanFig.4 pointcloud data of Fig. 3136

International Workshop on Informatics ( IWIN2023 )Estimating the best time to view cherry blossoms using Natural Language Processing
tasks
Ryoga Sato*, Yusuke Takamori**, Kenji Terada***, Masaki Endo***, Tsuyoshi Tanaka****, Shigeyoshi
Ohno***, and Hiroshi Ishikawa******

Electronic Information Course, Polytechnic University, Japanb20311@uitec.ac.jp**
Division of Advanced Science and Technology, Japan Advanced Institute of Science and Technology,Japany-takamori@jaist.ac.jp***
Information and Communication Engineering, Polytechnic University, Japan{k-terada, endou, ohno}@uitec.ac.jp****
Digital Transformation of Skill, Polytechnic University, Japant-tanaka@uitec.ac.jp*****
Graduate School of Systems Design, Tokyo Metropolitan University, Japan
ishikawa-hiroshi@tmu.ac.jp
Abstract – With the rapidly expanding use of social
networking services (SNSs), information is exchanged inreal time. Taking advantage of the immediacy of thisinformation, we have conducted research to extract seasonal
events such as the best time to view cherry blossoms. Earlier
studies have proposed methods using co-occurring words
post information to increase the number of posts targeted for
estimation of the best time to view cherry blossoms. Thosestudies have shown improvement in the accuracy of
estimating the best time to view cherry blossoms. However,
earlier methods of co-occurring word extraction also treat
posts including negative words such as "not," "don't," and"nothing" as estimating the best time to view cherry
blossoms, which might result in the estimation of a different
time from that announced by the Japan Meteorological
Agency. In this study, we focus on the inclusion of posts
that do not indicate the blooming of cherry blossoms andpropose a method to improve the accuracy of estimating thebest time to view cherry blossoms using text classification
based on natural language processing in addition to theconventional method of using co-occurring words toincrease the number of data. Using this method, theeffectiveness of natural language processing was verified
and compared to that of conventional methods for
estimating the best time to view cherry blossoms.

Many people use X [1], is a leading social networking
platform, for sharing tourist information, posting and browse
information about events, hobbies, etc. Users can also add
location information to their posted messages, which called"posts." Such posts with location information, called"geotagged posts"[2], are used to share current events andlocations. Because the information obtained from geotagged
posts reflects the actual real-world situation, they are
expected to play an important role as a tool for tourists toobtain local tourism information in real time. Therefore, we
use information related to geo-tagged posts on X to estimate
the best time to view cherry blossoms. Endo et al. [3]propose a method for estimating the best time to view cherry
blossoms using simple moving averages. This methodcollects geotagged posts that include specific keywords andthen estimates the best time to view seasonal organisms byobtaining a simple moving average of the number of posts
per day. Takahashi et al. [4] propose a method using
weighted moving averages to improve the estimation
accuracy of the method described by Endo et al. Thismethod can improve the results of estimation of the best
time to view cherry blossoms in each area while maintaining
lower costs than methods using simple moving averages.
Takamori et al. [5] investigated the frequency of words usedwith the word "cherry blossom" from the content of posts,
then defined the words for which skewness and kurtosis
coincide with the word "cherry blossom" as cherry blossom
co-occurrence words. They proposed a method of using
these co-occurrence words to increase the volume of post
data for estimating the best time to view cherry blossoms.
This method provides more accurate estimation of the times
to view cherry blossoms than conventional estimation of thetime to view cherry blossoms using only the word "cherry
blossom". However, when co-occurrence terms to increase
the number of data are used, even posts that do not indicate
the blooming of cherry blossoms are treated as targets for
estimating the best time to see cherry blossoms. Therefore,
even posts that have nothing to do with the blooming ofcherry blossoms are treated as data for estimating the best

Keywords: Cherry blossoms; NLP; SNS; text mining;
tourism

1

INTRODUCTION

With the spread of the information society, people areaccessing information related to a daily basis and using theinternet more and more. Furthermore, with the
popularization of SNSs, an increasing number of tourists areusing SNSs to gather tourist information and refer to tourist
guidebooks and reviews of tourist attractions. Then, that
tourist information posted by users is shared among SNS
users. Tourists often benefit by this shared information.

137

International Workshop on Informatics ( IWIN2023 )time to see cherry blossoms. As described herein, we
propose a method for estimating the best time to view cherry
blossoms by applying text classification using natural
language processing to information posted on X. Natural
language processing is a technology or method that allows
computers to understand and process sentences in thenatural language spoken by humans. One type of natural
language processing technology is Bidirectional Encoder
Representations from Transformers (BERT) [6], a natural
language processing model developed by Google Research[7] that can understand context. Compared to conventional
natural language processing models such as statisticallanguage models such as n-gram language models andvector space models such as LSA, BERT can understand not
only the meanings of words but also their context. Therefore,
BERT can analyze natural language with higher accuracy,
and can classify and extract information. This paper isorganized as follows: Section 2 describes research related tothis paper; Section 3 presents an explanation of the proposed
method; Section 4 presents the experimentally obtained
results and their evaluation; Section 5 explains the findings
obtained from this study.

2

accuracy of estimating the best time to see cherry blossoms.
We also chose natural language processing as the method
for classifying posts. In this paper, we propose a method for
estimating the timing of cherry blossom viewing by using
natural language processing to classify text posted on X.
This section describes the proposed methodology. First, we
explain the main flow of the proposal. For this method,experiments were conducted in Tokyo and Kyoto based on
descriptions in a report by Takamori et al. First, we selectwords that co-occur with cherry blossoms from posts which
include the word "cherry blossom" in their text, which wereused in an earlier study. Next, the selected co-occurringwords are used to collect posts which contain one of thosewords in the text. Next, we use the selected co-occurringwords to collect posts that contain one of those words in the
body of the text. After the collected posts are used to finetune BERT, then text classification is done using the BERT
model created for the collected posts. Finally, the posts
collected and the posts with text classification are used toinfer the best time to view cherry blossoms during theperiod from March 1 to April 30 in 2021 and 2022.

3.1 Collecting
selection

RELATED RESEARCH

Sato et al. [8] propose a method to estimate the best time toview cherry blossoms for a certain period in the future using
time series prediction for geo-tagged posts.
Meira et al. [9] combine machine learning and natural
language processing methods to predict whether a user's
review is positive or negative. They combine supervised andunsupervised methods to develop a model to infer whether aparticular tourist attraction is liked or disliked.
Mathias et al. [10] propose a new method of personalized
tour recommendation for museum visits by optimizing
visitor preferences and by automatically extracting theimportance of artworks from museum information.
Following the proposed tour clearly has been shown to
improve visitor satisfaction.
As described above, several papers have described
estimation of cherry blossom viewing times using SNS andextraction of tourist information by combining natural
language processing. In this paper, we demonstrate theeffectiveness of estimating the best time to view cherry
blossoms by combining natural language processing and themoving average method.

3

posts

for

co-occurrence

We collect posts that include the kanji, katakana, andhiragana of the word "cherry blossom" in the text. The
period extended from February 2015, when we started
collecting posts, to May 2022. We collected posts for thetwo areas to estimate the best times to view cherry blossoms.

3.2

Morphological analysis

Morphological analysis of the posts we collected in 3.1
was performed using MeCab [11] to break down the post
body into words. MeCab, an open-source tool formorphological analysis of Japanese, breaks down sentences
into words and parts of speech. This study applied thefollowing preprocessing steps for morphological analysis.
First, URLs and symbols were eliminated as a cleaning
process to remove noise in the text. Next, the words werenormalizedtofacilitate
morphologicalanalysis.
Additionally, we converted full-width alphanumericcharacters to half-width alphanumeric characters, half-widthkana to full-width kana, and all alphabetic characters tolowercase. All numerals were replaced with "0". After suchpreprocessing, the post text was decomposed word-by-wordusing MeCab. Furthermore, using MeCab, a library formorphological analysis, and IPAdic, a dictionary formorphological analysis, we extracted words whichcorresponded to the three parts of speech "noun," "adjectival noun," and "verb" which could be adopted as
collocations. Then we counted their frequency of occurrence.
An adjectival noun, which is one part of speech in the large
category by UniDic [12], is the part of speech which
represents the stem part of an adjectival verb.

PROBREM AND METHOD

Since tourists make plans according to seasonal events,
such as cherry blossom viewing, and tourism workers
organize their events accordingly, there is a need for tools toestimate seasonal events. Therefore, a X-based method for
estimating the best time to view cherry blossoms is one way
to solve this problem. On the other hand, when we looked at
the content of posts used to estimate the best time to see thecherry blossoms, we found a scattering of posts unrelated tothe blooming of cherry trees, such as the names of places
and foods, such as "Sakurazaka" and "Sakura mochi.
Therefore, we thought that removing posts that had nothing
to do with the blooming of cherry trees would improve the138International Workshop on Informatics ( IWIN2023 )3.3

cherry blossom viewing by eliminating posts unrelated tothe blooming of cherry trees through text classification by
BERT.
For text classification by BERT, fine-tuning was
performed using a pre-training model from Tohoku
University [13]. For this method, we collected posts inTokyo that contained either kanji, katakana, hiragana word
"sakura" or one of the co-occurring words among the post
information accumulated to date. During the period of 13days from March 23 through April 4, 2015, we collected
9704 posts used as training data for natural language
processing. This is true because the difference of 6 days
between the bloom date and the full bloom date was takenon the back side as well because the bloom date in Tokyo in2015 was March 23 and because the full bloom date wasMarch 29. The following preprocessing was performed for
creating the training data used for fine-tuning. First, line
feed codes in the text data were removed and alphabetic
characters were converted to lowercase. Next, URLs were
removed to eliminate noise in the text. Emoji characters
were removed using demoji [14]. Half-width and full-width
symbols were then removed and the Japanese text was
normalized with neologdn[15]. Also, as a process fornumbers, numbers separated by commas were replaced with
0. All numbers were replaced with 0.
As a criterion for fine-tuning, we classified posts as posts
indicating that cherry blossoms are in bloom (LABEL_1)
and posts that are unrelated to cherry blossoms (LABEL_0).
Specific post classifications are shown in Table 1.

Co-occurrence selection

We only performed co-occurrence judgments for the top
1% of words we collected in section 3.2 in terms offrequency. The method used for determining co-occurrence
was to calculate the skewness kurtosis of the frequency
distribution of each word and to identify those words which
met certain criteria as co-occurring words. Skewness
kurtosis is a statistic that expresses how much the
distribution is skewed from a normal distribution and thedegree to which it is sharp. It was judged to be an
appropriate indicator for identifying the co-occurring words
which have the same peak as the keyword in the annualchange of posts. The criteria for judgment are equations (1)and (2) shown below. The formulas are expressed as Ss,
where S and K respectively denote the skewness kurtosis of
the frequency distribution of each word, and where Ss is theskewness obtained from posts that include "cherry blossom"in the text.
(𝑆𝑠 − 1) ≤ 𝑆 ≤ (𝑆𝑠 + 1)

(1)

𝐾>2

(2)

Skewness kurtosis was calculated by extracting posts inwhich words were included in the text from January 1, 2018
to December 31, 2018. Skewness 𝑆𝑘𝑒 kurtosis 𝐾𝑢𝑟 is
obtained from equations (3) and (4) below, where 𝑛 is thenumber of days in the period and 𝑥𝑖 (i:1, 2, …, n) is the value
for each day in the period. From the time period used forthis study, n is 365. Also, 𝑥̅ denotes the mean value for the
specified period; 𝑠 stands for the sample standard deviation.
𝑛𝑥 −𝑥̅ 3

𝑆𝑘𝑒 = (𝑛−1)(𝑛−2) ∑𝑛𝑖=1 ( 𝑖𝑛(𝑛+1)𝑠

𝐾𝑢𝑟 = (𝑛−1)(𝑛−2)(𝑛−3) ∑𝑛𝑖=1)

3.5

We collected posts including the word "cherry blossom" inkanji, katakana, and hiragana in the text and any of the cooccurring words in the text for the period of February 1,
2015 − February 28, 2022 for the two areas. As a preprocessing step for text classification by BERT, the samecleaning process that was performed for the training data in
3.4 was performed for post data. Then, the BERT model
tuned in 3.4 was used to perform text classification using therespective post data.

(3)(𝑥𝑖 −𝑥̅ )4𝑠4−

3(𝑛−1)2(𝑛−2)(𝑛−3)(4)

For this study, we set the period to include all months during
January−December because it was necessary to check thetrend of posts showing the same trend as the cherry
blossoms throughout the year. In addition, the posts werecollected on a server managed by our manuscript team, butthere were periods during which collection was stopped
because of power outages, etc. The calculated skewness
kurtosis values would be skewed if such periods were
included in the calculation. Therefore, to include as little aspossible of the period during which we were not able tocollect data, we adopted the period of January 1, 2018 −December 31, 2018 as the period used to calculate theskewness kurtosis, which satisfies these two conditions.

3.4

Text Classification using BERT

3.6

Estimating the best time to view

The target posts for estimating the best time to view
cherry blossoms were posts containing "cherry blossom" inkanji, katakana, and hiragana, posts containing any of theTable 1: Examples of labeling criteria for post contents
Label
Post contents
お花見シーズンsoon come
LABEL_1
今週末にはお花見散歩出来そうですね
今日は花粉日和っぽいいかにも大学に来たら改修工事真っ最中
LABEL_0
桜餅食べてる
桜坂に来てみた

Fine-Tuning of BERTIn the conventional method, if a post includes a word to be
searched for, it is treated as a target for estimating the best
time to view cherry blossoms, even if the post does notindicate the blooming of cherry trees. Examples of posts that
do not indicate cherry blossoming include names of places,
stores, and people. The purpose of this method is to increase
the number of data using co-occurring words, and in
addition, to improve the accuracy of estimating the timing of139

International Workshop on Informatics ( IWIN2023 )co-occurring words, and the results of text classification
using BERT for posts of these two types. The period for
estimating the best time to view the cherry blossoms wasfrom March 1 through April 30 in 2021 and 2022.
Estimation of the viewing period was done using the same
simple moving average method as used for the method
described earlier. Since the number of posts tends toincrease on Saturdays and Sundays, a moving average is
obtained. An n-day simple moving average is obtained using
equation (5), where 𝑥 represents the number of posts per day.
1

𝐴𝑣𝑔 𝑛 = ∑𝑛−1
𝑖=0 𝑥(𝑘 − 𝑖)𝑛Table 2: List of Co-occurring Words in Tokyo
ソメイ
cherrysakurauenopark
さく
blossom
ヨシ
ミッド
ヒルズ
七
五反田
年
タウン

(5)

Periods satisfying equations (6) and (7) below
simultaneously for three or more consecutive days wereinferred as estimating the best time to view cherry blossoms.
The period with more posts than the one-year average was
identified using equation (6). The period with a sharp
increase in the number of posts, as derived using equation(7), is inferred as the best time to view cherry blossoms. In
equations (6) and (7), xi represents the number of posts on
day i; Avg n represents the moving average over n days.
𝑥𝑖 > Avg 365(6)

Avg 10 < Avg 20

(7)

義

初

咲い

咲き

国際

大学

大崎

始め

川沿い

恩賜

日和

日野

最後

本髪

来年

毎年

江東

皆

花

花びら

花園

見物

見頃

調布

開花

靖国

Table 3: List of Co-occurring Words in Kyoto
cherry割咲き宇治平野blossom
年

散っ

淀川

神社

蹴上

開花

疏水

祇園

Next, Figures 1 and 2 present results obtained fromestimation of the best time to view cherry blossoms forTokyo and Kyoto based on post data from March 1, 2021
through April 30, 2021. Figures 3 to 4 present results
obtained for estimating the best time to view for Tokyo andKyoto based on post data of March 1, 2022 through April 30,
2022. Arrows in the figure indicate the period from the dateof bloom to the date of full bloom as announced by theJapan Meteorological Agency. They were used as thecorrect period for evaluating estimation results. In eachfigure, the number of posts with co-occurring words in the
text and the number of posts with text classification by
BERT are shown as line graphs, with the estimated time
period and times of observations shown as a fill color.
However, since the objective is to see changes in the numberof posts, the vertical axis values of posts for theconventional and proposed methods are different. Figure 4
portrays the results obtained for Kyoto, where the proposed
method estimates the best time to view cherry blossoms
about two weeks longer than the correct period. Presumably,
this result was obtained because the correct prediction
period was from the date of bloom to the date of full bloom,
and because it takes about 10 days to 2 weeks from the dateof full bloom to the end of the period when the cherry
blossoms are at their best after they have fallen. Presumably,
the same reason applies for judging the period after thecorrect period as the viewing period in Figures 1−3. In
Figures 1 and 2, the existence of a viewing period before thecorrect period might be attributable to the earlier bloomingof cherry blossoms in 2021 because of abnormal weather
conditions compared to a normal year. Figure 3 shows awider period of time estimated as the viewing period thanthe correct period, presumably because the textclassification model classifies cherry blossoms as blooming,
irrespective of the degree of blooming. Therefore, posts
indicating that cherry trees are about to bloom or that they
are in the process of falling are also used as data. This

When only posts containing the word "cherry blossom" inkanji katakana hiragana in the text were retrieved, thenumber of posts per day was roughly 30, but the value of thetime-series prediction using posts with co-occurring words
in the text was about 400 on average, resulting in a large
error margin. Therefore, to avoid loss of estimation accuracy,
we used moving averages that were larger than the 5- and 7day moving averages used in previous papers.

4

付着

EXPERIMENT RESULTS

This section presents results of co-occurrence word
determination and fine-tuning of the text classification
model conducted in this study, as well as a result of
estimating the best view cherry blossoms. The loss rateachieved using the created text classification model was
16.7%, requiring 41 epochs. The results of the cooccurrence word judgments and the estimation of the best
time to view are presented below. Tables 2 and 3 present theresults we collected for each of the two Tokyo and Kyoto
co-occurrence word datasets obtained from the cooccurrence word judgments. As shown in Table 2, Tokyo
includes the name of Ueno Park, a famous cherry blossom
viewing spot, and petal-related words related to cherry
blossoms. For Kyoto, as shown in Table 3, the names"Gion" and "Uji" are included, which are also famous fortheir cherry blossoms. This finding suggests that the
proposed method for determining co-occurring words candetermine words that are related.

140

International Workshop on Informatics ( IWIN2023 )phenomenon is believed to be the same as that shown in
Figures 1, 2, and 4.
1000

Bert best time estimation

140

Nomal best time estimation

900

co-word nomal

120

co-word bert

800

Correct answer period

100

700
600

80

500
60

400
300

40

Number of Bert posts[posts]]

Number of Normal posts[posts]

We evaluated the results of estimating the best time toview cherry blossoms using conventional methods and theresults of classifying posts using text classification by BERT
to estimate the best times to view cherry blossoms. Table 4
presents each method and the recall rate compliance rate, an
index to evaluate the accuracy of each method, for Tokyo in
2021. Table 5 presents the recall rate compliance rates of therespective methods and the index used to evaluate its
accuracy in Tokyo in 2022. Table 6 presents the recall ratecompliance rates of the respective methods and the index
used to evaluate its accuracy in Kyoto in 2021. Table 7
presents the recall rate compliance rates of each method andits accuracy in Kyoto in 2022. The period from bloom to fullbloom announced by the Japan Meteorological Agency isthe correct period. The recall rate is an indicator reflecting
how well the estimated results fall within that period. The
proportion of the estimated results which are included in thecorrect period is calculated as the compliance rate. In Tables
4−7, the Earlier method refers to results of estimating thebest time to view cherry blossoms without text classification
by BERT. The proposed method refers to results obtainedby estimating the best time to view cherry blossoms basedon the classification of posts using text classification by
BERT. In addition, "or" in each table indicates that a post is
considered to be in the season if it is judged to be in season
either by the results of posts which include kanji, katakana,and hiragana in the body of the post or by the results ofposts that include co-occurring words in the body of the post.
Also, "and" is an estimation of the best time to view cherry
blossoms from a post when both the results of posts
containing kanji, katakana, and hiragana in the body of thepost and the results of posts containing co-occurring words
in the body of the post indicate that the post is estimating thebest time to view cherry blossoms.

200
20
100
0

0
3/1

3/8

3/15

3/22

3/29

4/5

4/12

4/19

4/26

Figure 1: Estimated results in Tokyo in 2021.
180

Nomal best time estimation

co-word nomal
25

co-word bert
140

Correct answer period

120

20

100
15
80

60

10

40

Number of Bert posts[posts]

Number of Normal posts[posts]

30

Bert best time estimation

160

5

20

0

0
3/1

3/8

3/15

3/22

3/29

4/5

4/12

4/19

4/26

Figure 2: Estimated results in Kyoto in 2021.
2000

Bert best time estimation

1800

160

co-word nomal
co-word bert

1600

140

1400

Correct answer eriod

120

1200
100
1000
80
800
60

600
40

400

Table 4: Best Viewing Results for Tokyo in 2021
Earlier
Proposedtype
methodmethodonly sakura
55.6
88.9
co-word0.0
77.8
Recall (%)
or
55.6
88.9
and
0.0
77.8
only sakura
27.8
44.4
co-word0.0
29.2
Precision
(%)
or
18.5
29.6
and
0.0
46.7

Number of Bert posts[posts]

Number of Normal posts[posts]

180

Nomal best time estimation

20

200
0

0
3/1

3/8

3/15

3/22

3/29

4/5

4/12

4/19

4/26

Figure 3: Estimated results in Tokyo in 2022.
Bert best time estimation
Nomal best time estimation

Number of Normal posts[posts]

co-word nomal
co-word bert

200

50

Table 5: Best Viewing Results for Tokyo in 2022
Earlier
Proposedtype
methodmethodonly sakura
50.0
50.0
co-word12.5
100.0
Recall (%)
or
62.5
100.0
and
0.0
50.0
only sakura
26.7
30.8
co-word
4.3
23.5
Precision
(%)
or
16.1
23.5
and
0.0
30.8

45
40

Correct answer period

35

150

30
25

100

20

15
50

Number of Bert posts[posts]

250

10
5

0

0

3/1

3/8

3/15

3/22

3/29

4/5

4/12

4/19

4/26

Figure 4: Estimated results in Kyoto in 2022.

141

International Workshop on Informatics ( IWIN2023 )Table 6: Best Viewing Results for Kyoto in 2021
Earlier
Proposedtype
methodmethodonly sakura
63.6
63.6
co-word0.0
45.5
Recall (%)
or
63.6
63.6
and
0.0
45.5
only sakura
41.2
41.2
co-word0.0
20.0
Precision
(%)
or
25.9
25.9
and
0.0
33.3

estimation, we proposed a method of text classification
using BERT to eliminate posts that are unrelated to cherry
blossom blooming. When the proposed method was used toestimate the best time to view cherry blossoms, the recall
rate and the precision rate were higher by 35.8% and 14.8%,
respectively, compared to results obtained using theconventional method, which indicates that the proposed
method has the potential to improve accuracy. The loss rateof the text classification model created in this paper was
16.7%, suggesting that there is admit for improvement.
Therefore, we would like to use other natural language
processing models such as RoBERTa and DistilBERT forcomparison and analysis in order to improve accuracy.
Using the proposed method to classify text using natural
language processing, it should be possible to collect moredata in the future without relying on co-occurrence
judgments. Therefore, we would like to adapt estimation of
the best time to view cherry blossoms posts to areas where
posts are few, and extend our estimation to areas with moredetailed regional divisions.

Table 7: Best Viewing Results for Kyoto in 2022
Earlier
Proposedtype
methodmethodonly sakura
57.1
100.0
co-word85.7
85.7
Recall (%)
or
100.0
100.0
and
42.9
85.7
only sakura
26.7
38.9
co-word33.3
33.3
Precision
(%)
or
26.9
30.4
and
42.9
46.2

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant
Number 20K12081, 22K13776, and the Telecommunication
Advancement Foundation Research Grant.

Regarding the recall rate precision rate for Tokyo in 2021
in Table 4, 'only sakura', 'co-word', 'or', 'and', the accuracy ofthe proposed method are better than those of theconventional method. In Table 5, for Tokyo in 2022, theaccuracy of the proposed method is better than that of theconventional method for all items, except for the recall rateof 'only sakura'. The recall rate of 'only sakura' is the samefor both the conventional and the proposed methods. In
Table 6, for Kyoto in 2021 the recall rates of 'co-word' and'and', and the precision rates of 'co-word' and 'and' theaccuracy of the proposed method are better than those of theconventional method. The recall rate of 'only sakura' and 'or'
and the precision rate of 'only sakura' and 'or', which areother items, are equal for both the conventional and the
proposed methods. In Table 7, for Kyoto in 2022 the recall
rates of 'only sakura' and 'and', and the precision rates of'only sakura', 'or' and 'and' the accuracy of the proposed
method are better than those of the conventional method.
The recall rate of 'co-word' and 'or' and the precision rate of'co-word', which are other items, are equal for both theconventional and the proposed methods.
In the two areas for which we estimated the best time toview cherry blossoms this time, the recall rate and precision
rates of the proposed method were higher by 35.8% and14.8%, respectively, compared to the conventional method,
indicating the effectiveness of the proposed method. These
results demonstrate that the proposed method can provide
better accuracy than the conventional method.

5

REFERENCES[1] X, https://X.com/ (referred in June 2023)[2] X Developers, https://dev.X.com/ (referred in June
2023)
[3] M. Endo, K. Mitomi, K. Saeki, Y. Ehara, M. Hirota, S.
Ohno, and H. Ishikawa, Study of information provided
by the best time to see estimation method ofphenological observations using tweets, Proceedings ofthe 12th Annual Conference on Japan Society for
Tourism Informatics, Shizuoka, Japan, pp.47-60(2016).
[4] M. Takahashi, M. Endo, S. Ohno, M. Hirota, and H.
Ishikawa, Automatic detection of tourist spots andbest-time estimation using social network services,
International Workshop on Informatics 2020, pp.65-72(2020).
[5] Y. Takamori, K. Terada, M. Endo, S. Ohno, and H.
Ishikawa, Co-occurring Word Determination Used for
Estimating Best Times for Viewing Cherry Blossoms,
MMEDIA 2023 Proceedings, pp.1-6 (2023) .
[6] J. Devlin, M. Chang, K. Lee, and K. Toutanova,
BERT:
Pre-trainingof
Deep
Bidirectional
Transformers for Language Understanding, arXiv
preprint arXiv:1810.04805, (2018).
[7] Google Research, https://research.google/ (referred inJune 2023)[8] J. Sato, Y. Takamori, M. Fujimoto, T. Horikawa, M.
Endo, S. Ohno, M. Hirota, and H. Ishikawa,
Verification of the Best Time to See Cherry Blossoms
Estimation Method using Time Series Prediction
Method, IWIN2022 - Proceedings, pp.95-101 (2022).
[9] J. Meira, J. Carneiro, V. Alonso-Betanzos, et al.,Anomaly detection on natural language processing to

CONCLUSION

This study demonstrated higher accuracy when estimating
the best time to view cherry blossoms using textclassification by BERT. To improve the accuracy of142

International Workshop on Informatics ( IWIN2023 )improve predictions on tourist preferences, Electronics,
Vol.11, No.5, pp.779 (2022).
[10] M. Mathias, A. Moussa, F. Zhou, et al., Optimisation
using Natural Language Processing: Personalized Tour
Recommendation for Museums, 2014 Federated
Conference on Computer Science and Information
Systems, IEEE, pp.439- 446 (2014).
[11] T. Kudo, Mecab: Yet another part-of-speech andmorphological analyzer, http://mecab.sourceforge.net/(referred in June 2023)[12] National Institute for Japanese Language andLinguistics, Electronic Dictionary with Uniformity and
Identity, https://clrd.ninjal.ac.jp/unidic/ (referred inJune 2023)[13] Tohoku NLP Group, cl-tohoku/bertbase-japanese-v2 ·Hugging Face, https://huggingface.co/cl-tohoku/bertbase-japanese-v2 (referred in June 2023)[14] B. Salomon, J. Spaans, Demoji, https://github.com/bsol-omon1124/demoji (referred in June 2023)[15] I. Yukino, neologdn, https://pypi.org/project/neologdn/ (referred in June 2023)

143

International Workshop on Informatics ( IWIN2023 )Relation between time-series forecasting methods and estimation of best times to see
cherry blossoms
Asahi Iha*, Kenji Terada**, Masaki Endo**, Tsuyoshi Tanaka***,Shigeyoshi Ohno**, and Hiroshi Ishikawa*****Electronic Information Course, Polytechnic University, Japanb20305@uitec.ac.jp**
Information and Communication Engineering, Polytechnic University, Japan{k-terada, endou, ohno}@uitec.ac.jp***
Digital Transformation of Skill, Polytechnic University, Japant-tanaka@uitec.ac.jp****
Graduate School of Systems Design, Tokyo Metropolitan University, Japan
ishikawa-hiroshi@tmu.ac.jp
Abstract – Analyzing SNS data enables low-cost

tourist information occurs through many postings. Using thisinformation, one can obtain detailed local information as if
one were talking to people at the sightseeing spot. One
example of a social networking service with shared tourism
information is X [1], a system that allows users to share their
daily post messages within 140 characters. X is generally
used for posting and viewing information about personal
events, hobbies, etc. Posting text, photos, videos, and otherinformation on X is called "posting. Among those messages,
posts with location information added to an arbitrary setting
are called "geotagged posts. Geotagged posts make it possible
to share what is actually happening in the real world byincluding current location information. It is also possible toobtain information related to tourist spots and events throughposts made by local people. Therefore, X is anticipated foruse as a social sensor for estimation and acquisition of local
tourist information in real time.
Using SNSs, it is possible to estimate the seasonality ofplants at a low cost. Currently, biological seasonalobservations, which record plants and organisms, are madeby the Japan Meteorological Agency based on PhenologicalObservation Guidelines.
Nevertheless, phenologicalobservations are costly because they involve visualobservations of sample trees. Furthermore, the Japan
Meteorological Agency has announced that it will drasticallyreduce the scale of phenological observations starting inJanuary 2021. Currently, plants of only six types are subject
to biological seasonal observation. Therefore, estimating thebest time to see cherry blossoms using SNSs is anticipated asa new observation method. We are estimating the best times
to see cherry blossoms by extracting information related tocherry blossoms using geotagged posts on X. Endo et al. [2]proposed a method that uses simple moving averages toestimate the best times to view cherry blossoms. This method,which collects geotagged posts that include specifickeywords, estimates the best time to see species by obtaining
a simple moving average of the number of posts for each date.
In other words, estimating the best time to view cherry
blossoms is possible in prefectures and localities where acertain number of geotagged posts are displayed. Using thisapproach, they confirm the real-world seasonality of posts
related to the names of organisms in each region. Then theyphenological observation. Tourism operators can use resultsof these analyses to plan seasonal events such as cherry
blossom viewing tours. In an earlier study, after posts
including the word "sakura" (cherry blossom) on the subjectof cherry blossom viewing were collected, a time-series
analysis was used to predict the number of future posts. The
distribution state of the weighted moving average of thispredicted time-series data made it possible to estimate thebest time to see cherry blossoms in the future. Nevertheless,results of an earlier study that estimated the best time to see
cherry blossoms using the number of actual posts were lessaccurate than the results of a time series analysis that
estimated the best time to view cherry blossoms. Afterinvestigating this finding, we surmised that rounding thenumber of posts in time-series prediction might have
improved accuracy. Therefore, to improve the accuracy ofcherry blossom season estimation, this study first compares
the number of posts predicted by time series analysis using
DeepAR+ and Prophet, which was used for an earlier study,
with the number of actual posts. Then we compared theaccuracy of each to ascertain how it affects estimation of thecherry blossom season. We investigate the effects of thenumbers of posts on estimation of the cherry blossom viewing
season by comparing the accuracy of the two. Based on thefindings obtained using the respective methods, we willcompare and contrast the methods for improving the accuracyof cherry blossom season estimation through changes in the
degree of accuracy of time-series predictions, and will thenpropose effective methods.

Keywords: Cherry blossoms; machine learning; SNS; text
mining; tourism

1

INTRODUCTION

Today, with the development of information technology,
many people are finding information via the internet.
Particularly with the spread of social networking services(SNSs), users' information is shared. The latest information isdiffused instantly. Therefore, SNSs might be used forgathering tourist information. On SNSs, an overflow of145

International Workshop on Informatics ( IWIN2023 )demonstrate the potential for presenting tourism informationin real time. Takahashi et al. [3] proposed a method using
weighted moving averages to improve the estimation
accuracy of the method presented by Endo et al. Compared tothe method using a simple moving average, this method can
improve the cherry blossom viewing season estimates foreach prefecture while maintaining low costs. Horikawa et al.
[4][5] applied the estimation method presented by Takahashi
et al. and proposed a method for estimating the best time tosee cherry blossoms in the future using time-series forecasts
based on machine learning. Until that study, estimation ofviewing times using actual measurements was only able toestimate the current time. Time-series forecasting is theprocess of predicting data trends over a future period basedon past time-series data. Because cherry blossoms areseasonal plants, time-series forecasting is useful to predict the
number of posts related to cherry blossoms for a certain future
period. The judgment of when the cherry blossoms are at theirbest is made by analyzing the relative trends in the number ofposts. Therefore, we have developed a method for highly
accurate estimation of the best time to view cherry blossoms
using time-series predictions of posts. Sato et al. [6] apply
time-series forecasting using machine learning by Horikawa
et al. to enable the application of a method for estimating thebest times to see cherry blossoms nationwide by covering awide area and by changing the start date of the time-series
forecasting. However, the relation between time-series
forecasting and estimation of cherry blossom viewing season
was unclear. Therefore, this study compares time-series
forecasts by Prophet and DeepAR+. Then, we will investigate
the relation between the time-series prediction and theestimation of cherry blossom season using the characteristicsof the time-series prediction.
The structure of this paper is the following: Section 2
presents a description of research related to this paper;
Section 3 presents the proposed method; Section 4 explains
the experimentally obtained results and their evaluation; andSection 5 gives some conclusions obtained from this study.

2

landscape route information from video captured by asmartphone mounted on a car and shares it with users in realtime. In the cherry blossom sensor research, only informationon the blooming of cherry blossoms along the road isavailable because the cherry blossoms are observed by a car.
In our research, however, many people who use X post posts
from various areas, which allows us to see information on the
blooming of many cherry trees.
Research undertaken to extract tourist information using
SNS, research on the estimation of cherry blossom viewing
times, and research on time-series prediction using machinelearning are progressing. This study was conducted todemonstrate the usefulness of a method for estimating futurecherry blossom viewing times using time series forecasting,especially using machine learning.

3

PROBLEMS AND METHODThis chapter presents descriptions of the data and the method
used to estimate the timing of cherry blossom viewing, as
well as the problems involved.

3.1

ProblemOur research aims at estimating the best time to view
cherry blossoms with a high degree of accuracy on a national
scale. However, prior research has the issue that "the relation
between time-series prediction and estimation of cherry
blossom viewing time is unclear".
This is the problem that although time-series forecastinghas made it possible to estimate the future timing of cherry
blossom viewing, it is not clear how the forecasting of posts
affects the estimation of cherry blossom viewing timing. In
the previous study, estimation of best time was madeimmediately using the conventional condition of best time
for cherry blossoms after the time-series prediction. In thisstudy, we compare algorithms suitable for estimating cherry
blossom viewing time. Next, time-series predictions are madeusing the algorithm suitable for estimating cherry blossom
viewing time, and the predicted results are compared withactual posts to investigate the information obtained bymaking predictions. Based on the obtained information, we
propose a new method by adding new conditions for
estimating the best time to see the cherry blossoms. By using
the proposed method to estimate the timing of cherry blossom
viewing, we can investigate the impact of the predicted postson the estimation of cherry blossom viewing. We hope that
this will solve the problem of "unclear relation between timeseries prediction and estimation of cherry blossom viewing
season".

RELATED RESEARCH

Silva et al. [7] proposed noise reduction in Neural
Network Autoregressive (NNAR) to improve forecasting
accuracy. Better prediction of future tourism demand wasachieved using the NNAR model.
Colladon et al. [8] proposed a new method for analyzing
tourism-related big data and a set of variables that can be
integrated into conventional forecasting models. They wereable to predict the number of passengers at international
airports in Europe, which is useful for the tourism industry.
Takamori et al. [9] proposed a method for estimating thebest times to see cherry blossoms at a certain period using
posts collected with co-occurring words related to cherry
blossoms.post The use of co-occurrence terms enabled themto increase the number of data, thereby improving accuracy.
The recall rate was 34.3%. Moreover, recall rate was 4.0%
higher than for the results obtained using the conventional
method.
Morishita et al. [10] proposed a participatory sensing
system called Sakura Sensor, which automatically extracts

3.2

Data preparation

For this study, we used the X Streaming API to collectgeotagged posts from Japan that include location information[11]. Then, using the simple reverse geocoding service [12]of the Ministry of Agriculture, Forestry and Fisheries basedon latitude and longitude information, the collected
geotagged posts were assigned to the prefecture from which
they originated. The timing of cherry blossom viewing was146

International Workshop on Informatics ( IWIN2023 )estimated by predicting and analyzing the transition of posts
including specific keywords from posts in each prefecture.
The target regions for the experiment were Tokyo and Kyoto,
each of which has a large population, and each of which isfamous for cherry blossoms. The key word was "cherry
blossom" in kanji, hiragana, or katakana; the analyzed period
was February 1, 2015 – February 28, 2022. However, we
excluded 2020 post data from our training data because wewere unable to collect those data.

3.3

through December for forecasting. Although 2020 data are
missing, Prophet is trained normally. We use it as-is without
making up for the missing data. Amazon Forecast's DeepAR+
is a modified version of DeepAR for time series analysis thatjointly trains a single model over several types of time series
data [17]. DeepAR+ is a supervised learning algorithm forforecasting scalar (one-dimensional) time series using
recurrent neural networks (RNNs). Therefore, DeepAR+ is apotentially usable algorithm for this study [18]. The
characteristic period of the number of posts about cherry
blossoms appears during the period from the beginning of
February to the end of April each year. However, thecharacteristic period becomes shorter when viewed on a
yearly basis. Therefore, the training data are processed to takeadvantage of DeepAR+. Data for three months, the period
from the beginning of February to the end of April of eachyear, are extracted. Next, the extracted data are processedwith February 28, 2022 as the base date to make themcontinuous values. To give a specific example of processing,the data for February−April from 2015 through 2021 and for
February 2022 were extracted, with February 1 through April
30, 2021 as November 4, 2021 through January 31, 2022,
February 1 through April 30, 2019 as August 7, 2021 throughNovember 3, 2018, and February 1 to April 30, 2018 as May
10 to August 6, 2021.

Time series forecast

Based on post information collected in the past, time-series
prediction of the number of posts was performed during theestimated cherry blossom viewing period. Because cherry
blossoms bloom mainly during March–April, the estimatedperiod of cherry blossom viewing was the two months ofMarch 1 – April 30, 2022. By performing time-series
forecasting of the number of posts during this period, it waspossible to estimate the best seasons for viewing in twomonth increments. The method uses statistical and machinelearning algorithms from Amazon Web Services (AWS) [13].
We also use a service called Amazon Forecast [14], which
provides very accurate time-series forecasts.
As training data for this study, we use a large dataset of
more than 300 days of data for cherry blossoms that bloom
around spring of each year. Therefore, the algorithms usedfrom Amazon Forecast are Prophet, which is suitable forseasonal training data, and DeepAR+, which is suitable forlarge data of more than 300 days. The predicted value is thevalue which satisfies 50% demand obtained from theweighted quantile loss: a metric used during forecasting with
Amazon Forecast. The predicted number of posts is rounded
down to the nearest whole number. If the number of posts isnegative, then that finding is treated as 0. We use the values
obtained from time-series forecasts to estimate the timing ofcherry blossom viewing. The data to be trained will beprocessed for each algorithm so that the algorithm
characteristics can be used.
Prophet, an additive regression model, combines the fourcharacteristics of periodicity, trend, event, and error to makeforecasts. It is most effective for time series with strong
seasonal effects and several seasons of historical data
[15][16].
Because this study is collecting posts related to cherry
blossom viewing season, the training data will be seasonal,
with features more likely to appear around springtime.
Prophet is an open source software released by Facebook's
Core Data Science team. It is a forecasting method for timeseries data based on an additive model that takes into accountannual, weekly, and daily seasonality as well as holiday
effects in the nonlinear trend. The model equation is shown
below.
y(t)=g(t)+s(t)+h(t)+ε_t

3.4

Best-time estimation method

The condition for estimating the best time to see cherry
blossoms by post transition uses the conventional method ofweighted moving averages, for which each value is weighted.
In the method used for this study, the median value is set as
1. The minimum and maximum values are set respectively as±0.5 from the median value. To estimate the best viewing
period for cherry blossoms, we analyze the frequency of
geotagged posts including the key word. The following threevalues will be analyzed by date and will be used as the basis
for estimating the optimal viewing period.
1.
2.
3.

1-year simple moving average of the number of posts
7-day weighted moving average of posts
5-day weighted moving average of posts

To assess the increase in the frequency of geotagged posts
including the key words, a one-year simple moving averageof the number of posts was obtained. Because posts including
key words also include words that are unrelated to actual
cherry blossoms, one can ascertain the increase in posts
related to actual cherry blossoms by comparing the numberof posts with the one-year simple moving average of thenumber of posts. To analyze the increasing trend of thenumber of posts, a 7-day weighted moving average and a 5day weighted moving average are obtained. Because thenumbers of posts tend to increase on Saturdays and Sundays,a 7-day weighted moving average is obtained. The formula to
be used is shown below. The expression xy denotes thenumber of posts x of y days prior.

(1)y(t) is the predictor variable, g(t) is the trend function, s(t)is the seasonal change, h(t) is the holiday and event term, and
ε_t is the error term that follows a normal distribution. Whenusing Prophet, we use data for all months from January

𝐻𝑎𝑣𝑔7 =

147

0.5𝑥7 +0.67𝑥6 +0.83𝑥5 +𝑥4 +1.17𝑥3 +1.33𝑥2 +1.5𝑥1
7(2)

International Workshop on Informatics ( IWIN2023 )One can also find the weighted moving average of 5 days,
which is the average number of days between the cherry
blossom bloom and full bloom. The formula to be used ispresented below. xy is the number of posts x of y days prior.
𝐻𝑎𝑣𝑔5 =

0.5𝑥5 +0.75𝑥4 +𝑥3 +1.25𝑥2 +1.5𝑥1
5

The mean squared error is calculated using the following
equation: 𝑦𝑖 denotes the ratio of the actual number of posts;
𝑦̂𝑖 represents the ratio of the predicted number of posts; and nstands for the total number of period data. Smaller values of
MSE indicate smaller errors.

(3)

1

𝑀𝑆𝐸 = ∑𝑛=1̂𝑖 )2𝑖=0 (𝑦𝑖 − 𝑦𝑛Using these estimation criteria, we can ascertain thatcherry blossoms are at their best when the following two
conditions are met.
1.
2.

(4)

Another evaluation is the correlation coefficient (R), which is
calculated as follows: 𝑥𝑖 is the ratio of the actual number ofposts, with 𝑥̅ representing its average; 𝑦𝑖 is the ratio of thepredicted number of posts, with 𝑦̅ representing its average;
and n is the total number of period data. An absolute value of
R that is closer to 1 signifies stronger correlation; a value
closer to 0 represents weaker correlation.

The number of posts is greater than the 1-year simple
moving average.
The 5-day weighted moving average of the number ofposts is greater than the 7-day weighted moving
average for three consecutive days.

𝑅=

These conditions are used to estimate the best time period
to see each attraction. Using conventional methods, theestimation conditions for cherry blossom viewing were
defined based on the actual number of posts.
For this study, the maximum number of posts per day in atime-series forecast period is designated as the maximum
forecast peak value. The total number of posts for the timeseries forecasting period is designated as the total number ofposts. The maximum predicted peak value and the total
number of posts can only be obtained by performing timeseries forecasting. This forecasting is used as a condition for
estimating the best time to see the cherry blossoms. We
propose two new conditions for estimating the best time toview cherry blossoms. The conventional method uses a oneyear moving average because the number of posts in the
future is not known. In proposed method 1, the maximum
predicted peak value is used to narrow down the estimation
period from the one-year moving average, thereby improving
the precision rate accuracy. In proposed method 2, theaccuracy of the estimated precision rate is improved byestimation from a new angle.

4
4.1

1 𝑛
∑ (𝑥 −𝑥̅ )(𝑦𝑖 −𝑦̅)𝑛 𝑖=1 𝑖1 𝑛
1
√ ∑𝑖=1(𝑥𝑖 −𝑥̅ )2 ×√ ∑𝑛(𝑦 −𝑦̅)2𝑛𝑛 𝑖=1 𝑖(5)

RESULTS AND DISCUSSION
Results

This chapter presents results obtained using Prophet and
DeepAR+ for the cherry blossom viewing season estimation
conditions from earlier studies, as well as cherry blossom
viewing season estimation using the proposed method.
Prediction of the best time to view cherry blossoms can beachieved by combining post information and the best time toview estimation conditions. The accuracy of estimation isevaluated by comparing the estimated results with the correct
prediction period, and by evaluating the recall rate andprecision rate. The correct prediction period is defined as theperiod from the cherry blossom blooming date to the fullblooming date, as observed by the Japan Meteorological
Agency. The cherry blossom bloom date is the first day on
which 5−6 or more cherry blossoms are in bloom on a sample
tree [19]. The cherry blossom full bloom date represents thefirst day on which more than 80% of the cherry trees in thesample tree are open. First, we present results obtained for
estimating the cherry blossom viewing time based on thenumber of posts obtained using Prophet as the algorithm fortime-series prediction. Figures 1 and 2 respectively present
results obtained for Tokyo and Kyoto for 2022: they are
estimates of the cherry blossom viewing season based on thenumber of posts obtained from the time-series forecast.
Figures 3−4 present the results obtained from estimating
the best time to view cherry blossoms in Tokyo and Kyoto in
2022, based on the number of posts obtained using DeepAR+as the algorithm for time-series prediction.

Proposed method 1:
A number of posts that is larger than the predictedmaximum number of posts multiplied by 0.3, 0.5, or 0.7, andmeeting the estimation criteria of the conventional method is
judged to be the best time to view the cherry blossoms.
Proposed method 2:
The ratio of the total number of posts to the cumulative
number of posts from March 1 to a certain date is used todetermine the time at which the cherry blossoms are at theirpeak.

3.5 Method for determining the relation
between time-series forecasts
To normalize the number of posts obtained from the timeseries forecast to the actual number of posts, we divide thenumber of posts per day for the period March 1 − April 29,
2022 by the total number of posts for that period to obtain theratio of posts per day for the period. The ratio of the numberof posts is then evaluated using the mean squared error (MSE).

148

International Workshop on Informatics ( IWIN2023 )400

Number of tweets[posts]

time series forecasts are shown respectively in Table 1.
From Figures 1 and 2, Prophet predicts a wide period oftime, with the correct response period included in theestimation period. This finding might be attributable to thefact that the number of posts in the time-series prediction
increases gradually, reaches a maximum value, and thengradually decreases, thereby making it easier to match theestimation conditions.
Figures 3 and 4 show that DeepAR+ has a distorted graphshape for the number of posts compared to Prophet. In Figure
3, the period differs from the correct period. There are two
peaks. It is unlikely that the cherry blossom viewing season
will return once it has subsided. In Figure 4, the prediction ismade before the correct period. Table 1 shows that DeepAR+
has a lower recall rate score than Prophet has. MSE values arelower for Prophet compared than for DeepAR+ in Tokyo andKyoto, indicating a smaller error between the actual numberof posts and the predicted number of posts. In addition, thevalue of R is closer to 1 for Prophet than for DeepAR+,
indicating strong correlation. However, the higher score for
DeepAR+ compared to that for Prophet is the precision rate
for Tokyo. Kyoto's precision rate was 0% because of the
failure of the prediction. Prophet is the superior prediction
method for estimating the best time to see cherry blossoms.
Next, Figures 5 through 7 show the results of estimating thebest time to see cherry blossoms using Proposal 1 from March
1, 2022 to April 29, 2022. Figure 5, Figure 6, and Figure 7
respectively portray the results obtained using Proposal
Method 1 in Tokyo by factors of 0.3, 0.5, and 0.7.

Forecasted
valueBest time
estimation
1-year moving
average5-day moving
average7-day moving
average

Correct answer period

350
300
250
200
150
100
50

0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

Figure 1: Tokyo Forecast for 2022 (Prophet).
70

Number of tweets[posts]

Forecasted value Correct answer period
6050Best time
estimation40

1-year moving
average5-day moving
average30

7-day moving
average

20
10
0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10 4/15

4/20

4/25

Figure 2: Kyoto Forecast for 2022 (Prophet).
70

Forecasted value

Number of tweets[posts]

Correct answer period

Best time estimation

60

Table 1: DeepAR+ and Prophet ratings

1-year moving average
50

Area

RecallPrecision

MSE

100.0%

33.3%

8.90e-05 0.682

Deep
AR+

75.0%

42.9%

1.32e-04 0.212

Prophet100.0%

33.3%

8.46e-05 0.794

0.0%

0.0%

4.96e-04 -0.289

5-day moving averageR

7-day moving average

40

Prophet

30

Tokyo

20
10
0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

Kyoto

Figure 3: Tokyo Forecast for 2022 (DeepAR+).
Correct answer period

Number of tweets[posts]

45
40

35

Forecasted value
Best time estimation

400

1-year moving average

350

Number of tweets[posts]

50

Deep
AR+

5-day moving average

30

7-day moving average

25
20
15
10

5
0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

300
250
200
150
100

Best time
estimation
1-year moving
average5-day moving
average7-day moving
average
0.3 times the
maximum value

50
0
3/1

Figure 4: Kyoto Forecast for 2022 (DeepAR+).

Forecasted value Correct answer period

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

Figure 5: 0.3 times of proposed method 1 (Tokyo).
The recall rate and precision rate, MSE, and R for
estimating the cherry blossom viewing season using thenumber of posts obtained from the Prophet and DeepAR+149

International Workshop on Informatics ( IWIN2023 )400
Best time
estimation
1-year moving
average5-day moving
average7-day moving
average
0.5 times the
maximum value

300
250
200
150

Number of tweets[posts]

Number of tweets[posts]

70

Correct answer period

Forecasted value

350

100
50
0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

Forecasted value Correct answer period

60Best time estimation
1-year moving average

50

5-day moving average

40

7-day moving average

0.7 times the maximum
value

30

20
10
0
3/1

Figure 6: 0.5 times proposed method 1 (Tokyo).

3/6

3/11 3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

Figure 10: 0.7 times proposed method 1 (Kyoto).
400

Correct answer period

Number of tweets[posts]

Forecasted value

350

Table 2 shows the recall rate and precision rate for estimating
the cherry blossom viewing season using proposed method 1.
Figures 11−12 present results of the estimation of cherry
blossom viewing time using proposed method 2 from March
1, 2022 through April 29, 2022. Table 3 presents the recall
rate and precision rate of the estimation of cherry blossom
viewing season using suggested method 2.

Best time estimation

300

1-year moving
average250

5-day moving
average200

7-day moving
average

150

0.7 times the
maximum value

100
50
0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

Table 2: Recall rate and Precision rate of viewing time
estimation method using proposed method 1

4/25

Figure 7: 0.7 times proposed method 1 (Tokyo).

Area

Next, Figures 8−10 portray the results of the viewing period
estimated using the proposed method 1 from March 1, 2022
to April 29, 2022. Figure 8, Figure 9, and Figure 10
respectively portray results obtained using proposed method
1 in Kyoto by factors of 0.3, 0.5, and 0.7.

Number of tweets[posts]

70

RecallPrecision

100.0%

33.3%

100.0%

38.1%

100.0%

42.1%

0.7

62.5%

33.3%

Actual100.0%

33.3%

100.0%

36.8%

0.5

100.0%

46.7%

0.7

71.4%

41.7%

Actual
0.3

Tokyo

0.5

Forecasted value Correct answer period

60Best time
estimation
1-year moving
average5-day moving
average7-day moving
average
0.3 times the
maximum value

50
40
30

20

0.3
Kyoto

400
Forecasted value

10

Correct answer period

0
3/1

3/6

3/11 3/16 3/21

3/26 3/31

4/5

4/10

4/15 4/20

Number of tweets[posts]

350
4/25

Number of tweets[posts]

Figure 8: 0.3 times proposed method 1 (Kyoto).
70

Forecasted value

60

Best time
estimation
1-year moving
average5-day moving
average7-day moving
average
0.5 times the
maximum value

50
40
30

Correct answer period

Best time estimation

300
250
200
150

1-year moving
average5-day moving
average7-day moving
average

100
50
0
3/1

20

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

Figure 11: Proposed Method 2 (Tokyo).

10
0
3/1

3/6

3/11

3/16

3/21

3/26 3/31

4/5

4/10

4/15

4/20

4/25

Figure 9: 0.5 times proposed method 1 (Kyoto).

150

4/25

International Workshop on Informatics ( IWIN2023 )Number of tweets[posts]

70

CONCLUSION

Correct answer period
Forecasted value

60

Time-series forecasting using DeepAR+ resulted in
distorted graphs and low accuracy. On the other hand,
Prophet increased the number of posts in the time-series
prediction gradually, reaching a maximum value and thengradually decreasing, which is considered to have increased
the accuracy. These results indicate that the time-series
forecasting method changes the prediction of the number ofposts and affects the accuracy of the estimation of the cherry
blossom viewing season.
As a result of time-series forecasting using Prophet, wewere able to know the total number of posts for a certain
period of time and the maximum value for that period. We
proposed a new method for estimating the best time to viewthe cherry blossoms by adding a new condition for estimating
the best time to view the cherry blossoms using the values
that can be obtained by the prediction. After the time-series
forecasting, the proposed method was able to refine theestimation of the best time to view the cherry blossoms andimprove the recall rate and the precision rate. This is thought
to be due to the effect of time-series forecasting on theestimation of cherry blossom viewing time.
In the future, we intend to further clarify the relation
between time-series forecasts and estimation of cherry
blossoms viewing time, and based on this, improve theaccuracy of estimation of cherry blossom viewing time andexpand the area covered.

Best time estimation50
1-year moving average

40

5-day moving average

30

7-day moving average

20
10

0
3/1

3/6

3/11

3/16

3/21

3/26

3/31

4/5

4/10

4/15

4/20

4/25

Figure 12: Proposed Method 2 (Kyoto).
Table 3: Recall rate and Precision rate of viewing time
estimation method using proposed method 2
Area
ActualProposed
Method 2

Precision

100.0%

33.3%

100.0%

66.7%

100.0%

33.3%

100.0%

70.0%

Tokyo

ActualProposed
Method 2

RecallKyoto

Describe the results of comparing Figures 5-12. The recall
rate was higher when estimation was done at 0.3x, but the
precision rate tended to be low. At 0.5x, both the recall rate
and the precision rate were higher. At 0.7x the precision rate
was higher, but the recall rate tended to be low. The reason
for this result is thought to be that the estimation conditions
for cherry blossom viewing at 0.3x were less stringent and theestimation period was wider, thereby yielding the results
described above. The result is also thought to be that theestimation conditions for cherry blossom viewing at 0.7x
were stringent and the estimation period was narrower,
thereby yielding the results described above. The use of afactor of 0.5, which is intermediate between 0.3 and 0.7, is
expected to improve the accuracy of the estimation results.
Compared to the conventional method, proposed method 2
increased the precision rate by 33.1% in Tokyo and by 36.7%
in Kyoto. Results demonstrated that changing the parameters
used for estimating the cumulative ratios increased theaccuracy. These results indicate that the proposed method canprovide higher accuracy than the conventional method.

4.2

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Number
20K12081, 22K13776, and the Telecommunication
Advancement Foundation Research Grant.

REFERENCES[1] X, https://twitter.com/ (referred in June 2023)[2] M. Endo, K. Mitomi, K. Saeki, Y. Ehara. M. Hirota, S.
Ohno, and H. Ishikawa, Study of information provided by
the best time to see estimation method of phenologicalobservations using posts, Proceedings of the 12th Annual
Conference on Japan Society for Tourism Informatics,
Shizuoka, Japan, pp.47–60 (2016).
[3] M. Takahashi, M. Endo, S. Ohno, M. Hirota, and H.
Ishikawa, Automatic detection of tourist spots and besttime estimation using social network services,
International Workshop on Informatics 2020, pp.65-72(2020).
[4] T. Horikawa, M. Takahashi, M. Endo, S. Ohno, M. Hirota,and H. Ishikawa, Estimating the best time to see cherry
blossoms using SNS and time-series forecasting of post
numbers using machine learning, International Workshop
on Informatics 2021, pp.37-44 (2021).

Discussion

Other researches that have conducted post forecasting haveincluded forecasting word fads and hits [20][21]. Theircommon problem is the quantity and quality of data. This isthought to be due to the fact that they are dealing with data
based mainly on human behavior. This has led to low
forecasting accuracy. On the other hand, my research deals
with posts based on natural phenomena. Unlike predicting
human behavior, posts based on natural phenomena areperiodic. Therefore, it may be said that we were able to maketime-series predictions at a practical level.

[5] T. Horikawa, M. Takahashi, M. Endo, S. Ohno, M. Hirota,and H. Ishikawa, Estimating the Best Time to View Cherry
Blossoms Using Time-Series Forecasting Method,
Multidisciplinary Digital Publishing Institute, pp.418-431(2022).

151

International Workshop on Informatics ( IWIN2023 )[6] J. Sato, Y. Takamori, M. Fujimoto, T. Horikawa, M. Endo,
S. Ohno, M. Hirota, and H. Ishikawa, Verification of theBest Time to See Cherry Blossoms Estimation Method
using Time Series Prediction Method, IWIN2022Proceedings, pp.95-101 (2022).
[7] E. S. Silva, H. Hassani, S. Heravi, and X. Huang,
Forecasting tourism demand with denoised neuralnetworks, Annals of Tourism Research, Vol.74, pp.134154 (2019).
[8] A. F. Colladon, B. Guardabascio, and R. Innarella, Usingsocial network and semantic analysis to analyze online
travel forums and forecast tourism demand, Decision
Support Systems, Vol.123, pp.113075 (2019).
[9] Y. Takamori, K. Terada, M. Endo, S. Ohno, and H.
Ishikawa, Co-occurring Word Determination Used for
Estimating Best Times for Viewing Cherry Blossoms,
MMEDIA 2023 Proceedings, pp.1-6 (2023).
[10] S. Morishita, S. Maenaka, D. Nagata, et al.,SakuraSensor: Quasi-Realtime Cherry-Lined Roads
Detection through Participatory Video Sensing by Cars,
Proceedings of the 2015 ACM International Joint
Conference on Pervasive and Ubiquitous Computing, pp.
695-705 (2015)[11] X Developers, https://dev.twitter.com/ (referred in June
2023)
[12] Agricultural Research Institute, Simple Reverse
GeocodingService,https://aginfo.cgk.affrc.go.jp/
rgeocode/index.html.ja (referred in June 2023)[13] AWS, https://aws.amazon.com/jp/console/ (referred inJune 2023)[14] Amazon Forecast, https://aws.amazon.com/jp/forecast/(referred in June 2023)[15] Prophet, https://docs.aws.amazon.com/ja_jp/forecast/latest/dg/aws-forecast-recipe-prophet.html (referred inJune 2023)[16] S. J. Taylor and B. Letham, Forecasting at Scale, The
American Statistician, Vol.72, pp.37-45 (2018)[17] D. Salinas, V. Flunkert, J.Gasthaus, and T. Januschowski,
DeepAR: Probabilistic forecasting with autoregressive
recurrent networks, International Journal of Forecasting,
Vol.36, pp.1181-1191 (2020)[18] DeepAR+, https://docs.aws.amazon.com/ja_jp/forec
ast/latest/dg/aws-forecast-recipe-deeparplus.html(referred in June 2023)[19] Japan Meteorological Agency https://www.data.jma.
go.jp/sakura/data/index.html (referred in June 2023)[20] A. Khare, Q. He and R. Batta, Predicting gasoline
shortage during disasters using social media, Or Spectrum,Vol.42, pp.693-726 (2020)[21] PF Pai and CH Liu, Predicting vehicle sales by sentiment
analysis of tweet data and stock market values, IEEE
Access, Vol.6, pp.57655-57622 (2018)

152

Ke
Prof. Dr. Hitoshi Aida
(Emeritus Professor
The University of Tokyo)(Chair: Yoshia Saito)

International Workshop on Informatics ( IWIN2023 )155

International Workshop on Informatics ( IWIN2023 )156

International Workshop on Informatics ( IWIN2023 )157

International Workshop on Informatics ( IWIN2023 )158

International Workshop on Informatics ( IWIN2023 )159

International Workshop on Informatics ( IWIN2023 )160

International Workshop on Informatics ( IWIN2023 )161

International Workshop on Informatics ( IWIN2023 )162

International Workshop on Informatics ( IWIN2023 )163

International Workshop on Informatics ( IWIN2023 )164

International Workshop on Informatics ( IWIN2023 )165

International Workshop on Informatics ( IWIN2023 )166

International Workshop on Informatics ( IWIN2023 )167

International Workshop on Informatics ( IWIN2023 )168

International Workshop on Informatics ( IWIN2023 )169

Session 5:
Multimedia and Communication(Chair: Yuichi Tokunaga)

International Workshop on Informatics ( IWIN2023 )Comparative of Experienced and Inexperienced People by Shoot Form Analysis from
Free Throws of Basketball using 2-D Joint Position information and Slow Motion
TETSUTO MUKAI *, SHOTA ISHIGAKI **, and KEIICHI ABE *,* Kanagawa Institute of Technology, Atsugi City, Kanagawa, Japan** Q’SFIX CO.,LTD.
mukai.tetsuto@gmail.com, abe@he.kanagawa-it.ac.jp
Abstract -In this study, we analyzed the shooting form of

movements. By the way, OpenPose [1] is a technology for
analyzing the human posture without using markers.
OpenPose, developed by Zhe Ca et al., is a system that uses
deep learning to estimate a person's skeleton. Using
OpenPose, the 2D coordinates of 18 joint positions of aperson can be obtained without using markers. This enablesdetailed analysis of the coordinates of characteristic points
on the human body. In a previous study[2] of sport analysis
using OpenPose, shooting scenes are detected in soccer
videos and classified into three levels of skill from beginner
to expert based on skeletal estimation. Another study[4]analyzed the spike form of a volleyball player by skeletal
estimation and investigated the relationship between the
spike form and athletic ability. There is a study [5] from thebasketball free throw line using OpenPose. In this study, we
analyzed the form when the knees were bent to the
maximum during free throw shooting, the width of the knees
at the time of release, and the height of the arms. We areproposing a technology to display in real time. However, it
is not described in detail as an ideal form in this study. In
this study, we investigated the ideal shooting form ofbasketball players by comparing the posture of skilled andinexperienced players based on 2D skeletal analysis of theirshooting from the free throw line.

basketball players, using the OpenPose software to acquire
two-dimensional (2D) joint position information without
using markers. Slow-motion imaging proved to be effective
for the posture analysis of the free throw motion. The
shooting form of experienced players was compared withthat of inexperienced players to obtain skeletal informationthat is useful for determining an optimum shooting form.
Keywords: Basketball, 2D Joint Position information, shootform.

1. INTRODUCTION
Basketball is a very popular sport that has spread to morethan 1 million people in Japan, including basketballenthusiasts. Shooting is essential to score points inbasketball. However, the shooting ability in basketballvaries from person to person, with some players having ahigh probability of scoring a goal and others having only alow probability. Even the players who have the same
experience and practice significantly vary in shooting ability.
Therefore, this study aims to improve the shooting rate ofbasketball players based on scientific findings.
In previous studies, there are technologies using
OpenPose[1][2] and Kinect[3] for estimating the humanskeleton from video images and for form analysis during
sports. Kinect technology requires the use of a marker and a
depth-sensor camera to acquire skeletal information, while
OpenPose can acquire 2D Joint Position information from acamera image without using a marker or a depth-sensor
camera. Therefore, we believe that OpenPose is a suitable
analysis method for analyzing dynamic sports, such as
basketball, and we focused on 2D Joint Position informationacquisition using OpenPose technology in this study. This
study utilized this two-dimensional skeletal informationacquisition technology to analyze the optimal shooting form
from the free-throw line in basketball by comparing theposture information of experienced and inexperienced
basketball players to determine which form is most likely toscore a goal. The effectiveness of slow-motion imaging of
the subjects' shooting form was also evaluated, and thedetails are described in this paper.

3. 2D JOINT POSITION INFORMATIONACQUISITION SYSTEM
Figure 3-1 shows an overview of the 2D Joint Position
information acquisition system used in this paper, and Table
3-1 shows the specifications of the equipment and software
used in this system. The system uses a smartphone and ahigh-precision camera to acquire video of the subject
shooting from the free-throw line, and then performs posture
analysis based on 2D skeletal information using OpenPose,
which is open-source software. As shown in Figure 3-1, thehardware used in this study was a Jetson NANO equippedwith both a GPU and CPU.
Table 3-1: Specifications of equipment and software used.

2. RELATED WORKS
Microsoft's Kinect[3] is a representative technology for
acquiring 2D Joint Position information. Kinect requires amarker to be attached to the body, making it difficult to use
for sports such as basketball, which involve vigorous

173

International Workshop on Informatics ( IWIN2023 )θ1 from the two coordinate data of the right shoulder and theright elbow obtained by OpenPose, as shown in Figure 4-1.
The right knee angle θ2 was calculated from the threecoordinate data of the right hip, right knee, and right ankle.
The right elbow angle θ3 was calculated from the threecoordinate data of the right shoulder, right elbow, and rightwrist using the inverse sine function and the cosine theorem.
Figure 4-1 shows the calculation method and the coordinate
numbers of the acquired 2D skeletal data. First, the angle θ1of the right armpit with reference to the right shoulder was
calculated using equations (4-1) and (4-2) according to thesign of H1, where H1 is the height of the right shoulder fromthe right elbow, and D1 is the horizontal distance between
the right shoulder and the right elbow as shown in Figure 41. Since the origin in the OpenPose analysis image is at theupper left corner, the angle θ1 of the right armpit can becalculated using the positive and negative sign of theparameter H1.

Figure 3-1:Overview of 2-D Joint Position acquisition
system．

When H1≥0,(4-1)
When H1<0,(4-2)
Next, the right elbow angle θ3 was calculated from thecosine theorem by equation (4-3), using the lengths L fromthe right shoulder to the right elbow, L2 from the rightelbow to the right wrist, and L3 from the right shoulder tothe right wrist.

Figure3-2:Skeleton information analyzed by OpenPose.

(4-3)

The 2D joint position information was obtained in theComma Separated Value (CSV) format from still images
with 2D skeletal information analysis and 2D Joint Position
information (X-axis and Y-axis coordinate data) for every
two frames. The upper left corner of the still image is theorigin of coordinates. In this study, to enable detailed
analysis of the armpit angle, elbow angle, and knee angle,
which are important for shooting from the free-throw line inbasketball, we extracted and analyzed the data for the rightshoulder, right elbow, right wrist, right hip, right knee, andright ankle from the body part data actually acquired.

Next, the right knee angle θ2 was calculated using thecosine theorem by equation (4-4), where, as shown in
Figure.4-1, a is the length from the right hip to the rightankle, b is the length from the right knee to the right ankle,
and c is the length from the right hip to the right knee.
(4-4)

4. Analysis Methods

4.1 Body part analysis
In this paper, we evaluated the shooting analysis from thefree throw line by comparing the data of experienced
basketball players with those of inexperienced players.
Experienced players in this study are those who still belong
to a basketball club or those who used to belong to abasketball club but have been away from basketball for ashort period of time (less than one year), and all otherplayers are treated as inexperienced players. In analyzingbasketball shooting from the free throw line in this paper,
we considered the armpit angle, knee angle, and elbow angle
to be important factors, based on our analysis ofexplanations by experienced basketball players. As shown in
Figure 4-1, we transformed the 2D skeletal information into
these three parameters and analyzed the posture during
shooting. In this paper, we calculated the right armpit angle

Figure 4-1:Method of calculating angles.

174

International Workshop on Informatics ( IWIN2023 )the rapid movement from the motion break to the release.
This is thought to be due to the fact that the arm swinging
motion becomes faster when releasing the ball, causing thevideo image to be blurred during the analysis. In theprevious study[6] , the distance between the subject and thecamera was too far, which made the subject appear to besmaller, and the blurring of the analyzed images during fastarm movements such as during release is considered to havecaused a detection error when acquiring skeletal information.
For this reason, the authors considered that it would bebetter to place the camera closer to the subject and shootfrom right beside the subject. Therefore, based on the resultsof the previous studies, we devised the following
improvement measures (1) to (4) to reduce the detection
error in skeletal information acquisition, and verified themin a demonstration experiment.
1) Shoot at a close distance so that the subject appears
larger2) Shoot from right beside the subject instead of 45
degrees from the subject
3) Change from normal speed to slow motion shooting
4) Increased frame rate from 60 fps to 120 fps

4.2 Methods of form analysis during shooting
According to the Basketball Medical Support Lab
website[5], the important points in the flow of motion fromthe ball set to the release are the posture at the "motion
break", which is between the set and release motions, aswell as the posture at the "release". The definitions of"motion break" and "release" in this paper are as follows.
First, the definition of "motion break" focuses on the knee
extension motion during the shooting motion. Figure 4-2
shows a graph of the right armpit angle θ1, right knee angle
θ2, and right elbow angle θ3 of an experienced player at thetime of a successful shooting, obtained by OpenPose. The
horizontal axis is the number of frames. As shown in Figure
4-2, the gray graph is the right knee angle θ2, and the point
where the right knee fully extends again after the angle
reaches the minimum value is the point where the shot isreleased. As shown in Figure 4-2, the point where the rightknee begins to extend again after the right knee angle θ2reaches its minimum angle is defined as the "motion break"in this paper. On the other hand, as shown in Figure 4-2, the"release" is defined as the point where the development
movement of the right knee angle θ2 is near its maximum
value and the ball is away from the hand. In the case of an
NBA professional player active in the U.S., it is suggested
that at the "motion break," the right arm is lifted to an armpit
angle of 90°, the elbow rises further toward the release, andthe elbow joint extension motion begins at the same time. In
other words, the shoulder joint and elbow joint movesimultaneously after the 90° angle of the armpit. Therefore,
in this paper, we analyze the shooting form from the free
throw line in detail by simultaneously calculating the
subject's right armpit angle θ1, right knee angle θ2, and rightelbow angle θ3 using 2D Joint Position informationacquisition during the "motion break" and "release" motions.

4.3 Demonstration experiment method
In our previous study [6], the subjects were six righthanded males, two of whom had basketball experience, andthe remaining four were inexperienced. The shooting
location was the Arena Gymnasium of the Kanagawa
Institute of Technology, and the shooting video was takenfrom the free throw line of the basketball. We used an
iPhone8puls camera (specifications: 4k, 60fps), and thecamera was set up at 45 degrees from the subject.

Figure 4-2: The right armpit angle θ1, right knee angle θ2
and right elbow angle θ3 of an experienced player.
Table4-1:Analysis error in 2-D joint position information.

4.4 Discussion of previous studies
Figure 4-3 shows an image of 2D Joint Position
information acquired as a result of the analysis of the past
study by using the commercially available Vision Pose fromNEXT-SYSTEM. In the previous study[6] , detection errors
often occurred during skeletal information acquisition in therelease motion, making the analysis difficult. Figure 4-4
shows the image in which errors occurred during skeletal
information acquisition, and Table 4-1 shows the values ofthe errors in the analysis of 2D Joint Position informationacquisition during the motion break and release motion. The
image in Figure 4-4 is the image at the time of release. It canbe seen that the skeletal detection points of the right elbow
and right wrist are displaced from the right arm in theforeground of the image and cannot be detected correctly.
Table 4-1 shows that the detection errors are larger at theangle of the right armpit and especially during the period of

Figure 4-3:2-D joint position information image[6].

175International Workshop on Informatics ( IWIN2023 )5.1 Evaluation of the proposed methodology
In this section, we evaluate the effectiveness of acquiring
2D Joint Position information using OpenPose on videos
captured with the proposed method (improvement measures)
described in section 4.4. Figure 5-1 shows a comparison
between the proposed method and previous research data.
The horizontal axis of the graphs in Figure 5-1 is the numberof frames, and the vertical axis shows the right armpit angle
θ1, right knee angle θ2, and right elbow angle θ3 acquiredwith 2D Joint Position information. The video shot using themethod proposed in this paper was shot in slow motion at aframe rate of 120 fps, more than two times that of theprevious studies, so the number of acquired frames
increased as shown in Figure 5-1. In addition, the graphs of
the right armpit angle θ1, right knee angle θ2, and rightelbow angle θ3 obtained with the 2D Joint Position
information are very detailed, and in particular, the data foreach angle during the release can be obtained accurately
without missing any data. This confirmed the effectivenessof the proposed method, and we proceeded with the analysis
under these conditions.

Figure 4-4:Error in 2-D joint position information.
Table5-1 Specifications of Camera

5.2
Motion analysis of experienced andinexperienced basketball players
In this study, the motion analysis of the experienced andinexperienced players was conducted using the videos of
their successful shooting. Figure 5-2 shows still images ofexperienced and inexperienced players shooting from thefree throw line, converted into 2D Joint Position information
by OpenPose. The images in Figure 5-2 are divided into
three parts: set, motion break, and release, with the numberof frames indicated for each part. Table 5-2 shows the rightarmpit angle θ1, right knee angle θ2, and right elbow angle
θ3 calculated from the 2D Joint Position information dataacquired during the "motion break" and "release" phases ofthe successful shooting by experienced and inexperienced
players. The data of NBA player in Table 5-2 were manuallycalculated from the video images on the Internet [5]. As
shown in Table 5-2, the armpit angle θ1 of Experienced
Player 1 at the motion break is 87°, which is close to the
NBA player 90°, while the value of θ1 is as high as 116° forInexperienced Player 1. On the other hand, the armpit angle
θ1 of Experienced Player 1 at the time of release was 144°,
which was close to the NBA player's 140°, while that ofInexperienced Player 1was lower at 125°. The results showthat there is a difference in the values of θ1 and θ2 between
the experienced and inexperienced players at the "motion
break" and "release," and that the experienced player's
posture and movement are closer to those of the NBA player
in the values of θ1 and θ2. Figure 5-3 shows the graphs of
the right armpit angle θ1, right knee angle θ2, and rightelbow angle θ3 against the number of frames on thehorizontal axis for the experienced player. Red vertical lines
indicate the number of frames for the set, motion break, andrelease motion. The graph of the knee angle θ2 in Figure 5-3
shows that at the motion break, the right knee angle θ2 once
reaches its minimum value and the knee begins to extend
from the minimum value, while the release motion starts at
the motion break and ends with the right armpit angle θ1
being at its maximum value.

Figure 5-1: Comparison of data obtained with the proposed
method and those obtained in previous study.

5. DEMOSTRATION EXPERIMENT
We conducted a demonstration experiment byimplementing improvement measures based on the results ofthe previous studies. In this demonstration experiment, thevideo shooting was conducted twice, with four subjects(including one with basketball experience) for the firstshooting and eight subjects (including five with basketball
experience) for the second shooting. All subjects were righthanded males. The location of the shooting was inside the
Arena Gymnasium of Kanagawa Institute of Technology. A
high-performance camera was used to shoot a slow-motion
video of the subject throwing a ball from the center of thefree throw line, 4.225 m from the basketball goal. The
camera was set up on the extension of the free throw line ata distance where the subject shooting the ball was seen on
the entire screen of the camera, and the camera was fixed ona tripod so that the subject was right beside the camera. The
camera used for this shooting was SONY α 7 Ⅲ . The
specifications of this camera are shown in Table 5-1. This
high-performance camera has a slow-motion shooting
function, which was used in this study.

176

International Workshop on Informatics ( IWIN2023 )indicate the amount of change from the motion break to therelease. As shown in Table 5-3, the right armpit angle θ1
showed a large difference between the experienced andinexperienced players. Experienced Player 1's right armpit
angle θ1 at the motion break was around 90°, and the rightarmpit angle θ1 at the time of release was around 140°.
These values are comparable to those of the NBA player. By
the way, the values of θ1 and θ3 of Inexperienced Player 3
are similar to those of Experienced Player 1. However, as
shown in Figure 5-4, when we look at the series of shooting
motions, Experienced Player 1 shoots with his arms closed
and his left hand on the ball, while Inexperienced Player 3
shoots with his arms open and in the same position.
Inexperienced Player 3's shooting form is different from thatof Experienced Player 1. However, when Inexperienced
Player 3 shoots with both hands, the values of θ1 and θ3
happened to be comparable to those of Experienced Player 1.
Next, the amount of change in the right elbow angle θ3 fromthe "motion break" to the "release" is examined. The Δθ3 ofExperienced Player 1 is 104°, which is close to that of the
NBA player (120°). The Δθ3 values of Inexperienced
Players except for Inexperienced Player 3 were less than100°. Table 5-3 also shows the number of frames in which
the right armpit angle θ1 was 90±5° during the transition
from "motion break" to "release" in the movie of a
successful shot, and the presence or absence of the angle. It
was confirmed that the number of frames in which the rightarmpit angle θ1 was 90±5° was more than one for all
subjects except for Inexperienced Player 1. Table 5-3 shows
the success rates of the experienced and inexperienced
shooters. The success rate was calculated as the percentageof shots that resulted in a goal in five throws, but when all of
the five throws resulted in failure, the player was asked tothrow the ball again until it went in. The results show thatthe success rate of Experienced Player 1 was 75%, while theother inexperienced players had a success rate of less than
20%.

Figure5-2:Shooting motion of experienced andinexperienced players.

Figure 5-3: Armpit angle θ1, knee angle θ2 and elbow angle
θ3 of Experienced Player 1 plotted against the number offrames.
Table 5-2:Comparison between experienced andinexperienced players.

Table5-3: Comparison of θ1, θ2 and θ3 between
experienced and inexperienced players in successful
shooting

5.3 Comparative evaluation of Experienced
and Inexperienced Players in successful
shooting
Next, a comparative evaluation of experienced andinexperienced subjects at the time of the successful shooting
was conducted. Four subjects, one experienced and three
inexperienced, participated in this study. Table 5-3 shows
the right armpit angle θ1, right knee angle θ2, and rightelbow angle θ3 calculated from the two-dimensional skeletal
data obtained at the "motion break" and "release" during the
successful shooting. Data from NBA players were also
included for reference. Δθ1, Δθ2, and Δθ3 in Table 5-3

177

International Workshop on Informatics ( IWIN2023 )the "Basketball Instruction Manual"[9] published by theJapan Basketball Association. This "Basketball Instruction
Manual" describes shooting in basketball as follows: "Thearea that the ball can pass through the hoop depends on theangle of incidence of the ball in relation to the hoop. If theangle of incidence is large, the ball will enter the hoop moreeasily, but the loop needs to be higher, and the distance willbe less. If the angle of incidence is too small, the area
through which the ball can pass through the hoop is reduced,
making it unsuitable. To obtain the ideal loop and distance,
it is best to shoot at a projection angle of 45 to 50 degrees.”
In other words, this "Basketball Instruction Manual" [8]states that the angle of throw of the ball at the release shouldbe 45° to 50°, which is considered to be the ideal angle.
Therefore, if the ball is thrown from a right armpit angle of
90° in the "motion break," the right armpit angle θ1 at therelease is considered to be 135° to 140°. This figure is
supported by the fact that the right armpit angle θ1 at therelease for the NBA players shown in Table 5-4 is 140°.

Figure5-4:Comparison of shooting motion between
Experienced Player 1 and Inexperienced Player 3.
Table5-4 Comparison of θ1, θ2 and θ3 between experienced
and inexperienced players in failed shooting.

7. SUMMARY
In this paper, we used OpenPose to perform a twodimensional posture analysis of the basketball shooting formof experienced and inexperienced players from the free
throw line. This time, a posture analysis was performed
using videos taken in slow motion. As a result, it was
confirmed that experienced players were characterized by
the armpit angle θ1, which is within around 90° at the
motion break and around 135° at the release.
Also, for inexperienced players trying to efficiently acquire
the shooting form of experienced players, it is good to beaware of the armpit angles θ1, knee angle θ2 and elbow
angle θ3 and of shooting with a good coordination between
the upper and lower body forces. In addition, when the twodimensional skeletal information posture was evaluated withand without slow motion, it was confirmed that the slowmotion shooting particularly improves the accuracy of the
2D Joint Position information at the time of release.
Based on the findings of this research, we would like toprove through demonstration experiments that when
inexperienced players are taught effective practice methods,
their form becomes similar to that of experienced players,
and that their shooting rate increases.

5.4 Comparative evaluation of experienced andinexperienced shooters in failed shoots
Next, Table 5-4 shows a comparison of the experienced
and inexperienced players at the time of shooting failure.
Table 5-4 shows the results of calculating the right armpit
angle θ1, right knee angle θ2, and right elbow angle θ3
based on the 2D Joint Position information at the "motion
break" and "release.” The right armpit angle θ1 was not
90±5° for both experienced and inexperienced shooterswhen they failed to shoot, and the number of frames inwhich the right armpit angle θ1 was 90±5° during thetransition from the "motion break" to the "release" was zero.
Compared with Table 5-3, which shows the results ofexperienced and inexperienced players at the time ofsuccessful shooting, it is considered that for successful
shooting from the free throw line, the right armpit angle θ1should be around 90° at the "motion break" and the numberof frames during which the right armpit angle θ1 reaches
90±5° until the "release" should be at least one. The rightelbow angle θ3 is also noteworthy. In the difference ∆θ3during the transition from the "motion break" to the"release", Experienced Player 1 had a value of 104° for the
successful shooting and 70° for the unsuccessful shooting.
Therefore, in shooting from the three-throw line, the amountof movement of the right arm during the transition period
from the "motion break" to the "release" is also considered
to be important.

REFERENCES[1] Zhe Cao，Tomas Simon， Shih-EnWei， Yaser Sheikh，"Realtime Multi-Person 2D Pose Estimation using Part
Affinity Fields" ， <https://arxivorg/abs/1611.08050>,(Referred:2022-04-29).
[2] N. Takuma， Y. Ikuko， H. Hitoshi，" Classificationof soccer skills using OpenPose" ， The 34th Annual
Conference of the Japanese Society for Artificial
Intelligence, 2020, Vol. 3M5-GS-12-05, pp. 1-4, 2020．
[3] Y. Iida, S. Uchino," Verification of accuracy of marker
type joint angle measurement during free throw-likemovement using KINECT v2 sensor ", Vol. 4, The Japan
Journal of Basketball Studies，pp. 55-63, (2018).
[4] Kira Yusei, "Study of the Correlation between Successof Volleyball Spike and Exercise Ability using Skeletal
Estimation"，Vol. ISS-SP-029, p. 188, (2020).

6. CONSIDERATION
Chapter 6 discusses efficient shooting from the free throw
line, based on the findings of this study, with reference to178

International Workshop on Informatics ( IWIN2023 )[5] M. Nakai,Y. Tsunoda, H. Hayashi, H. Murakoshi,"Prediction of Basketball Free Throw Shooting by
OpenPose", New Frontiers in Artificial Intelligence,
JSAI-is AI 2018 Workshops, (2018). <https://doi.org/
10.1007/978-3-030-316051_31>,(Referred:2022-0720).
[6] Basketball Medical Support Lab ， ” Japan national
basketball team! Shot analysis of #8 Rui Hachimura!”,<https://www.baske-medical-support.com>,(Referred:2022-04-29).
[7] Shota Ishigaki, Keiichi Abe," Comparative of Experienced and Inexperienced People by Shoot Form
Analysis from Free Throws of Basketball using 2D Joint
Position information " IPSJ Symposium series
DICOMO Multimedia, Distributed, Cooperative, andMobile Symposium, (DICOMO 2022), 8E-3, pp. 16591663, (2022).
[8] NBA all-time free-throw success rate rankings The
number one free-throw success rate all-time in a season
is Steph Curry (as of 29 October 2020 locally) ，<https://ikeple.com/nba-historical-records007/>,(Referred:2023-02).
[9] Japan Basketball Association, "Basketball Instructional
Manual”, pp. 63-65, (2006).

179

International Workshop on Informatics ( IWIN2023 )181

International Workshop on Informatics ( IWIN2023 )182

International Workshop on Informatics ( IWIN2023 )183

International Workshop on Informatics ( IWIN2023 )184

International Workshop on Informatics ( IWIN2023 )185

International Workshop on Informatics ( IWIN2023 )186

International Workshop on Informatics ( IWIN2023 )187

International Workshop on Informatics ( IWIN2023 )“I’m Going Right” or “Please Go to the Right”?: Disambiguation in Arrow Display
on Mobile Robots to Avoid Collision with Passersby1
Yo Kuwamiya† , Taketo Imagawa‡ , and Minoru Kobayashi‡†

Graduate School of Advanced Mathematical Sciences, Meiji University, Japan‡
Faculty of Interdisciplinary Mathematical Sciences, Meiji University, Japan{yo.kuwamiya, taketo.imagawa}@koblab.org, minoru@acm.org

Abstract - In recent years, mobile robots that actively interact with humans have started to be used within human living
areas. When the robot and pedestrians pass each other, theymight collide because it is difficult for a pedestrian to predictthe robot’s movement from its appearance. We are considering a method of indicating the robot’s direction of movement
by a horizontal arrow which is displayed on a flat display attached to the robot’s body. However, this approach has the
problem that pedestrians may interpret the displayed arrow as“indicating the direction passing pedestrians should go”. To
avoid misinterpretation, we are researching what kind of arrows to use and how to display them to convey the directionof movement with an arrow. In this research, we investigated
the recognition of a horizontal arrow presented by a mobile
robot. At the same time, we varied the timing and position of
the display by using a subjective video in which the cameraman passed the mobile robot. The results showed that an arrow presented by the robot just before passing by pedestrians
tended to be interpreted as “indicating the direction passing
pedestrians should go”.

Figure 1: Image of the proposed method. The robot indicate its direction of movement using a horizontal arrow. Thismethod has the problem that the interpretation of the arrowvaries depending on individuals.

using arrows to indicate the robot’s movement [1]–[4]. In
these methods, an arrow pointing in the robot’s direction of
movement is displayed on the top of the robot [1] or projectedon the floor in front of the robot [2]–[4].

Keywords: Arrow, Mobile Robot, Direction of Movement,
Nonverbal Communication, Metaphorical Design

1 INTRODUCTION

We are proposing a method of indicating the robot’s direction of movement using a horizontal arrow which is shown ona flat display attached to the robot’s body to avoid collisions
when a pedestrian and mobile robot pass each other. Figure.1
shows an image of the proposed method. This method has the
problem that the interpretation of the arrow varies depending on individuals. If a pedestrian passes by a mobile robotthat displays a horizontal right arrow as in Figure.1, the direction to avoid differs from person to person depending on how
they interpret the arrow. Those who avoid the left side of therobot interpret the arrow as indicating the robot’s directionof movement . Conversely, those who avoid the right sideof the robot interpret the arrow as indicating the direction
passing pedestrians should go .

In recent years, mobile robots that actively interact withhumans have started to be used within human living areas tosolve labor shortages in industrial fields and to reduce humanto-human contact in order to prevent the spread of infectious
diseases. Such robots are expected to move without causing
harm or anxiety to those around them. However, it is difficult
for a human to predict the robot’s movement from its appearance. Therefore, even if the mobile robot recognizes the people around it for movement, the people around it do not recognize it and become fearful of the robot’s movements. Whenpedestrians pass by a mobile robot, they might collide withor feel fear from the robot because it is difficult for a pedestrian to predict which direction the robot will go. Therefore,
mobile robots must visually indicate the intended direction of
movement.
Robots currently under development vary widely depending on their use and design. Hence, it is promising to use arrows, a socially widespread sign, to communicate the robot’s
direction of movement. Some studies have proposed a method

In this research, we investigated how the interpretation of ahorizontal arrow changes depending on when and where therobot displays the arrow when a mobile robot and a pedestrianpass each other. Based on the findings of this research, we
aim to develop a method that communicates the direction of
the robot’s movement without misunderstanding. This paper
presents the results of the research and the outlook for the
future of the research.

1 This work was supported by JSPS KAKENHI Grant Numbers
JP18K11410,JP22H03635.

189

International Workshop on Informatics ( IWIN2023 )2 RELATED WORK: ACTION
ANNOUNCEMENTS OF MOBILEROBOTS

3 METHOD USING A HORIZONTAL
ARROW DISPLAYED ON THE BODY
We are proposing a method of indicating the robot’s direction of movement by a horizontal arrow which is displayed ona flat display attached to the robot’s body. The advantages ofthe method are as follows.
Easy to implement ― The use of displays to present an arrow is feasible without the need for large equipment or complex systems.
No restriction on arrow display position ― In related studies using an arrow [1]–[4], the authors proposed a method ofusing the top surface of the robot and the floor surface as arrow display locations. In contrast, the proposed method candisplay an arrow on the robot’s body. This feature is free fromrestrictions on the location of output devices and arrow display positions.
High visibility ― The proposed method has higher visibility than the method that displays the arrow pointing forward [1]–[4]. Therefore, people in a wide area around therobot can be made aware of the robot’s movements.

2.1 Method using non-verbal communication
When people pass each other, they use non-verbal communication such as line of sight [5] and head direction [6]to communicate the direction they will go. There are somerelated studies that proposed a motion announcement that applies such a communication method [7]–[9]. Matsumaru etal. [7] proposed a method of indicating the robot’s movement
by displaying eyeballs on an Omni-directional Magicball(R)display attached to the top of the robot’s head and transmitting
the direction of movement by the position of the displayed
eyeballs and the speed by the degree of eye-opening. Similarly, in [8], [9], a method has been proposed in which a faceis mounted on the upper part of a robot and the direction of
travel is transmitted by the direction of the face.
Equipping robots with facial and eye expression devices
has the effect of improving the emotional affinity with therobot. This technique is particularly used when the robot
moves autonomously and interacts with people. However, it isdifficult to apply such a method to all robots. This is because
the shapes and uses of robots vary greatly from one individual to another. To give an example, the mobile telepresence
robot, Beam [10] (see Figure.1), moves freely and interacts
with people around it. It displays the user’s face, which is
captured from the user’s device, on the screen at the top of
the robot. People who interact with the robot recognize theuser’s face as the robot’s face. Therefore, displaying humanlike expressions such as eyes or a face in addition to the user’s
face may confuse humans interacting with the robot.

4 RESEARCH SUBJECT
4.1 Perception of the horizontal arrow
presented by the robot
To communicate the robot’s direction of movement by anarrow, it is necessary to indicate clearly what the arrow represents. In related studies using the arrow [1]–[4], it points
forward so that the pedestrian understands it indicates the direction of travel of the robot. On the other hand, in our proposed method it is not clear whether the arrow indicates the
direction of travel of the robot or a passing pedestrian. This
uncertainty may cause a collision between the robot and thepedestrian.

2.2 Method using arrows
Some studies have proposed using arrows to announce therobot’s direction of movement [1]–[4]. Matsumaru et al. proposed a method of displaying an arrow pointing in the direction of travel on a flat panel display attached to the top of therobot [1] and a method of projecting an arrow on the floor in
front of the robot [2]. Methods of projecting an arrow pointing in the robot’s direction of travel on the floor have been
proposed in several further studies [3], [4].
Arrows make it easier for pedestrians to understand therobot’s direction of movement. In addition, a method using
arrows can be applied to a variety of robots with large individual differences in shape, usage, and movement because it
is not a communication method based on social rules or common sense such as blinkers [11]. There are people who do
not understand the rules for blinkers, and if they are put on
by a robot, it may not be thought of as a blinker. In contrast with the spoken communication method [12], an arrowcan communicate the direction of movement to people who
speak different languages because an arrow is a globally used
sign. Furthermore, it is possible to convey additional information such as the speed of movement by flexibly changing
the thickness and length of the arrows.

4.2 Consideration of research subject
To convey the direction of travel of the robot, it is necessary
to display the arrow in such a way that the pedestrian caneasily interpret the arrow as “indicating the direction in which
the robot will move”. Therefore, we need to know how people
perceive the arrow when it is attached to the mobile robot.
However, there are no known studies that have investigatedthe perception of arrows attached to moving objects, because
they are usually attached to stationary objects.
In this research, we investigate the recognition of an arrow attached to a mobile robot from the viewpoints of display
timing and display position. Based on the findings of this research, we aim to realize a method of displaying the arrowthat resolves the research problem.

4.3 Research contents
In this research, we first investigated the difference in interpretation when the robot presented an arrow at a close distance and a far distance from a passing pedestrian. Second,
we similarly investigated the interpretation of the arrow when

190

International Workshop on Informatics ( IWIN2023 )it was presented in nine positions, multiplied by three different heights (top, center, and bottom of the robot) and three different left-right positions (left, center, and right of the robot).
This paper describes the details and results of the first experiment in Chapter 5, and the second experiment in Chapter
6.

5 INVESTIGATION OF PERCEPTION BY
THE TIMING OF DISPLAY
This chapter describes the details and results of the experiment that investigated the perception of a horizontal arrow
presented by a mobile robot in terms of the timing of the presentation. We experimented by creating a subjective video inwhich the cameraman passed the mobile robot.

Figure 2: Dimensions of the robot, display, and the arrow
used in the experiment.

5.1 Experiment details
We conducted an experiment in which the participants watcheda video of the robot moving straight ahead and responded toquestions asking in which direction participants intended to
go to avoid the robot.
Participants. 20 persons (14 males and six females).
Experimental environment. Participants responded to a form
created using Microsoft Forms [13] on their PCs.
Robot. We used a mobile telepresence robot, Beam [10]. By
operating the robot from a PC, users of this robot can moveit freely and communicate with people around the robot. Although the robot was originally designed to display the user’s
face on the screen at the top of the robot, we hid this withblack paper to eliminate the possibility that the face and theline of sight of the user might affect the participant’s judgmentof the direction of movement. We equipped the robot with an
LED panel display to present an arrow. The display was an
11-pixel high and 72-pixel wide LED screen controlled by
M5Stack [14], manufactured by M5Stack Technology Co.,
Ltd. Figure.2 shows the dimensions of the robot, the display,
and the arrow.

Figure 3: The captured images of the experimental video.

Figure 4: Length and width of the hallway and compositionof the experimental video.

Experimental video. Figure.3 shows captured images of theexperimental video. Figure.4 shows the length and width of
the hallway and the composition of the experimental video.
The hallway width was 1.8 m, and the length was 8.5 m. The
robot and the cameraman moved straight along the centerline
at the same speed from both ends as soon as the video started.
The robot displayed an arrow at a point between 2.5 m and6.5 m (1.0 m intervals) from the cameraman. The speed of
the robot was approximately 0.5 m/sec. Therefore, the point
at which the arrow was displayed was from one second to fiveseconds before the video ended. The video finished when the
distance between the robot and the cameraman was 1.5 m. We
informed the participants that the robot would start avoiding
action at the end of the video.

Instructional contents. We gave the participants the following instructions to explain the situation in the experimental
video.
• You will pass a mobile robot.
• When the robot gets close to a certain distance fromyou, it will show a horizontal arrow. (There are also
videos without displaying an arrow.)• The arrow may be pointing in the direction you shouldgo, or it may be pointing in the direction the robot will
go .

191

International Workshop on Informatics ( IWIN2023 )• After the video ends, the robot passes you.
Experimental procedure. We experimented as follows.
1. Participants watched the experimental videos. The videos
were presented in a random order for each participant.
2. Participants were asked to choose which direction theywanted to go to avoid the robot from five options: {left,
likely left, I cannot judge, likely right, right}.
3. Each participant repeated this process for a total of threesets of 12 videos (36 trials per participant). One set
consisted of five videos with right-pointing arrows at
each timing, five videos with left-pointing arrows at thesame timing, and two videos with no arrow displayed.
The order of the videos in each set was randomized. After each set, we asked participants to respond with theiropinions and impressions in the free-response section.
Evaluation method. We weighted the five choices sequentially by {-2, -1, 0, 1, 2}, and calculated the mean and percentage of responses at each presentation timing.

5.2 Results
Figure 6: Percentage of responses by the timing of presentation. (Right Arrow)

Figure.5 shows the average of the responses, and Figure.6,7
shows the percentages of responses by timing. When the
robot presented the right arrow, the mean values of the responses at 2.5 m and 3.5 m, which were presented at close
distances, were 1.0 and 0.93, indicating a tendency for more
respondents to avoid the right side. Similarly, when the robot
presented the left arrow, the values of the responses were 0.73, and -0.80, indicating that more participants tended to
avoid the left side. On the other hand, when the robot presented the arrows at a far distance of 6.5 m, the value wasclose to zero. As can be seen from Figure.6,7, the reason for
the value being close to zero was not because most participants chose “cannot judge”, but because the number of participants who chose “left” and “right” was about the same.

Figure 5: Mean value of responses by the timing of presentation.

Figure 7: Percentage of responses by the timing of presentation. (Left Arrow)

192

International Workshop on Informatics ( IWIN2023 )5.3 Consideration
In this section, we discuss the results of the experiment.
When the arrows were displayed at a distance closer than
4.5 m from the cameraman, many participants responded thatthey intended to go in the direction the arrows were pointing.
This indicates that many participants recognized the arrows as“indicating the direction passing pedestrians should go” when
the arrows are displayed just before they pass each other. Possible reasons for this result are the usual use of arrows and aproblem with space on one side of the hallway used for theexperimental video.
First, the arrows that people usually see when walking areused to indicate the direction to a specific place. Therefore,
we consider that if the robot displays an arrow just before
passing a pedestrian, it is likely to be reflexively judged asindicating the direction passing pedestrian should go . On
the other hand, if the arrows are presented from a far distance,
many people begin to recognize the possibility that the arrow
indicates the direction of travel of the robot. This is because
they have more time to judge the meaning of the arrow. As
a result, the number of people who interpreted the arrows as
indicating the direction of the robot increased to be about thesame as the number of people who interpreted it as indicating
the direction passing pedestrians should go.
Second, the hallway used in this experiment has a space
on one side, as shown in Figure.3. In addition, the videos
in which the robot presented the left arrow were created byreversing the videos in which the robot presented the right arrow, so in all the videos there was a space in the direction thearrow was pointing. In this experiment, participants watched
the two videos in which no arrows were presented and there
was a tendency to avoid the direction in which the space was
located in both videos (Figure.6,7 no arrows). Furthermore,
we obtained the comment, “Because the hallway was asymmetrical, I chose the direction with more space” in the freeresponse section.

Figure 8: The height positions of the LED panel display.

Figure 9: The display position of the arrow.
Experimental video. Figure.10 shows the captured imagesof the experimental video. To eliminate the problem with thespace of a hallway in the first experiment, we shot the videos
at a place where the spaces on both sides of the robot wereof the same size. Figure.11 shows the length of the path traveled by the robot and cameraman and the composition of theexperimental video. The robot and cameraman move straight
along the centerline at the same speed from both ends as soon
as the video starts. The speed of the robot is about 0.5 m/sec.
The robot presents an arrow when the distance from the cameraman reaches 3.5m. We determined the distance regarding the maximum social distance which is proposed by E. T.
Hall [15], because a participant in the first experiment commented, “It is hard to see the arrow when it is presented froma far distance, and I don’t think it is presented toward me”.
The robot displayed an arrow at one of nine positions, multiplied by three different heights (top, center, and bottom of therobot) and three different left-right positions (left, center, andright of the robot). As in the first experiment, the video ended
when the cameraman and the robot approached 1.5 m. We
informed the participant that the robot would begin to avoid
them at the end of the video. The instructional content explaining the situation in the experimental video was the sameas in the first experiment.

6 INVESTIGATION OF PERCEPTION BYDISPLAY POSITION
This chapter describes the details and results of the experiment that investigate the perception of a horizontal arrow presented by a mobile robot in terms of its presentation position,using the same method as the first experiment.

6.1 Experiment details
Participants. 20 persons (17 males and three females).
Robot. As in the first experiment, we used Beam [10]. An
LED panel display can be attached to the robot at three different height positions, as shown in Figure.8. These positions
are located at the top, center, and bottom of the robot. The
LED panel display can display a right arrow on the left where
the arrow tip is in the middle, in the center where the arrow
center is in the middle, and on the right where the arrow starts
in the middle. Equally, the left arrow is displayed at the left,
center, and right of the display. Figure.9 shows the display
position of the arrow.

193

International Workshop on Informatics ( IWIN2023 )Table 1: Mean value of responses by presentation position.
(Right Arrow)

UpCenter
Bottom

Left
0.733
0.067
0.450

Center
0.417
0.750
0.500

Right
0.533
0.883
0.667

Figure 10: The captured images of the experimental video.
Table 2: Mean value of responses by presentation position.
(Left Arrow)

UpCenter
BottomLeft-0.733-0.783-0.583

Center-0.333-0.650-0.683Right-0.6330.000-0.450Figure 11: The length of the path traveled by the robot andcameraman and the composition of the experimental video.
Experimental procedure. The experimental procedure was
the same as in the first experiment. We presented a total of 21
videos. These videos included nine videos with right arrows,
nine videos with left arrows, and three videos with no arrows.
The participants repeated the experiment for a total of threesets (63 trials per person).
Evaluation method. The evaluation method was the same asin the first experiment. We asked the participants to state thedirection they intended to avoid, choosing from five options{left, relatively left, I cannot judge, relatively right, right},
and then we calculated the mean and percentage of the weighted
values in order {-2, -1, 0, 1, 2}.

6.2 Results
Table.1,2 show the mean values. Figure.12,13 shows thepercentage of responses by display position. The mean of the
responses was above zero for all positions in which the rightarrow was presented (Table.1), and many participants tended
to avoid the right side of the robot. The same was true for theleft arrow, which was below zero in all positions (Table2), andmany respondents tended to avoid the left side of the robot.
We obtained the comment from the free-response section that“it was difficult to understand when the right arrow was presented to the left and the left arrow to the right”. When the
robot displayed right arrows to the left and left arrows to theright, a few participants chose “I cannot judge”, and a few participants answered “right” or “left” with certainty. We foundthat the display which placed the tip of the arrow in the middleof the robot made the interpretation difficult.

Figure 12: Percentage of responses by the position of the presentation. (Right Arrow)

194

International Workshop on Informatics ( IWIN2023 )In the proposed method using arrows [1]–[4], an arrow
pointing forward was displayed when going straight ahead.
On the other hand, we were considering a method of displaying a horizontal arrow just before the robot passes a pedestrian. We believe that this difference is the reason why the
robot could not transmit the direction of movement in the experiment. If a passerby can recognize that the arrow indicates the robot’s direction of movement when it is moving ina straight line, they can also make judgments without confusion when it turns. We considered this analogy to be crucial
in establishing the meaning of the arrow. Therefore, the robot
should display the arrow not only when turning but also whenmoving straight ahead to make the surrounding people awarethat the robot is using the arrow to indicate the direction of
travel.
In our proposed method, however, the robot presents thearrow on a display attached to the body, so the arrow pointing forward cannot be displayed. Therefore, we will address
how to display an arrow when moving straight ahead on a flat
display in future research.

8 CONTINUOUS ARROW EXPRESSION
In this chapter, we consider two methods of continuous arrow presentation to indicate that the robot is moving forward
and turning right or left.
Figure 13: Percentage of responses by the position of the presentation. (Left Arrow)

8.1 Down arrow
The first method entails using use the down arrow to indicate that the robot is moving forward. Down arrows are
sometimes used on information boards to represent arrows
extending toward the rear of pedestrians. Therefore, when
the robot displays a down arrow, a passing pedestrian may
recognize it as indicating the direction in which the robot ismoving. Based on this possibility, we consider a method inwhich the down arrow indicates the robot’s forward motion,and the arrow rotates to match the angle of the turning motion.
Figure.14 shows an image of the method.

6.3 Consideration
In this section, we discuss the results of the experiments.
The results of the experiment showed that participants tended
to avoid the direction indicated by the arrow no matter where
it was displayed. In the first experiment, we considered thatwhen deciding the direction in which they intended to go, alarge number of participants might be influenced by the space
on one side of the hallway. However, in the second experiment, many participants also avoided the direction of the arrow. We prepared three videos without arrows, and the meanvalues of the responses to these videos were -0.1, 0.0, and0.1, confirming that there was no effect of the experimental
video environment. These results indicate that when the robot
presented the arrow just before passing by a pedestrian, thearrow was often perceived as “indicating the direction passing pedestrians should go” even when the problems with thespace of a hallway were eliminated.

7 CONTINUITY OF ARROW DISPLAY
From the two experiments conducted in this research, we
found that an arrow presented by the robot just before passing by pedestrians tended to be interpreted as indicating thedirection passing pedestrians should go , regardless of thearrow’s display position. Regarding this knowledge, we consider that a mobile robot should always display an arrow toconvey the direction of its movement because the arrow presented just before passing by a pedestrian tended to be misinterpreted.

Figure 14: Image of the method using use the down arrow
to indicate that the robot is moving forward. The method inwhich the down arrow indicates the robot’s forward motion,and the arrow rotates to match the angle of the turning motion.

195

International Workshop on Informatics ( IWIN2023 )8.2 Double-headed arrow

to be interpreted as “indicating the direction passing pedestrians should go”, regardless of their display position. Based on
this knowledge, we believe the robot needs to always display
an arrow to indicate its direction of movement. We considered methods of continuous arrow presentation using a downarrow and a double-headed arrow.

The second method entails using a double-headed arrow toindicate the robot’s motion. When the robot is moving forward, it displays a double-headed arrow where both sides are
the same length, and when the robot is turning, it displays adouble-headed arrow where one side extends in the directionin which the robot will go. Figure.15 shows an image of themethod.
This method was inspired by bicycle hand signals. Bicycles
do not have signals like blinkers, so cyclists indicate which
direction they will take by extending their hands in the direction they are going. Although hand signals are socially known
rules, the indication method is easy to understand visually.
We consider that a double-headed arrow can also indicate thedirection of the robot’s movement in the same way as handsignals. In addition, the extending movement of the arrowcan express the angle of the turn.

REFERENCES[1] Matsumaru. T, Mobile Robot with Preliminaryannouncement and Indication Function of Forthcoming
Operation using a Flat-panel Display, 2007 IEEE Int.
Conf. on Robotics and Automation (ICRA’07), ThA4.4,
pp.1774-1781 (2007).
[2] Matsumaru. T, Mobile Robot with Preliminaryannouncement and Display Function of Forthcoming
Motion using Projection Equipment, ROMAN 2006 The 15th IEEE International Symposium on Robot andHuman Interactive Communication, pp.443-450 (2006).
[3] Coovert. M. D, Lee. T, Shindev. I and Sun. Y, Spatial augmented reality as a method for a mobile robot
to communicate intended movement, Computers in Human Behavior, Vol.34, pp.241-248, (2014).
[4] Shrestha. M. C, Onishi. T, Kobayashi. A, Kamezaki.
M and Sugano. S, Communicating Directional Intent
in Robot Navigation using Projection Indicators, 2018
27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp.746751 (2018).
[5] Nummenmaa. L, Hyönä. J and Hietanen. JK, I’ll WalkThis Way: Eyes Reveal The Direction of Locomotionand Make Passersby Look and Go the Other way, Psychol Sci., vol.20, no.12, pp.1454-1458, (2009).
[6] Ueda. S and Kitazaki. M, Collision Avoidance Affectedby Walker’s Head Direction in a Virtual Environment,
HCI International 2013 - Posters’ Extended Abstracts,
CCIS 374, C. Stephanidis, Ed. Springer, Berlin, Heidelberg, pp.727–731 (2013).
[7] Matsumaru. T, Iwase. K, Akiyama. K, Kusada. T and
Ito. T, Mobile Robot with Eyeball Expression as thePreliminary-Announcement and Display of the Robot’s
Following Motion, Autonomous Robots, Vol.18, No.2,
pp.231-246 (2005).
[8] Yamashita. S, Ikeda. T, Shinozawa. K and Iwaki. S,
Evaluation of Robots that Signals a Pedestrian Using
Face Orientation Based on Moving Trajectory Analysis,
2019 28th IEEE International Conference on Robot andHuman Interactive Communication (RO-MAN), pp. 1-8(2019).
[9] Lyu. J, Mikawa. M, Fujisawa. M and Hiiragi. M, Mobile
Robot with Previous Announcement of Upcoming Operation Using Face Interface, 2019 IEEE/SICE International Symposium on System Integration (SII), pp.782787 (2019).
[10] Beam, https://gobe.blue-ocean-robotics.com/beam-togobe accessed 11 June 2023
[11] Shrestha. M. C et al, Exploring the use of light anddisplay indicators for communicating directional intent,

Figure 15: Image of the method using a double-headed arrow.
The method of communicating the direction of movement byusing an arrow extending in the direction in which the robot
turns.

8.3 Future perspective
We will verify whether these methods can convey the direction of movement of a mobile robot in an experiment in whicha pedestrian passes by the robot. In the experiment, we willcompare methods with no indication and a horizontal arrowthat the robot presents just before passing by a participant,and evaluate these methods by observing the participants’ behavior and evaluating impressions of the robot.

9 CONCLUSION
We are studying a method to avoid collisions when a mobile robot and a pedestrian pass each other by indicating therobot’s direction of movement with a horizontal arrow attached to the robot’s body. In this research, we investigatedthe perception of arrows presented by a mobile robot in termsof the timing and position of arrow presentation through anexperiment using a subjective video of passing by a robot. As
a result of the two experiments, we found that an arrow presented by the robot just before passing by pedestrians tended196

International Workshop on Informatics ( IWIN2023 )2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), pp. 1651-1656 (2016).
[12] Muramatsu. S, Higashi. S, Chugo. D, Yokota. S andHashimoto. H, Consideration of the preliminary announcement function for the human friendly service
robot, IECON 2016 - 42nd Annual Conference of
the IEEE Industrial Electronics Society, pp. 5868-5872(2016).
[13] Microsoft Forms,
https://www.microsoft.com/enus/microsoft-365/online-surveys-polls-quizzes accessed
11 June 2023
[14] M5Stack, https://m5stack.com/ accessed 11 June 2023
[15] E. T. Hall, The Hidden Dimension, Doubleday, NewYork (1969).

197

International Workshop on Informatics ( IWIN2023 )A Broadcast-based Online Machine Learning System
for Environmental Value Prediction
Tomoki Yoshihisa*,**, Rikuto Kotouge***

Department of Data Science, Shiga University, Japan
Graduate School of Engineering Science, Osaka University, Japan
yoshihisa@biwako.shiga-u.ac.jp**

Abstract -Various online machine learning systems have

Data Sources
Temperature

been studied recently. In ordinary online machine learning
systems, machine learning servers cyclically collect training
data from data sources such as connected vehicles, smart
phones, etc. By collecting only the data that contribute toimprove the performances of the machine learning models,
the times needed for the data collections in each cycle are
reduced without the performance degradations. However, it
is difficult for the data sources to judge whether their data
contribute to the improvements or not because they do nothave the latest learned models. Hence, in this paper, we
propose a broadcast-based online machine learning system.
In our proposed system, the server broadcasts the latest
model to the data sources. Each data source sends its
obtained data to the server if the data can contribute to theperformance improvement with a high probability.

Broadcast Server

PM2.5

Traffic
Traffic(Machine Learning)
PollenTraining
Data

Suspicious Person

Air Purity

ML ModelTraffic
Weather

Traffic

Traffic

Internet

Figure 1: A brief image of the broadcast-based online
machine learning system
be sufficiently trained with less data because there will be alarger machine learning loss. As the loss increases, theaccuracy of the model decreases. Therefore, selecting suchdata as the system collects training data is brilliant. In thisapproach, to reduce the number of training data to becollected, it is necessary to compute the result of the loss
function before the system collects training data. That is,each data source must calculate the results of its training
data. This calculation requires the parameters of the machinelearning model.
Therefore, in this paper, we propose an online machinelearning system that uses data broadcasting to reduce thedelay in reflecting observed environmental values in amachine learning model 1 . In the proposed system, theparameters of the machine learning model are periodicallybroadcasted to the data source, and if the data source
calculates the loss and the result is greater than a given
threshold, the training data is sent for machine learning.
With the above approach, the proposed system reduces timeby reducing the data size collected. The main novelties ofthis report are as follows: 1) an online machine learning
system using data broadcasting, 2) a method for selecting
highly efficient training data, 3) experimental evaluation of
the proposed system.
The rest of the paper is organized as follows. Section 2
presents related works on the proposed system. Section 3
describes an online machine learning system using databroadcasting. Section 4 expounds on the proposed training
data selection method, and Section 5 evaluates the proposedsystem.

Keywords: federated learning, neural networks, distributed
computing.

1

Temperature

INTRODUCTION

Various artificial intelligence (AI)-based information
systems have recently attracted much attention. AI
technologies have various functions and are used in various
fields. Many of these artificial intelligence technologies use
machine learning. Machine learning is a technology forlearning decision criteria from data. The most critical aspectof machine learning is prediction accuracy, which is
impacted by the type and size of training data used to train
machine learning models. Two learning frameworks areused in machine learning: online learning and batch learning.
In online learning, the machine learning models arecontinuously trained each time data is obtained. Therefore,
these models are constantly considering the latest situations.
The advantage of the online machine learning system is thatthe model can be trained while collecting training data, so
the model is always reflecting the latest situation. The
learning server in the above example can provide more upto-date data than the offline machine learning system used inbatch learning. Therefore, online machine learning systems
are often used when the latest situation changes frequently.
However, the online machine learning model is trained intime intervals after collecting a specific data size. If thesetime intervals are long, the prediction accuracy of online
machine learning models too can be negatively impacted.
Although these time intervals can be reduced by reducing
the data size required for the next training, the model cannot

2

1

RELATED WORKThe work reported in the paper was supported by a Grants-in-Aid
for Scientific Research (B) numbered 21H03429, 22H03587,
22H03700, (C) numbered 18K11316, 20K11829, 22K12009.

199

International Workshop on Informatics ( IWIN2023 )2.1

Data Collection in Sensor Networks

RepeatBroadcast Server

Various data collection techniques in sensor networks
have been studied in the past decade.
Among them, a data collection technique has been
proposed that performs aggregation such as averaging andsumming [1,2]. In this technique, each sensor device
aggregates its observed values and the values received fromother sensor devices. Aggregation also reduces the amountof data to be collected and enables faster data collection. In
the system proposed in [3], when the user sends aprobabilistic query, the system collects sensor data based on
the order determined by the probabilistic model of sensor
values. The system can reduce the time required toaggregate sensor data by creating an efficient aggregation
plan based on the submitted query. However, thesetechniques aggregate sensor data, so the sink cannot collectevery raw data.
Several data collection techniques have been proposed toreduce data collection latency in sensor networks. In asystem called Ken, the base station generates a probabilistic
model of the sensor network and collects only model update
information from the physical sensor network [4]. In thissystem, even though the collected values areprobabilistically correct, the data collection latency is
reduced using a probabilistic model.
A practical probabilistic model for health care systems has
been proposed in [5]. The article [6] proposes a query
processing technique to reduce the waiting time. The authors
consider the predicates of the submitted query and construct
the order in which queries are executed to reduce thewaiting time to collect related sensor data. However, theapplication of these techniques is not online machinelearning, and they do not account for machine learning loss.

2.2

Sensor Device

Deliver thelatestparameters andthe thresholds

Update the modelby training it using
the received data
and modify thethresholdCalculate the loss
values using its
observed data andthe received
parameters

If the loss value
exceeds thethreshold, send
the data to theserverTime

Figure 2: The procedure of the assumed system
ActiveClean, proposed in [15], is a data cleaning systemtargeted at machine learning. The system trains a machinelearning model using the original and cleaned data, checks
the accuracy of both models, and cleans the data so as not todegrade the accuracy. However, ActiveClean does notconsider the time required to collect training data, which is
the objective of this report.

3 BROADCAST-BASEDONLINE MACHINE LEARNING SYSTEM
In this chapter, we describe our proposed broadcast-based
online machine learning system.

3.1

SummaryThis study aims to rapidly reduce the loss of online
machine learning by effectively selecting the training data to
be collected. We propose an online machine learning system
to achieve this goal in a data broadcasting environment.
A brief image of the system is shown in Figure 1. In our
proposed system, we set an upper bound on the number ofdata sources that can send training data to a computer for
online machine learning (broadcast server). The upper limit
is set according to the communication bandwidth and thelearning time. To keep the number of data sources close to
the upper limit, the broadcast server adjusts the loss
threshold and broadcasts the data.
As explained in Section 1, in systems that optimize
machine learning models using gradient descent, training
data with a large impact on the loss have large gradients,
and the loss decreases significantly after learning. Therefore,
training data with large losses are preferentially transmitted
in our proposed system. The weight parameters of themachine learning model are necessary to calculate the lossof each data source.
Moreover, the proposed system uses data broadcasting to
distribute these weight parameters to all data sources in ashort time. The broadcasting server receives the training
data sent from the data source and starts training themachine learning model when it finishes collecting thetraining data. When the training is complete and the weightparameters are updated, the updated weight parameters arebroadcasted together with the threshold values, and thesubsequent training data collection begins.

Communication Traffic Reduction

Several techniques have been proposed to reduce thecommunication traffic for machine learning. In some cases,
the data size sent from the data source can be reduced byconverting the data types to those required for machinelearning. Such transformations are performed on the data
source and are referred to as fog/edge computing [7-10].
The article [10] proposes a machine learning-based sensor
data analysis system using the fog computing paradigm.
This system converts data sources into data suitable forfeed-forward neural networks, reducing the data size. The
literature confirms that the proposed system can reduce thedata size sent from the data source in practical situations. In
addition, the loss of accuracy of the neural network in thissystem is small, but this system does not consider the loss intraining the machine learning model.
Reduction of communication traffic can also be achievedby data cleaning. Data cleaning is a technique to remove
dirty data such as incomplete or inaccurate data fromobserved data. The Sample-and-Clean proposed in [11] andProgressive Data Cleaning proposed in [12-15] can reducethe computational cost of data cleaning. These studies
indicate that the losses on these systems may increase
because the target application is not machine learning.

3.2
Assumed Data Broadcasting
Environment

200

International Workshop on Informatics ( IWIN2023 )Recently, with the development of wireless technology,
the data broadcasting environment has attracted muchattention. Two data distribution channels exist in a databroadcasting environment: a broadcasting channel and acommunication channel. The image is shown in Figure 1.
The broadcasting server is equipped with broadcasting
equipment to broadcast data to the source. The broadcasting
devices are assumed to broadcast data using wirelesscommunication protocols such as WiFi, ISDBT, andBluetooth. In this case, data sources located in the area
where radio waves for wireless communication can reachcan receive the broadcasted data. A data source that is notencountering communication errors can receive broadcast
data. The broadcasting server is connected to the Internet
and can receive the data sent from the data sources. A
broadcasting server can be a computer for training machinelearning models in a data broadcasting environment formachine learning.
Data sources are devices with various sensors such as
smartphones, smart cars, and sensors installed in buildings.
These devices can continuously acquire sensor data andtransmit them to other computers via the Internet. The
broadcasting server can also receive data broadcast frombroadcasting equipment.

3.3

be prepared by the system side. Therefore, data broadcastingis effective for sharing data for machine learning.

3.5

Consider a broadcast-based online machine learning
system for predicting rain at a random location. The
procedure of the assumed system is shown in Figure 2. The
broadcasting server and the data source are connected via
the Internet.
The data source is a sensor terminal with sensors mounted
on various IoT devices such as Raspberry Pi. These sensor
terminals are equipped with 5G communication modules
connected to the Internet by wireless communication
protocols.
The broadcasting server broadcasts data to the data source
over the 5G network using an IP address. We construct theneural network by setting the period, the loss threshold, andthe upper threshold of the data. Upon receiving the weightparameters of the model, the data source checks whether theloss exceeds the threshold value. If the threshold is exceeded,
the data (latitude, longitude, and rain) is sent to thebroadcast server.
The broadcast server starts training the online machinelearning model with the received training data. If thenumber of training data is more than the upper limit, thethreshold value for the next broadcast is increased to reducethe data size. Otherwise, if the data size is small, thethreshold is reduced, and the data size is increased. Whenthe training is completed, the broadcasting server broadcasts
the machine learning model's updated weight parameters
and the data source's threshold values.

Advantages and Disadvantages

The most significant advantage of data broadcasting is thatthe broadcasting server can simultaneously distribute thesame data to all data sources. When the same data isdelivered over a communication channel, the broadcasting
server sends the data to each data source sequentially(unicast). In the case of sequential transmission, thecommunication time is proportional to the number of data
sources, and the more data sources there are, the longer ittakes. Therefore, the time required for the delivery of thebroadcast channel is shorter than that for the communication
channel.
One of the disadvantages of data broadcasting is that boththe broadcasting server and the data source needs to manage
the channel. Therefore, the required computational resources
and power consumption increase. However, this is not asignificant disadvantage because wireless communication
modules in recent desktop computers and smartphones canreceive broadcast and unicast data.
For example, a typical WiFi module can receive data sent
to a broadcast IP address and data sent to a unicast IP
address.

3.4

Application Example

4

PROPOSED METHODThis chapter describes the details of the proposed datacollection method.

4.1

Summary

The proposed broadcast-based online machine learning
system aims to improve accuracy by preferentiallycollecting data with large errors rapidly. Therefore, we
select the data with the largest error among all the data.
If we know the errors of all the data and then train the datawith large errors, we can optimally train the data with large
errors. However, the proposed system limits the number ofdata sources transmitting data due to the limitation ofcommunication bandwidth and learning time, so thebroadcasting server cannot know the error of all data.
Therefore, the data source needs to judge whether theobserved data has a large error without the error informationof other data sources. The broadcast server gives thethreshold value for this decision.
By transmitting data if the error is larger than the thresholdvalue, the data source can determine whether the observed
data has a large error without error information from otherdata sources. If this threshold is too small, the data size to betransmitted will be too large, and the upper limit will be
exceeded. Meanwhile, if the size is too small, less data willbe transmitted, and the accuracy will decrease due to theApplication to Machine Learning

As described above, the advantage of data broadcasting isthat it reduces the time required for the broadcasting server
to deliver the same data to all data sources. In other words,
the data sources can share the data in a short time. Machinelearning data sources do not need data because they only
send training data to train machine learning models.
However, the proposed system shares the parameter data ofthe training model. Data for machine learning include
training data, model data, and parameter data, which need to201

International Workshop on Informatics ( IWIN2023 )decrease in the training data; to prevent this, the broadcast
server dynamically adjusts the threshold value.

4.2

PhasesIn online machine learning, the training of the machinelearning model is repeated every time the computer obtains
training data. This iterative period is the learning interval in
our proposed broadcast-based online machine learning
system, as shown in % Figure 2.
A learning interval consists of the following two stages.
We assume that the time managed by the broadcasting
server and the data source are synchronized. Timesynchronization can be easily achieved using the Network
Time Protocol (NTP).
⚫ Broadcasting Phase: The broadcasting phase is when the
broadcasting server trains the machine learning modeland updates the weight parameters and thresholds. The
broadcasting server broadcasts the data sources' weightparameters and threshold values. In order for thebroadcasting server to learn all the collected data, thetime of the broadcasting phase should be longer than thetime needed for learning.
⚫ Collection Phase: Each data source observes
environmental values and generates training data. Upon
receiving the model weight parameters, the data source
calculates the loss and then sends it to the broadcast
server if it is greater than a threshold value. In our
proposed system, the data sources with large losses
preferentially send training data to reduce learning loss.

4.3

Figure 3: The road map for the evaluation
when broadcasted from the broadcast server. This process isrepeated periodically.

4.5

The time of the broadcasting server and the data source aresynchronized. The broadcasting server broadcasts theparameters and threshold values of the learning model every
specific time according to a predefined cycle. Therefore, it isnecessary to complete the learning process before thebroadcast. If enough bandwidth is available, this learning
time determines the number of stations that can receive thebroadcast. The broadcasting server broadcasts the data
without training if the training is not completed by the next
cycle. The collection phase begins when the data source
receives the model weight parameters and threshold values.
When the data is received in the collection phase, it moves
to the broadcasting phase, and the broadcasting server learns
the data.

5

Broadcast Server Behavior

EVALUATIONThe evaluation results of the proposed system arepresented in this section. This evaluation presents the resultsof the evaluation of the proposed system under the rain
scenario.

The broadcast server receives the training data sent fromthe data source. If the received training data is larger thanthe upper limit, the threshold value for the next broadcast isincreased to reduce the data size. However, if it is small, thethreshold is reduced, and the data size is increased.
When it is time to move to the broadcast phase, thebroadcast server starts training the online machine learningmodel. Since the broadcasting server and the data sources
are synchronized in time, the start of the broadcasting phasecan be recognized.
When the training is finished, the broadcasting server
broadcasts the machine learning model's updated weightparameters and the data sources' threshold values. Then thenext learning interval is started.

4.4

Time Synchronization

5.1

Configuration

In this experiment, we assumed a system of the section 3
and performed simulations. We considered a connected car
equipped with sensors for rain or temperature always
connected to a 5G network.
The broadcasting server and the vehicle are connected to
the Internet and can communicate.
The data source and the broadcasting server share theonline machine learning model and start learning.
The car driving simulation was performed using SUMO,an open-source traffic flow simulator. The map used in thesimulation is shown in Figure 3.
The figure shows a 1000 m square area for the simulation.
Cars were randomly selected from one endpoint for
departure and arrival points, respectively, and were allowed
to run for 1000 [s] with one car per second inflow. When the
car arrives at an intersection, it chooses its course with equal
probability and proceeds. There are always approximately
100 cars running on the map.

Data Source Behavior

Each data source always observes the environmental
values. Upon receiving the initial values of the model
weight parameters and threshold values from the broadcast
server, the collection phase begins, and the errors arecalculated using the model weight parameters and theobserved values.
The data is sent to the broadcast server if the calculated
error exceeds the threshold. The updated values are received
from the model weight parameters and threshold values

202

International Workshop on Informatics ( IWIN2023 )1

The online machine learning model is LSTM, a recurrent
neural network suitable for predicting time series data. The
input data of this model is continuous location information,and the output data is environmental data such as rain,assuming the example described in the subsection 3.5. The
input data is continuous two-dimensional data, and theoutput is one-dimensional data. Each data source has all the
input and output data (training data) of the neural networkand computes the loss using its observed training data. In
this experiment, the number of hidden layers is 2, and thenumber of neurons in each hidden layer is 64. The uniform
random algorithm gives the initial values of the weightparameters of the neural network. The activation function ofeach layer is assumed to be a hyperbolic tangent. The
gradient descent algorithm used to optimize the weightparameters is RMSProp.
Since it took approximately 10 [s] to train 1000 epochs ofthe neural network model on the author's computing
environment, we assume that the time required to train one
epoch is 10 [ms]. In order to systematically evaluate thesystem, the observed rain by a vehicle located at (x, y) attime t is given by 1/(1+exp(-0.05(x+y+t-2000))). Value forrain ranges from 0 to 1.
In this experiment, the locations on the map shown in
Figure 3 are clear at the initial stage (t =0), The upper right(1000, 1000) indicates that rain clouds are approaching, andrain begins to fall in the area covered by the clouds. The
clouds move to the lower left (0, 0), and the rain changes
from 0 to 1 at the point of cloud cover. Therefore, at theinitial stage (t =0), rain at all locations is 0, and at the end of
the simulation (t =1000), rain is 1 because all regions haverain.
The broadcast period is 10 [s], and the model's updated
weight parameters and threshold values are broadcasted tothe sensor terminals every 10 [s]. The initial value of thethreshold is 0.1 and is adjusted every 10 [s].
If the transmitted data size is more than the upper limit, thethreshold is increased by a factor of 1.1 to reduce the size ofthe subsequently transmitted data; if the number is less thanthe upper limit, the threshold is increased by a factor of 0.9
to increase the size of the subsequently transmitted data.
To evaluate the accuracy of the proposed method, which
selects and collects lossy data, we compared the results ofthe proposed method with those of the randomly selected
data. In the proposed method, data with an extensive root
square error between the predicted and measured values are
preferentially collected as data with a significant loss.

5.2

0.9

提案手法
Proposed Method

ランダム
Random600

800

Prediction Error

0.8
0.7

0.6
0.5
0.4
0.3
0.2
0.1
0
0

100

200

300

400

500

700

900

Elapsed Time [sec]Figure 4: The elapsed time and the prediction error(rain scenario)
0.6

提案手法Method ランダム
ProposedRandomPrediction Error

0.55

0.5

0.45

0.4

0.35
350

400

450

500

550

600

650

Elapsed Time [sec]Figure 5: The moving average for 100 [s] of the prediction
error (rain scenario)
prediction error becomes very large at approximately 400 [s].
However, the proposed method can keep the prediction error
small. This is because the proposed method improves theaccuracy at high speed by preferentially collecting data withsignificant losses. At the end of the simulation, it is raining
at all locations, the proposed method and random selection
have minimal prediction errors and good accuracy.
Because of the large variation of the forecast error in
Figure 4, the vertical axis is changed to the moving averageof the forecast error for 100 [s], and a graph showing theperiod from 350 [s] to 550 [s], when the variation of theforecast error of the proposed method and random selection
is large, is shown in Figure 5.
As shown in Figure 5, the prediction error of the proposed
method is smaller at approximately 400 [s]. However, atapproximately 500 [s], the prediction error in random
selection is smaller than that of the proposed method, andthe accuracy of the proposed method is lower than that ofrandom selection. This is because while the proposed
method can easily reflect changes, it predicts too large achange. All data in this experiment were trained with large
losses, but this can probably be improved by partiallytraining data at random.
The experimental evaluation results will show that the
proposed method can reduce the prediction error byapproximately 10% compared to random selection. In thisexperiment, the proposed method can improve theprediction accuracy even when sudden rain is assumed.

Results

A graph of the results of the proposed method and random
selection is shown in Figure 4. The vertical axis indicatesthe mean value of the error between the predicted rain andthe measured rain (prediction error) for all vehicles, and thehorizontal axis indicates the time t [s]. Since no rain falls at
the beginning of the simulation and rain is 0 at all locations,the prediction errors of both the proposed method andrandom selection are minimal, indicating good accuracy. As
the rainfall gradually changes from 0 to 1, the prediction
error becomes larger, and the accuracy decreases. The

6203

CONCLUSION

International Workshop on Informatics ( IWIN2023 )In this study, we proposed an online machine learning
system under a data broadcasting environment to reduce thedelay in reflecting observed environmental values into themachine learning model. In the proposed system, the databroadcasting server periodically broadcasts the model
weight parameters and threshold values, and only the data
sources of training data whose loss is larger than thecollection threshold sends training data to the databroadcasting server. This method reduces the number ofdata to be collected and latency.
Future issues include adjusting the threshold value andevaluating the proposed method on complex maps. We
could not evaluate the method by adjusting the thresholdvalue in the present experimental evaluation.

[11] J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T.
Kraska, and T. Milo: A Sample-and-Clean Framework
for Fast and Accurate Query Processing on Dirty Data,in Proc. ACM SIGMOD International Conference on
Management of Data, pp. 469–480 (2014).
[12] Y. Altowim, D. V. Kalashnikov, and S. Mehrotra:
Progressive Approach to Relational Entity Resolution,
the VLDB Endowment, vol. 7, no. 11, pp. 999–1010(2014).
[13] T. Papenbrock, A. Heise, and F. Naumann:
Progressive Duplicate Detection, IEEE Transactions on
Knowledge and Data Engineering, vol. 27, no. 5, pp.
1316–1329 (2015).
[14] M. Yakout, A. K. Elmagarmid, J. Neville, M.
Ouzzani, and I. F. Ilyas: Guided data repair, arXiv
preprint arXiv:1103.3103 (2011).
[15] S. Krishnan, J. Wang, E. Wu, M. Franklin, and K.
Goldberg: Activeclean: Interactive Data Cleaning for
Statistical Modeling, the VLDB Endowment, vol. 9, pp.
948–959 (2016).

REFERENCES[1] Y. Yin, L. Long, and X. Deng: Dynamic Data Miningof Sensor Data, IEEE Access, vol. 8, pp. 41637–41648(2020).
[2] J. Considine, F. Li, G. Kollios, and J. Byers:
Approximate Aggregation Techniques for Sensor
Databases, in Proc. IEEE International Conference on
Data Engineering (ICDE), pp. 449–460 (2004).
[3] A. Deshpande, C. Guestrin, S. R. Madden, J. M.
Hellerstein, and W. Hong: Model-driven Data
Acquisition in Sensor Networks, in Proc. International
Conference on Very Large Databases, volume 30, pp.
588–599 (2004).
[4] D. Chu, A. Deshpande, J. M. Hellerstein, and W.
Hong: Approximate Data Collection in Sensor
Networks using Probabilistic Models, in Proc.
International Conference on Data Engineering (ICDE),
pp. 48–59 (2006).
[5] N. Nikzad, J. Yang, P. Zappi, T. S. Rosing, and D.
Krishnaswamy: Model-driven Adaptive WirelessSensing for Environmental Healthcare Feedback
Systems, in Proc. IEEE International Conference on
Communications (ICC), pp. 3439–3444 (2012).
[6] A. Deshpande, C. Guestrin, W. Hong, and S. Madden:
Exploiting Correlated Attributes in Acquisitional
Query Processing, in Proc. IEEE International
Conference on Data Engineering (ICDE), pp. 143–154(2005).
[7] E. I. Gaura, J. Brusey, M. Allen, R. Wilkins, D.
Goldsmith, and R. Rednic: Edge Mining the Internet ofThings, IEEE Sensors, vol. 13, no. 10, pp. 3816–3825(2013).
[8] A. Jain, E. Y. Chang, and Y.-F. Wang: Adaptive
Stream Resource Management using Kalman Flters, in
Proc. ACM SIGMOD International Conference on
Management of Data, pp. 11–22 (2004).
[9] H. M. Raafat and A. Tolba: Homoscedasticity for
Defect Detection in Homogeneous Fat Surface
Products, Textile Research Journal, vol. 85, no. 8, pp.
850–866 (2015).
[10] H. M. Raafat, M. S. Hossain, E. Essa, S. Elmougy, A.
S. Tolba, G. Muhammad, and A. Ghoneim: FogIntelligence for Real-Time IoT Sensor Data Analytics,
IEEE Access, vol. 5, pp. 24062–24069 (2017).

204

Session 5:
Systems and Applications 2(Chair: Kei Hiroi)

International Workshop on Informatics ( IWIN2023 )207

International Workshop on Informatics ( IWIN2023 )208

International Workshop on Informatics ( IWIN2023 )209

International Workshop on Informatics ( IWIN2023 )210

International Workshop on Informatics ( IWIN2023 )211

International Workshop on Informatics ( IWIN2023 )212

International Workshop on Informatics ( IWIN2023 )213

International Workshop on Informatics ( IWIN2023 )A Spherical POV Heatmap using Mixed Reality
in 360-degree Internet Live Broadcasting
Yoshia Saito* and Junichiro Suto**

Graduate School of Software and Information Science, Iwate Prefectural University, Japany-saito@iwate-pu.ac.jp

Abstract - 360-degree internet live broadcasting enables

displays a spherical heatmap of the viewers’ POV on top of
the omnidirectional camera in real space using the MRtechnology through an MR headset. Since the broadcastercan check the MR spherical POV heatmap in real space, it isexpected that the response time to viewer comments can beshortened.
The contributions of this paper are summarized asfollows:

viewers to change their point of view (POV) while watching
the 360-degree live video and has an issue that thebroadcaster cannot check the viewers’ POV. To solve theissue, we have proposed a spherical POV heatmap using
Augmented Reality (AR) on a smartphone so that thebroadcaster can be aware of the viewers’ POV. Although thespherical POV heatmap reduced communication errors, theresponse time increased due to the need to check theheatmap on a smartphone. In this paper, we propose a new
spherical POV heatmap using Mixed Reality (MR) throughan MR headset. The proposed system displays the heatmap
in real space through the MR headset and reduces the
response time by eliminating the need to check thesmartphone. We implemented a prototype system and foundit could reduce the response time through the evaluation.

Keywords:

360-degree Internet
Viewers’ POV, Mixed Reality.

1

Live



We developed and evaluated a prototype system of the
MR spherical POV heatmap using HoloLens 2.

We clarified that the MR spherical POV heatmap
enabled the broadcaster to reduce the response time toviewer comments.

The rest of this paper is organized as follows. Section 2
describes the AR spherical POV heatmap in the 360-degreeinternet live broadcasting as our previous work. Section 3
describes an overview of our proposed system. Section 4
describes the implementation of the prototype system.
Section 5 describes an evaluation experiment to clarify theeffects of the proposed system. Section 6 summarizes thisstudy.

Broadcasting,

INTRODUCTION

360-degree video frees viewers from the constraints ofviewing direction. The viewers can change their point ofview (POV) and enjoy all directions of the video. Recently,
360-degree video has also been introduced in internet live
broadcasting, where the viewers can typically communicate
with the broadcaster via text chat. However, there is an issue
that the broadcaster cannot check the viewers’ POV in the360-degree internet live broadcasting. Since the broadcasterdoes not know what the viewers saw and commented on, it
can cause communication errors between the broadcasterand the viewers.
To solve the issue, we have proposed a spherical POV
heatmap using Augmented Reality (AR) technology [1-2](hereinafter called AR spherical POV heatmap). The AR
spherical POV heatmap displays a spherical heatmap of theviewers’ POV on a QR code using AR technology through asmartphone screen. The broadcaster checks the AR spherical
POV heatmap and can know which direction the viewers arewatching. Although it can reduce communication errors
between the broadcaster and the viewers, it increases the
response time to viewer comments due to the need to checkthe heatmap on the smartphone. The response time should
be short to realize smooth communication between the
broadcaster and viewers.
To overcome the disadvantage of the AR spherical POV
heatmap, we propose a new spherical POV heatmap using
Mixed Reality (MR) technology (hereinafter called MR
spherical POV heatmap). The MR spherical POV heatmap

2

AR SPHERICAL POV HEATMAP

The AR spherical POV heatmap shows a sphere that
represents the broadcasting space on an AR marker througha smartphone as shown in Figure 1. The spherical heatmap
visualizes the viewers’ POV by displaying the angularcoordinate vector on the spherical surface as a heatmap. By
synchronizing the direction of the image taken by theomnidirectional camera with the vector of the sphericalheatmap, the broadcaster can grasp the viewers’ POV in real
space simply by checking the heatmap. In addition, theheatmap is easy to visualize multiple data at the same time.
It is also possible to check the POV of multiple viewers. By
checking the density pattern of the viewers’ POV on theheatmap, the broadcaster can grasp where many of theviewers are interested, and where the other viewers of theminority are watching. Therefore, it gives some hints about
the viewers’ interests and achieves smooth communication
between the broadcaster and viewers.
In the evaluation experiments, the AR spherical POV
heatmap could reduce communication errors by visualizing
the viewers’ POV and the speaking timing is easier for thebroadcaster to grasp when waiting for the viewers’ attention.
On the other hand, the time it takes for the broadcaster to
respond to the viewer’s comment was increased by215

International Workshop on Informatics ( IWIN2023 )Omnidirectional
Camera360-degree
live video

360-degree Internet
Client for360-degree live broadcasting server 360-degree
Broadcasterlive video
live video
Node.js(WEB browser)

Broadcaster

Kurento Media
Server

UI forBroadcasterMR spherical POV heatmap,
Comments

MR device (HoloLens 2)

Client for Viewers（WEB browser）

POV,
Comments

POV/Comment
ServerMR
Application

POV andComment
Send/Recv
Function

POV,
Comments

UI forViewers
Viewers

Comments

Figure 3: System architecture of the prototype system
broadcasting system and an MR spherical POV heatmap
function. The broadcaster sends a 360-degree live video tothe proposed system and the viewers watch the live video.
The viewers send comments to the proposed system, which
are sent to the broadcaster and the viewers. The proposedsystem receives viewers’ POV in real-time and makes an
MR spherical POV heatmap. The MR spherical POV
heatmap is provided to an MR headset of the broadcaster.
The proposed system is expected to have several
advantages by reducing the burden on the broadcaster asfollows: (1) the time it takes for the broadcaster to respondto the viewer’s comment is expected to be shorter than theprevious study with few communication errors, (2) the
speaking timing is easier for the broadcaster to grasp thanthe previous study when waiting for the viewers’ attention.
The advantages of the previous study will be further
enhanced, and the disadvantage is solved by the proposedsystem.

Figure 1: AR spherical POV heatmap throughsmartphone display
approximately 10 seconds compared with the broadcasting
without the AR spherical POV heatmap. This result was dueto the time required to check the heatmap through asmartphone. If the wasted time can be eliminated, theeffectiveness of the spherical POV heatmap can be
improved.

3

4

IMPLEMENTATION

We implemented a prototype system of the MR spherical
POV heatmap using HoloLens 2. In this section, we describe
the architecture of the prototype system and its main
application.

PROPOSED SYSTEM

In our previous study [1-2], the issue was that checking theviewers’ POV was a burden for the broadcaster. This isbecause the spherical POV heatmap was implemented using
AR and forced broadcasters to check their smartphone to see
the heatmap. To solve this issue, we propose a system that
displays the spherical POV heatmap using MR technology
through an MR headset. The proposed system is expected toeliminate the need to confirm the POV heatmap using asmartphone in the previous study. The use of a headmounted type MR device also solves the issue of both hands
being occupied and reduces the burden on the broadcaster in
360-degree internet live broadcasting.
Figure 2 shows a system model of the proposed system.
The proposed system consists of a 360-degree internet live

4.1

System Architecture

The prototype system was implemented by replacing the
AR spherical POV heatmap in the previous study with an
MR application for the MR spherical POV heatmap.
Figure 3 shows the system architecture of the prototype
system. The red square in the figure shows new
implementation in this study and the other parts are diverted
from the previous study. In the prototype system, we used
Microsoft HoloLens 2 as an MR headset for the broadcasterand THETA V as an omnidirectional camera.
A broadcaster can start 360-degree internet live
broadcasting using the broadcaster client on a web browser.

216

International Workshop on Informatics ( IWIN2023 )web browsers and the broadcaster can read the comments on
the HoloLens 2. The viewer client also automatically sends
the viewer’s POV to the POV server every second. The
POV is represented by spherical coordinates (r, θ, φ) wherer denotes the radial distance, θ denotes the polar angle, andφ denotes the azimuthal angle. The POV server forwards
the received viewers’ POV to the MR application on the
HoloLens 2.

4.2

MR Spherical POV Heatmap

The MR application on the HoloLens 2 is implemented by
Unity and it shows a spherical POV heatmap and comments
to the broadcaster. The MR application receives the viewers’
POV and displays their heatmap on a sphere object which isshown above the omnidirectional camera. Based on a list of
the viewers’ POV received within a certain period, it createscircles for the heatmap. In this circle, the closer to the center,the higher the heat value, and the farther from the center, thelower the heat value. If a circle overlaps another circle, theheat values of the two circles are added and the overlapping
area has a higher heat value.
Figure 4 shows an example of the heatmap when multiple
viewers are watching the 360-degree live video fromdifferent POV. The broadcaster can see that the viewers'
interests are dispersed in different directions. Figure 5 shows
an example of the heatmap when several viewers’ POV areconcentrated. The broadcaster can see that several viewers
are interested in a particular direction of the size of the red
area.
Figure 6 shows the user interface for the broadcasterthrough the HoloLens 2. The MR application gets acoordinate and direction of the omnidirectional camera, andit displays the MR spherical POV heatmap which is fixed
above the omnidirectional camera. The comment window
tracks the broadcaster’s sight so that the broadcaster canread the viewers’ comments. The MR spherical POV
heatmap is updated every second. The broadcasters can walkfreely around the room wearing hololens2.

Figure 4: The heatmap when multiple viewers
are watching from different POV

Figure 5: The heatmap when several viewers
are watching in a particular POV

5

EVALUATION

We conducted an evaluation experiment using theprototype system compared with the AR spherical POV
heatmap. In this section, we describe the evaluation
environment and the results.

5.1

Environment

The experiment was conducted under the same conditions
as in the previous study [2]. The purpose of the experiment
was to confirm whether the MR spherical POV heatmap
reduced the burden on the broadcaster compared to the AR
spherical POV heatmap.
Each experiment was conducted with one broadcaster andthree viewers, for a total of four times. The broadcaster andviewers were in different rooms in the experiment. The
equipment used in the experiment was one notebook PC forthe broadcaster, three laptop PCs for the viewers, one PC for
the server, Ricoh Theta V for the omnidirectional camera,and HoloLens 2 for the MR headset.

Figure 6: The user interface for the broadcasterthrough the HoloLens 2
The 360-degree internet live broadcasting server distributes
it to viewers. The viewer can watch the 360-degree live
video using the viewer client on a web browser and send
text comments to the comment server. The comment server
forwards the received comments to the viewers and thebroadcaster. The viewers can read the comments on their

217

International Workshop on Informatics ( IWIN2023 )The content of the broadcast was a chat about objects inthe broadcaster’s room. In the room, there were various
objects. The viewers commented on objects in the
broadcaster’s room freely changing their POV. The
procedure of the experiment is as follows:The MR spherical POV heatmap was not able to reduce thewait time although we expected to be able to reduce the timeby eliminating the need to check the smartphone as with theprevious result of the response time. One possible reason forthis result is that when several heatmap circles weregathered into one heatmap circle, it was difficult todetermine how many viewers were in the heatmap.
Furthermore, the broadcaster was asked to freely describe
what they found advantages and what they founddisadvantages about the prototype system. As an advantage
point, many participants said that "it is easy to visually
understand where the viewer is looking". As a disadvantage
point, many participants said that "when the viewers’ POV
is focused on one place, the heatmap becomes one circle,
and it is difficult to grasp how many viewers are there".
There was also an opinion that "the conversation is delayedby one step because we have to look at the POV heatmap
before speaking". This is because it is necessary to check the
MR spherical POV heatmap above the omnidirectional
camera before finding the target object. To solve thisproblem, instead of displaying the heatmap on a sphere, amethod may be effective in which the heatmap is displayed
directly on the target object.

[Practice] The broadcaster and viewer practice theoperation of the prototype system for 5 minutes.
[Task 1] One of the viewers sends a designatedquestion comment to the broadcaster 4 times at 3minute intervals.
[Task 2] The broadcaster directs the viewer's
attention to a specified object and chats about it 3
times at 2-minute intervals.

In Task 1, the viewer communicates to the broadcaster bysending a question comment about an object in the room,such as "What kind of animal is this stuffed animal?". The
text to be sent, the timing of the comment, and the object towhich the viewer is directed are specified in the procedure
manual. The broadcaster determines which object the viewer
is commenting on and responds to the viewer's comment. If
the broadcaster understands what the viewer is commenting
on, the time it takes for the broadcaster to respond correctly
to the viewer’s comment is expected to be shorter. We
measure the time and define it as the response time. We alsocount the number of communication errors if thebroadcaster makes a mistake with the object the viewers aretalking about.
In Task 2, the broadcaster points to a specified object andinstructs the viewers "Please look at this". The broadcasterstarts chatting with the viewers about the object when their
POV are gathered. If the broadcaster is aware of theviewers’ POV, it is expected to be able to start chatting at
the same time when the viewer's POV is gathered. We
measure the time between the broadcaster's attention
instruction to the viewers and when the broadcaster begins
to speak. It is defined as the wait time.

5.2

6

CONCLUSION

We proposed the MR spherical POV heatmap to reducethe burden on the broadcaster in 360-degree internet live
broadcasting, which displayed a spherical POV heatmap on
the MR space. We compared it with our previous study, theAR spherical POV heatmap to clarify its advantages andissues. The results of evaluation experiments showed that
the MR spherical POV heatmap reduced the response timecompared to the AR spherical POV heatmap, and the objects
could be found quickly. In addition, the use of the MR
headset eliminated the need to check the smartphone andreduced the burden on the broadcaster. On the other hand,
the wait time could not be reduced. One of the reasons forthis result was that it was difficult to grasp how many
viewers were there on the MR spherical POV heatmap. In
future work, we will improve the way the heatmap circles
are displayed so that it is more intuitive for the broadcasterto know approximately how many viewers are on theheatmap. Moreover, we will implement a method in which
the heatmap is displayed directly on the target object.

Results

In Task 1, the mean response time was 21.47 seconds for
the MR spherical POV heatmap, compared to 30.88 seconds
for the AR spherical POV heatmap. In comparison, the MR
spherical POV heatmap reduced the response time by about
9 seconds. In terms of the number of communication errors,
there were 4 communication errors in the MR spherical
POV heatmap, while there was no communication error in
the AR spherical POV heatmap. This is because somebroadcasters selected the target object without checking the
MR spherical POV heatmap at the first time. The number ofcommunication errors could be reduced to zero if checking
the MR spherical POV heatmap.
In Task 2, the mean wait time was 14.78 seconds for the
MR spherical POV heatmap, compared to 16.67 seconds for
the AR spherical POV heatmap. In comparison, the MR
spherical POV heatmap shortened the wait time byapproximately 2 seconds. Meanwhile, we conducted a
Mann-Whitney U test for the results and there was nodifference between the mean wait time (p = 0.20402 > 0.05).

REFERENCES[1] Masaya Takada, Dai Nishioka and Yoshia Saito:
Proposal of a Spherical Heat Map in 360-Degree
Internet Live Broadcasting Using Viewers' POV, The
2nd International Workshop on Mobile Ubiquitous
Systems, Infrastructures, Communications andApplications (MUSICAL2019), pp.596-600 (2019).
[2] Masaya Takada, Dai Nishioka and Yoshia Saito:
Proposal of the Viewer’s “POV Heat Map” in 360degree Internet Live Broadcasting, IPSJ Journal,
Vol.62, No.1, pp. 145–159 (2021) (in Japanese).

218International Workshop on Informatics ( IWIN2023 )Were you speaking to me?:A trial to use physical avatars to establish gaze awareness in hybrid meetings1
Kenta Ohnaka*, Taai Tsukidate*, Yo Kuwamiya*, Kazuyuki Iso**, and Minoru Kobayashi****Frontier Media Science Program, Graduate School of Advanced Mathematical Sciences,
Meiji University, Japan**Tokyo Information Design Professional University, Japan***
Department of Frontier Media Science, School of Interdisciplinary Mathematical Sciences,
Meiji University, Japan{Kenta.Ohnaka, Taai.Tsukidate, Yo.Kuwamiya}@koblab.org
iso@tid.ac.jp, minoru@acm.org
Abstract -Hybrid meetings involving co-located and remote

information includes the atmosphere of the meeting room,
the way participants look at the screen, and the way they
speak to other participants. To this end, we propose ameeting design that involves using physical avatars at themain meeting site as a surrogate for the remote participants,
and conveying a bird’s-eye view video of the main site,
including the avatar, to remote participants. We seek
answers to the following question: Does a bird’s-eye view
video that includes avatars of remote participants facilitatehybrid meetings? This paper describes trials for this purposein two types of environments and discusses the findings,
which suggest that a bird's-eye view video can help theremote participants understand the spatial arrangement of
the main site, and the physical avatar can act as a surrogate
for the remote participant.

participants have been facing problems such as presence
asymmetry between participants at each site, and difficultyin participants understanding what is happening at each site
This is problematic because if the remote participants are
not well represented, the meeting may proceed only with thelocal participants. To overcome these limitations, we
propose a meeting environment that includes a bird’s-eyeview video and physical avatars. This paper provides visual
representations of meeting environments using two types ofphysical avatar, a simple human-like object, and atelepresence robot, kubi, combined with the use of a bird’seye view video. Through the trials we conducted, we gained
an insight that the combination of the physical avatars andthe bird’s-eye view has the potential to help the remoteparticipants understand the spatial arrangement of the main
meeting site including who is being spoken to in the mainsite. The trials also demonstrated that the design of theavatar was important for participants at both sites.

2

Researchers have explored the embodiment of remoteparticipants in various ways in remote meetings, especiallyin hybrid meetings. For example, Kuzuoka et al. [1]proposed a telepresence robot that gestured and pointed toremote participants, and Sakamoto et al. [2] proposed ahumanoid robot (i.e., android) with a realistic presence.
Making a physical embodiment of remote participants inthis way made it possible to improve their experience of themeeting. Sellen et al. [3] proposed a tabletop videoconferencing system to enable cross-referencing between
participants. Equally, today’s remote meetings makeeffective use of telepresence robots such as Beam [4],Double [5], and kubi [6].
The physical embodiment of remote participants makes
both remote and co-located participants feel that thoseattending remotely are physically participating at the mainsite. In addition, telepresence and humanoid robots are
excellent at conveying the status of remote participants to
those at the main site. However, users of these systems areusually limited in their field of view, and it is difficult tograsp the situation at the main site when avatars are used assurrogates.

Keywords: Video Communication, Telepresence, Gaze
Aawareness, Physical Aavatar

1

RELATED WORKSINTRODUCTION

Advancements in technology have allowed remotecommunication on a daily basis. The last few years inparticular have seen remote working promoted around theworld, and we increasingly communicate with friends andcolleagues at a distance.
Decades of research have attempted to address problems intelecommunications. In particular, difficulties in “hybrid
meetings” involving both co-located and remote participants
include the asymmetric presence between participants, andturn-taking issues when participants lack an understandingof what is happening in each site.
In order to solve problems with these types of hybrid
meeting environment, we consider it important to reduce theasymmetry of presence between co-located and remoteparticipants, and to convey information that would generally
be available to participants in face-to-face meetings. Such
1

This work was supported by JSPS KAKENHI Grant Numbers
JP18K11410, JP22H03635.

219

International Workshop on Informatics ( IWIN2023 )The telepresence robot proposed by Jouppi et al. [7] andMorita et al. [8] has a camera with a wide field of view,which makes it easy for remote participants to understandthe situation at the main site. These studies show thatproviding remote users with a wide view facilitatesinteraction, but remote users must operate the robots with acamera to obtain information.
In remote communication using Mixed Reality (MR),which includes the increasingly popular Virtual Reality andAugmented Reality (AR), the use of virtual avatars makes
the presence of remote and co-located participants more
symmetrical.
Piumsomboon et al. proposed a system called Mini-Me[9] and reported that by displaying virtual avatars in an MR
environment and transmitting the gaze and gestures ofremote participants, the presence of remote participants
could be noticeably improved. In the VROOM [10] system
proposed by Brennan et al., a life-sized avatar of the remoteparticipants is presented to the local participants through AR,
and remote users have an immersive 360° view of the mainsite through a Head Mounted Display
In terms of MR, immersive room-sized communication
systems such as t-Room [11], proposed by Hirata et al., andRoom2Room [12], proposed by Pejsa et al., visualize eachparticipant’s non-verbal information, including their gaze,and make participation more symmetrical.
The above examples therefore suggest that using avatars assurrogates for remote participants, whether physical or
virtual, can resolve asymmetry in hybrid meetings.
In face-to-face communication, people obtain verbal andnon-verbal information from others in a natural way. In
particular, the way participants look at each other isimportant in terms of taking turns, because it conveys theintention of an utterance and the object of attention. The
same is true for video or audio-mediated communication,
and a number of studies have helped raise awareness of gazein remote meetings.
GAZE-2 [13], a system proposed by Vertegaal et al.,establishes accurate eye contact between participants by
conveying their gaze, which is measured by an eye-tracker
in a virtual-space video conference. The Mona Lisa effect invideo communication is also regarded as a problem. It iscaused by the combination of a single camera and a 2D
display in the video link, so that participants find it difficult
to understand who is talking to whom, and who is looking at
whom. To resolve this problem, Iso et al. [14] proposed a
3D display to represent the gaze direction of a remote user.
These attempts to establish gaze awareness are approaches
common to systems such as Hydra by Sellen et al. [3], as
well as methods that use multiple viewpoints and sharevision and space remotely, as described below.
A number of researchers have shown how an approachthat transmits the camera images from multiple viewpoints
is also effective for interaction using non-verbal information,and for sharing the atmosphere of a meeting with remoteparticipants.
In addition to the face-to-face view which is usual in videocommunication, Gaver et al. [15] designed multiple views
showing the whole room, and showing materials related tothe task. This enabled the participants to switch freely

between views. They showed that the availability ofmultiple viewpoints was effective in carrying out
collaborative tasks remotely. Referring to this example,
Yamaashi et al. [16] found that it was important to support
peripheral awareness by offering remote participants a fixed
global view. In addition, “ESP” [17], a system designed by
Venolia et al., contains multiple cameras that can be used in
hybrid meetings. Camera images in ESP show co-locatedparticipants, their reactions, and the objects they arereferring to. The authors reported that the ESP camera
helped remote participants understand the context of theconversation and take turns smoothly.
These examples suggest the importance of the views
conveyed between participants in remote meetings, and theway views are designed can help develop successful remotecommunication.
In remote collaboration using MR, more methods haverecently been made available for conveying the situation at
the main site to remote participants using a 360° live image.
Kasahara et al. [18] enabled remote users to view the fullmeeting environment by transmitting panoramic images inreal time from cameras worn by co-located participants. Lee
et al. [19] developed a system to display a virtual handsuperimposed on a panoramic image. This allowed allparticipants to supplement non-verbal communication cuesby visualizing where remote participants were pointing andby seeing other participants’ views. Teo et al. built on thisby [20] proposing a system that combined a 360° view and3D virtual space. They investigated the various views,
suggesting that these helped carry out tasks efficiently andimprove the social presence of participants.
These methods give remote participants a very natural
view of what is happening at the main site and enable
participants to share non-verbal communication cues
between sites. However, as these methods demand high-spec
devices, they may take longer to spread for common use.

3

METHOD: A BIRD’S-EYE VIEW VIDEO& PHYSICAL AVATARSAs described in Chapter 2, many methods have beenstudied to facilitate remote meetings, including methods thatphysically embody remote participants and methods that use
multiple viewpoints. However, there are still difficulties forremote participants with regard to understanding thesituation, what is happening at the main site, and feeling thatthe remote participants are being spoken to by the main site
participants. To solve these two problems, we used a bird'seye view video and a physical avatar in combination to
make a hybrid meeting environment. We designed themeeting environments shown in Figure 1. In thisenvironment, we expect that using a bird’s-eye view video
will help a remote participant to understand the situation at
the main site. Additionally, we set up the physical avatars to
represent the remote participants in the environment withbird’s-eye view video. Since we are still in the process ofconsidering what kind of physical avatar would beappropriate, we have prepared two types of avatar.

220

International Workshop on Informatics ( IWIN2023 )3.1

Bird’s-eye view video

meetings.
We believe that it is important for remote participants to beable to understand that other people are talking to them. The
whole view helps with this as it contains information such as
what the room looks like, the gaze of the participants colocated at the main site, and the location of objects including
avatars. A bird’s-eye view video which covers the whole
site including the remote participant’s avatar enables the
remote participant to see that someone is addressing him. In
addition, it makes it possible to view both the remote andmain site participants simultaneously. We consider that thiseffect allows co-located participants at the main site to carryout the meeting in a natural way, involving avatars in turntaking and feeling that they can be seen.
Methods of transmitting a panoramic view from the center

Today, people are very familiar with picture-in-picturedevices and the face-to-face view offered by video-meeting
services such as Zoom [21] and Microsoft Teams [22]. The
effect has already been discussed by pioneers such as [15][16] [17], but we wish to return to a discussion on the
viewpoint of images transmitted to remote participants.
In a face-to-face meeting, people unconsciously perceive
the situation in the meeting room surrounding them, as wellas peripheral information. The information helps them tosense the atmosphere, understand the context, and take turns
smoothly in conversation. This leads to the idea thatconveying information involving the whole meeting room
could enable remote participants to do these things in hybrid

Figure 1: A hybrid meeting environment that includes a physical avatar of aremote participant and a bird's-eye camera to capture the environment.

Figure 2: Simple human-like objects (left) and theseobjects in a meeting environment (right).

Figure 3: ‘kubi’ as a telepresence robot (left) andkubi within a meeting environment (right).

221

International Workshop on Informatics ( IWIN2023 )of the meeting room, which have become increasingly
available recently, may satisfy these requirements. However,
it is not always easy to understand the gaze direction or who
is looking at whom with a panoramic view image because of
the distortion of the image.

In the trials described below, we observed the influence of
these two kinds of avatar on hybrid meetings when they
were used with a bird’s-eye view camera.

3.2

We tried using the bird`s eye view video and the physicalavatar in a hybrid meeting. In this chapter, we report ourreflections on this trial use.

4

Physical avatar

As research has revealed, the physical embodiment of
remote participants in telecommunications has significant
implications in terms of making all participants feel that
remote participants are present at the main site. In hybrid
meetings in particular, it could help make the presence ofparticipants more symmetric.
An important point here is that it is necessary to make theavatar physical. This approach allows participants to handle
objects in real space and preserves the atmosphere of a real
meeting room. These features make the use of physicalavatars very different from situations where avatars are used
for all participants in a virtual space, or where virtual avatarsof remote participants are shown in a real-space image.
Above all, avatars should exist physically if people are tointeract naturally with each other.
To design the meeting environments, we prepared two
kinds of physical avatar. To represent the remoteparticipants, we first used a simple human-like object(shown in Figure 2), and then kubi [6] as a telepresence
robot (shown in Figure 3). As the human-like object did nothave any input/output functions such as cameras,
microphones, speakers, displays, it acted only as a physicalsurrogate for the remote participant. Telepresence robots,
however, can transmit the voices and images of the remoteusers to each other. A telepresence robot functions as theeyes, ears, mouth and living portrait of the remoteparticipants.

TRIAL USE OF PROPOSED SYSTEM

4.1
Avatar Type 1: A simple human-likeobjectWe first tested a simple human-like object as an avatar of
the remote participant (shown in Figure 4). As mentioned
above, this object had no means of communication and did
not resemble the remote participant. In this trial, it wasdifficult for the main site participant to communicate withthe remote participant, or to feel that this avatar was asurrogate for him. As seen in Figure 4, it was more naturalfor main site participants to speak to the remote partner inthe video on the screen, rather than the physical avatar that
had no visual or audio representation of the remoteparticipant.
On the other hand, the remote participant said that the
bird’s-eye view video made it easy to understand thesituation at the main site. Though it was unnatural for mainsite participants, we experimented with participants in themain site talking to the physical avatar to see what a bird'seye view video of talking to the physical avatar would look
like. When the remote participant saw the main site
participants talking to his physical avatar, the remoteparticipant could temporarily perceive the avatar as if itwere his surrogate.

Figure 4: The image transmitted to the remote participant in the meeting experiment using a simple
human-like object as the physical avatar. Top: the co-located participants gazing at the avatar. Lower
left: the co-located participants referring to the display. Lower right: the co-located participants
talking to each other.
222

International Workshop on Informatics ( IWIN2023 )4.2

the previous trial in which the remote participant’s status
was not presented by the avatar. The phenomenon istherefore considered to have been caused by showing the
remote participant’s live image on the display of the avatar.
The display of the avatar was highly effective fortransmitting the status of the remote participant to the mainsite. Though either the kubi display, the kubi camera, or the
display on the table could be the object that the main site
participants regarded as the remote participant, theparticipants mostly faced the kubi in this trial. It could beassumed that the main site participants paid less attention to
the kubi camera because they knew that the remoteparticipant was mostly referring to the bird’s-eye view video
to see the main site, as the remote participant mentioned in
the conversation. Therefore, it is suggested that the kubi
display, rather than the kubi camera, attracted more attention
from the main participants. This indicates that the main site
participants recognized the kubi display as representing the
remote participants.

Avatar TYPE2: Telepresence robots

We then tested with an environment in which atelepresence robot was used as an avatar of the remoteparticipant along with a bird’s-eye view video (shown in
Figure 5). This avatar was different from the simple humanlike object in that the device itself had a video phonefunction with which the remote participant’s image andhis/her voice were presented on the device, and the remoteparticipant had a first-person view of the main site from thelocation of the device (shown in Figure6). In addition, theremote user could control its orientation. In this way, theremote participant had two views of the situation at the mainsite: a fixed wide view and a controllable narrow view.
In this trial, the participants at the main site looked at theavatar of the remote participants as they talked to him. Thisbehavior suggests that the participants at the main site
instinctively understood that the avatar was the surrogate of
the remote participant. Such behavior was not observed in

Figure 5: The image transmitted to the remote participant in the meeting experiment using atelepresence robot (kubi) as the physical avatar. Top: the co-located participants gazing at the avatar.
Lower left: the co-located participants referring to the display. Lower right: the co-located participants
talking to each other.

Figure 6: First-person image captured by the telepresence
robot’s camera and transmitted to the remote participant.

223

International Workshop on Informatics ( IWIN2023 )The display of the telepresence robot simultaneously
helped the remote participant to understand that it was his
surrogate. The remote participant could confirm visuallythat his surrogate was physically participating in the meetingat the main site by viewing the avatar through the bird’s-eye
view. Furthermore, the remote participant referred to thebird’s-eye view video more often than the first-person view,
remarking that he felt comfortable using the bird’s-eye viewin a hybrid meeting. The differences between two kinds ofviews are shown in Figure 5 with Figure 6.
Incidentally, the remote participants could control theorientation of kubi, but did not do so very much. The reason
for this may be the large latency of the control. There was alarge latency between the remote participant’s control
operation and the actual change in orientation of the kubi.
Therefore, it was difficult to change the orientation of thekubi in response to the change of conversation.

5
5.1

design of the avatar that acted as a good surrogate for the
remote participant, displaying the facial image of the user,
was essential to realize such an environment.

5.2

FUTURE WORKS

In section 5.1, we showed our expectation that the
combination of the bird's-eye view video and the physicalavatar could create a hybrid meeting environment in which
the remote participant easily understands the spatial
arrangement of the meeting space and has the sense of being
spoken to.
We are planning two experiments to further examine theproposed approach. The first experiment is to observe theusage of the bird's-eye view video. In our trials, the remoteparticipant tended to focus on the bird's-eye view video, andthis helped him understand the situation at the main site. To
see if this is the case for everyone, we will use a gazetracking system to record the gaze movement path of theparticipants in hybrid remote meetings. While providing
both the bird's-eye view video and the first-person
perspective video, we will see which video receives more
attention from the participants.
The second experiment is to examine the design of thephysical avatars. The physical avatar needs to be perceived
as a surrogate for the remote participant by both the mainsite participants and the remote site participant. A physicalavatar with a video screen that displays the remoteparticipant's facial image would possibly be a good design.
Or, even with simpler devices, there might be designs thatwork as good surrogates. We will build and compare several
avatars, including a simple human-like object, a telepresence
robot, and a physical avatar with a video screen, to explore
the design principles for building better physical avatars. In
this experiment, again, we plan to use a gaze-tracking
system to evaluate how often and how long the participants
looked at the avatar.

DISCUSSION
Implication

In the hybrid meetings, in order to convey the spatial
arrangement of the main site to the remote participants andhelp them understand the situation at the main site, we set
up a bird's-eye view camera. In addition, the physical avatarof the remote participant was placed at the main site so thatthe remote participant could recognize that he was being
spoken to. We prepared two types of physical avatar, ahuman-like object and a telepresence robot, kubi. We usedeach physical avatar and a bird’s-eye view video in
combination to make a hybrid meeting environment.
Through the trials, the remote participant felt that the
bird’s-eye view video helped them understand the spatial
arrangement of the main site. In the Type 1 trial in which thesimple human-like-object was used as the physical avatar, it
was difficult for the main site participants to treat thephysical avatar as a conversation partner, and, as a result,
the remote participant did not recognize that he was being
spoken to. On the other hand, in the Type 2 trial in which atelepresence robot was used as the physical avatar, theremote participant felt that he was being spoken to.
To properly understand the difference, we need to detail
how the telepresence robot was used in the trial. The remoteparticipant could change its viewpoint, but he rarely did so,
mainly because of the large latency in the control operation.
In addition, the remote participant mostly focused on the
bird’s-eye view video rather than the first-person view videofrom the telepresence robot. Though these things are justwhat we experienced in a short-term trial, these facts imply
that the device functioned more to show the remoteparticipant’s facial image than as a means to enable theremote participant to look at the main site freely bychanging the orientation of the device.
Based on the experiences in the trials, we have come tothink that using the combination of the bird’s-eye view
video and the physical avatar is promising for realizing a
hybrid meeting environment where people easily feel they
are being spoken to by remote partners. We also found that a

6

CONCLUSION

Hybrid meetings involving co-located and remoteparticipants have faced problems such as asymmetry interms of the presence of participants at each site, anddifficulties in participants understanding what is happeningat each site. To overcome this limitation, we used a physicalavatar and a bird’s-eye view video in combination to createa hybrid meeting environment. By conveying a wide view ofthe main site to a remote participant through a bird’s-eyeview camera, we expected to help the remote participants
understand the spatial arrangement at the main site. We also
expected the physical avatar to help the remote participants
feel that they were being spoken to when the remoteparticipants, through the bird’s-eye view video, watched themain site participants speak to the physical avatar.
To examine the possibilities of the approach, we haveconducted a short-term trial of having meetings by ourselvesusing the bird’s-eye view video and the physical avatar.
Through the trials, we have come to think the bird’s-eyeview video has the potential to help the remote participants
understand the spatial arrangement of the main meeting site.

224

International Workshop on Informatics ( IWIN2023 )In addition, the use of a telepresence robot made it easy forthe main site participants to speak to the physical avatar,
which made the remote participant feel as if he was being
spoken to.
We are planning two experiments to further explore (1) theuse of the bird's-eye view video in hybrid remote meetings,
and (2) a design of a physical avatar that acts as a bettersurrogate for the remote participant. There are many things
to do to establish the meeting environment based on the
proposed approach, including finding the best bird's-eye
viewpoint and the best audio environment. Much work has
been done to create remote communication media thatconveys the participants’ gaze direction, including on eyecontact displays and gaze-aware displays [23]. For themultiparty remote meetings, which are popular in these days,
other approach that fit better is necessary. By carefully
designing the environment, we believe we can create ahybrid meeting environment in which we can feel otherparticipants’ gaze without having “eye contact” displays.

[10] B. Jones, Y. Zhang, P. N. Y. Wong, and S. Rintel,
VROOM: Virtual Robot Overlay for Online Meetings,
Extended Abstracts of the 2020 CHI Conference on
Human Factors in Computing Systems (CHI EA2020), pp. 1–10 (2020).
[11] K. Hirata, Y. Harada, T. Takada, S. Aoyagi, Y. Shirai,
N. Yamashita, K. Kaji, J. Yamato, and K. Nakazawa,t-Room: Next Generation Video Communication
System,ProceedingsoftheGlobalTelecommunications Conference, pp. 1-4 (2008).
[12] T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. Wilson,
Room2Room: Enabling Life-Size Telepresence in a
Projected
Augmented
Reality
Environment,
Proceedings of the 19th ACM Conference on
Computer-Supported Cooperative Work & Social
Computing (CSCW 2016), pp. 1716–1725 (2016).
[13] R. Vertegaal, I. Weevers, C. Sohn, and C. Cheung,
GAZE-2: conveying eye contact in group videoconferencing using eye-controlled camera direction,
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI 2003), pp. 521–
528 (2003).
[14] K. Iso, S. Ozawa, M. Date, H. Takada, Y. Andoh, and
N. Matsuura, Video Conference 3D Display That
Fuses Images to Replicate Gaze Direction, Journal of
Display Technology, vol. 8, no. 9, pp. 511-520 (2012).
[15] W. W. Gaver, A. Sellen, C. Heath, and P. Luff, One is
not enough: multiple views in a media space,
Proceedings of the INTERACT '93 and CHI '93
Conference on Human Factors in Computing Systems(CHI 1993), pp. 335–341 (1993).
[16] K. Yamaashi, J. R. Cooperstock, T. Narine, and W.
Buxton, Beating the limitations of camera-monitor
mediated telepresence with extra eyes, Proceedings of
the SIGCHI Conference on Human Factors in
Computing Systems (CHI 1996), pp. 50–57 (1996).
[17] G. Venolia, J. Tang, R. Cervantes, S. Bly, G.
Robertson, B. Lee, and K. Inkpen, Embodied social
proxy: mediating interpersonal connection in hub-andsatellite teams, Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems(CHI 2010), pp. 1049–1058 (2010).
[18] S. Kasahara, and J. Rekimoto, JackIn head: immersive
visual telepresence system with omnidirectional
wearablecamerafor
remotecollaboration,
Proceedings of the 21st ACM Symposium on Virtual
Reality Software and Technology (VRST 2015), pp.
217–225 (2015).
[19] G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, Mixed
reality. collaboration through sharing a live panorama,
SIGGRAPH Asia 2017 Mobile Graphics & Interactive
Applications (SA 2017), Article 14, pp. 1–4 (2017).
[20] T. Teo, L. Lawrence, G. A. Lee, M. Billinghurst, and
M. Adcock, Mixed Reality Remote Collaboration
Combining 360 Video and 3D Reconstruction,
Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems (CHI 2019), Paper 201,
pp. 1–14 (2019).

REFERENCES[1] H. Kuzuoka, J. Kosaka, K. Yamazaki, Y. Suga, A.
Yamazaki, P. Luff, and C. Heath, Mediating dual
ecologies, Proceedings of the 2004 ACM conference on
Computer-Supported Cooperative Work (CSCW 2004),
pp. 477–486 (2004).
[2] D. Sakamoto, T. Kanda, T. Ono, H. Ishiguro, and N.
Hagita, Android as a telecommunication medium with ahuman-like presence, Proceedings of the ACM/IEEE
international conference on Human-Robot Interaction(HRI 2007), pp. 193–200 (2007).
[3] A. Sellen, B. Buxton, and J. Arnott, Using spatial cues
to improve videoconferencing, Proceedings of the
SIGCHI conference on Human Factors in Computing
Systems (CHI 1992), pp. 651–652 (1992).
[4] Beam is moving to GoBe Robots, https://gobe.blueocean-robotics.com/beam-to-gobe,
last
accessed
2023/06/11.
[5] Double Robotics - Telepresence Robot for the Hybrid
Office, https://www.doublerobotics.com/, last accessed
2023/06/11.
[6] kubi, https://kubi-robot.com/, last accessed 2023/06/11.
[7] N. P. Jouppi, First steps towards mutually-immersive
mobile telepresence, Proceedings of the 2002 ACM
conference on Computer-Supported Cooperative Work(CSCW 2002), pp. 354–363 (2002).
[8] T. Morita, K. Mase, Y. Hirano, and S. Kajita,
Reciprocal attentive communication in remote meetingwith a humanoid robot, Proceedings of the 9thInternational Conference on Multimodal Interfaces(ICMI 2007), pp. 228–235 (2007).
[9] T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W.
Lindeman, B. H. Thomas, and M. Billinghurst, MiniMe: An Adaptive Avatar for Mixed Reality Remote
Collaboration, Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (CHI 2018),
Paper 46, pp. 1–13 (2018).

225

International Workshop on Informatics ( IWIN2023 )[21] Video Conferencing, Cloud Phone, Webinars, Chat,
Virtual. Events | Zoom, https://zoom.us/, last accessed
2023/06/11.
[22] Video Conferencing, Meetings, Calling | Microsoft
Teams,
https://www.microsoft.com/en/microsoftteams/group-chat-software, last accessed 2023/06/11.
[23] Hiroshi Ishii, Minoru Kobayashi, and Jonathan
Grudin, Integration of interpersonal space and sharedworkspace: ClearBoard design and experiments,
ACM Transaction on Information Systems, vol.11,
no.4, pp.349–375(1993).

226

International Workshop on Informatics ( IWIN2023 )Exploring Key Concepts in Arts and Mental Health: A Questionnaire-based
Analysis of Perception on Arts, Sports, and Therapy
Michi KOMURA*, Hajime KANEKO**, Midori ISHIHARA***, Akihiro HAYASHI*****

Osaka Metropolitan University, Osaka, Japan
Kobe University of Future Health Sciences, Hyogo, Japan***
ATAS Laboratory, Hyogo, Japan****
Shizuoka Institute of Science and Technology, Shizuoka, Japan{*mitikomm347, **kaneko68h, ***midorippa2021, ****pixysbrain}@gmail.com**

Abstract - The use of arts for healing and empowerment has
spread throughout Japan in recent years, a trend broadly
related to mental health. Our study defines this social
phenomenon as empowerment arts therapy (EAT) or arts
empowerment (AE). In this context, the terms “arts,”“therapy,” or a combination of these words have varying
intentions and implications due to the range of perceptions
these words evoke. A questionnaire survey was conducted onuniversity students to investigate their perceptions of theseterms. The questionnaire results indicated that perceptions of
the arts and therapy varied widely; in some cases, there was asignificant discrepancy between students’ perceptions anddictionary definitions. Specific insights were gathered toestablish a shared comprehension of the terms and concepts
related to arts and therapy, which are often used ambiguously.
EAT and AE practitioners must consider the diversity ofperceptions, effectively communicate their activities, andgain people’s understanding of EAT and AE. The survey
findings provide useful insights into understanding andaddressing the elusive nature of perceptions surrounding artsand therapy.

using art are being developed beyond high and fine art,
focusing on fostering healing and empowerment.
Our research project named Arts Therapy Activities Study(ATAS) explores a range of activities that employ arts
therapy outside of clinical settings. In the research process,we proposed the concepts of EAT and AE.
Arts therapy is originally a general term for psychotherapy
using various artistic techniques, such as painting, pottery,music, theater, and dance. As a form of psychotherapy, it isconducted by psychiatrists or specialized therapists in clinicalsettings, such as psychiatric care and mental counseling, as
well as in parallel disciplines. In our research of ATAS, wedifferentiate the form of arts therapy conducted in clinicalsettings as Psychopathological Arts Therapy (PAT) to
distinguish it from EAT.
In contrast to PAT, EAT is defined as an activity that aims
to rejuvenate the mind and body, alleviate stress, and improve
quality of life (QOL) in everyday settings. EAT is based on
concepts and methods of arts therapy, but unlike PAT, it is
not intended for medical treatment. Furthermore, AE is
defined as an art activity in which the therapeutic nature isinherent in expressive activities, even if they are notintentional.
Both EAT and AE are located at the intersection of art andtherapy, with distinctions based on the emphasis on theartistic or therapeutic aspect. EAT is characterized by astronger therapeutic effect, while AE leans more towardartistic expression. However, some activities fall in between
the disciplines, making it challenging to differentiate them.
Figure 1 shows the positional relationship between art,
therapy, and empowerment in all expressive activities.
Thus, artistic activities, including therapeutic ones, support
individuals’ mental well-being in their everyday lives.
Despite the increasing prevalence of such activities in Japan,
there have been few studies beyond ATAS thatcomprehensively examine various EAT practices, clarify
their unique requirements and significance, and address
associated challenges.
Many of these activities share the common problem ofresource scarcity. Although EAT is spreading at thegrassroots level, the environment surrounding these activities
is insufficient.

Keywords: Empowerment Arts Therapy, Arts Empowerment,
Mental Health.

1

INTRODUCTION

Interest in arts therapy has recently increased as social
awareness of mental health has grown. The therapeutic
effects of art have attracted attention as a tangible approachto improving mental health. Beyond its traditional application
as a form of psychotherapy, various arts therapy practices are
expanding outside clinical settings. Examples include its
implementation in elementary school classes [1], welfare
facilities [2], adult art classes [3], and a local community
atelier for children [4]. These activities apply arts therapy
methods and are carried out in everyday life; however, theydiffer from arts therapy as a form of psychotherapy aimed at
medical treatment within clinical settings. In addition, there
is interest in the intrinsic therapeutic properties inherent in art
and their correlation with well-being [5]. Various activities

227

International Workshop on Informatics ( IWIN2023 )*Marginal Art, Popular Art, Pure Art: Based on Shunsuke Tsurumi’s classification and definition. Marginal art is createdby nonexperts and enjoyed by non-specialists. Pure Art is created by experts and enjoyed by connoisseurs familiar with thefield. Popular Art is created by experts and companies and enjoyed (consumed) by nonexperts (masses) [6].

Figure 1: Positional Relationship between Art, Therapy, and Empowerment in All Expressive Activities in Japan [7]The ATAS has conducted comprehensive research on thecurrent state of EAT and AE using questionnaire surveys,
interviews with practitioners, activity observations, and thedevelopment of assessment methods to enhance the quality ofactivities and practitioner capabilities. The survey revealed
the diverse perceptions of arts and therapy among individuals,
as well as variations in perceptions among different EAT
practitioners.
Given these findings, we conducted a questionnaire survey
to explore perceptions of arts, sports, and therapy. The survey
results were then analyzed to identify key concepts related to
EAT and AE.
Chapter 2 summarizes the issues surrounding the basic
concepts of EAT and AE, outlining the issues to be solved
and the research objectives. Chapter 3 presents an overview
of the survey results. Chapter 4 concludes the study andhighlights future issues.

Existing issues with EAT can be summarized as follows:
1) Sustainability: Most EAT practitioners work voluntarily.
They are working to find and implement training
opportunities, but their activities are not economically
viable. This is a crucial problem but is difficult to solve inthe short term in the present Japanese social system. For now,
it is realistic to increase sustainability within the existing
constraints of assuming a volunteer basis [7].
2) Quality Assurance: Many EAT practitioners work
independently. There are no systematic networks oropportunities for information exchange or training. Thismeans that no standards or indicators have been establishedto ensure the quality of EAT activities or practitioner
capabilities. Self-assessment tools are in development toimprove practitioner capabilities [8-11].
3) Awareness / Understanding: Interest in mental health andawareness of arts therapy have recently increased in Japan.
However, there is still limited public understanding of arts
therapy and the distinction between arts therapy as
psychotherapy and EAT activities in daily life. This lack ofunderstanding makes it difficult for EAT practitioners to
explain their activities to society.

2 ESSENTIALARTHEALTH CONCEPTS

AND

MENTAL

2.1 Ambiguity of Terms
EAT encompasses elements of both arts and therapy. EAT
practitioners often draw upon ideas and methods from PAT,
incorporating them into their activities while adding theirown innovative approaches. Unlike PAT for medical
treatment, EAT activities are practiced to empower people byrevitalizing the mind and body, alleviating stress, andimproving QOL. However, many EAT practitioners lack
precise terms or concepts to convey the nature of theiractivities accurately.
In Japanese, “art(s)” is a concept introduced and translated
from a Western word. The Japanese language has three typesof characters: kanji, hiragana, and katakana. Foreign words
are translated into Japanese or written in katakana—the
Japanese syllabary used for loanwords—based on theirpronunciation in the original language. The word “art(s)” is

The first issue needs long-term and large-scale efforts to besolved. It needs to be addressed with a view to revising andreforming the social system.
Relating to the second issue, we have proposed selfassessment tools for EAT practitioners and are working on
the investigation of effectiveness [8-11].
This study addresses the third issue: the lack of awarenessand understanding of EAT activities in daily life. It is thepreliminary study to approach the issue related to people who
can be potential clients or users of EAT. We have conducted
research and practice mainly related to EAT practitioners so
far but have not done the study related to potential clients of
EAT yet.

228

International Workshop on Informatics ( IWIN2023 )translated as the Japanese term “geijutsu” or written as theloanword “āto” in katakana.
It is common for words and concepts translated from
Western languages to exhibit some differences or gaps inconnotation between the original terms and their Japanesetranslations, whether written in kanji or katakana. There is adistinction between the English “art(s)” and Japanese“geijutsu” or “āto”.
While the English word “art” encompasses techniques andskills as well as fine art, the Japanese term “geijutsu” in kanji
is limited to aesthetic meanings and may sound academic or
unfamiliar to the general public. In contrast, “āto” in katakana
has a broader connotation and gives a more casual impression.
The situation is similar with the term “therapy”. Its Japaneseequivalent words, such as “chiryō” and “ryōhō” in kanji, havea more professional connotation. Similarly, “serapī” inkatakana tends to be used in a broader or more casual context.
The polysemy of these Japanese terms suggests thatpeople’s perceptions of these concepts are diverse and evenambiguous. Loanwords written in katakana possess a broader
connotation that includes both the original meanings in
English and the Japanese translations in kanji.
Consequently, in Japanese, katakana loanwords havemeanings that are not entirely equivalent to the original
foreign words or their kanji translations. As a result, foreign
words in katakana are not always as clear in meaning as theoriginal terms or kanji translations; their usage andrecognition may vary depending on the individual. Therefore,
there are gaps in perceptions between EAT practitioners andpotential clients.

itself. Understanding the gaps and commonalities in
consciousness between EAT and AE practitioners and theirpotential clients makes it possible to facilitate thecomprehension of EAT and AE.

3 ANALYSIS OF THE CONCEPTS OF ART
AND THERAPY
3.1 Outline of Questionnaire Survey
We administered the following questionnaire to investigate
perceptions of arts and therapy:
・Target group: Students from two universities in Osaka and
Shizuoka.
・Number of respondents: 277
・Valid responses: 277
・Period of the survey: April 11 to 13, 2023
・Format: Selective questionnaire
・How to answer: Respondents answered the questionnairecreated with Google Forms on their smartphones or
personal computers.
The questionnaire was administered during class hours at
the authors’ universities. The male-to-female ratio of students
was approximately 8:2.

3.2 Contents of the Questionnaire
The questionnaire consists of the following main sections:[Section 1](1) Age of respondents(2) Place of residence during high school[Section 2](1) Classification of 60 items 1 of arts and sports genres into
the following categories: Entertainment/Hobby
 Competition
 Sport Education/Physical Education
 Art (geijutsu in Japanese) Art (āto in Japanese) Performing Art(2) Choose one word most strongly associated with the term
“therapy” (“serapī” written in katakana) as a loanword fromthe given options of events related to the arts or sports.
[Section 3](1) Respondents’ involvement in art and sports activities(with more than three years of experience).
(2) Whether the activities they focused on were athletic or
non-athletic.
[Section 4]
Classify events related to the arts or sports into one of thesame categories as in Section 2.

2.2 Issues to Be Solved
Through our survey so far, the following characteristics andsituations with EAT activities have been grasped. These weremainly revealed through the interviews to EAT practitioners
who were informants of the nationwide survey [12].
EAT practitioners utilize various methods of arts therapy
for their activities. However, they tend to hesitate about or
avoid using the term “arts therapy” because their activities do
not aim at psychotherapy. On the other hand, EAT
practitioners emphasize supporting and accepting individuals’
unique forms of expression rather than focusing on technicalinstructions or improvement in artistic activities, such as
painting, music, and dance. They view their activities as
distinct from general art activities. Therefore, EAT
practitioners recognize the challenge of finding precise words
to describe their activities accurately. The unique nature of
their approach which is different from both arts therapy as
psychotherapy and usual art activities makes it difficult toconvey the full scope of their work.
Given these situations and challenges, we conducted apreliminary survey as a first step to grasp how people
perceive the terms art(s) and therapy. EAT is a newly
proposed academic concept unfamiliar to people, so weinvestigated concrete items related to EAT instead of EAT*Different questions were given depending on whether the
respondent chose athletic or non-athletic in Section 3.

Two of the 60 items were treated as invalid due toincomplete questionnaires. Thus, the final valid number of
items was 58.

1229

International Workshop on Informatics ( IWIN2023 )Section 2(1) presented a wide range of questionnaire items
relating to arts and sports, from popular and familiar toclassical and unfamiliar. The intention was to observe theuniformity or dispersion of category choices across different
items.
The seven category options given in Japanese, including
synonymous terms like geijutsu and āto, were included toanalyze the distinction in usage between these two kanji andkatakana words based on the respondents’ answers. In
Japanese, there is an unconscious or implicit usage distinction
between terms in kanji and katakana, even when bothmeanings are identical.
The questionnaire included sports because some activities
are considered at the intersection of art and sports. Moreover,there are several concepts such as “competition (kyōgi),”“sport (supōtsu),” and “physical education (taiiku)” which areconsidered similar concepts in Japanese. Differences in theperception and usage of these terms should also be analyzed.
The purpose of the questionnaire was not to rigorously
define or organize the concepts and categories of art, sports,
or therapy. It was to grasp how people perceive these terms
and the activities they envision and to confirm the state inwhich the relevance between actual events and the categories
and terms representing them was fluctuating. It wassignificant to estimate how EAT and AE were likely to be
accepted by people by viewing what deviations occurred
between people with various perceptions. As a preliminary
study, the questionnaire survey 2 was conducted on university
students. They are considered to have more freedom in club
activities and less sense of belonging to their clubs than highschool students even if they join any clubs.
Respondents were expected to rely on the words anddescriptions commonly used in society to choose the option
reflecting their perceptions, regardless of whether they werefamiliar with the specific content of each item. It should be
noted that the questionnaire did not aim to rigidly define ororganize the concepts and categories of art, sports, or therapy.
Therefore, no categories were defined or explained in detail.
The purpose of the questionnaire was to highlight thediversity and fluctuations in the perceptions of these words
and concepts, with a focus on art-related questions.

“art as āto.” The selection of “geijutsu” surpassed “āto” forboth items. For paintings, 144 respondents chose “art asgeijutsu,” while 78 did “art as āto.” For sculptures/objects,
163 chose “art as geijutsu” and 61 did “art as āto.”On the other hand, “entertainment/hobby” was the mostcommon category for illustrations (133 respondents) (Figure4), followed by “art as āto” (99). “Art as geijutsu” (31)accounted for less than one-third of “art as āto” forillustrations.

Figure 2: Painting

Figure 3: Sculpture/Object

3.3 Results of the Survey
3.3.1 Perception of Art
This chapter examines the results of the responses to
Section 2(1) of the questionnaire regarding item categories.
This study addresses the items related to visual art, music,
theater, and dance (refer to the appendix) out of allquestionnaires including the athletic and miscellaneous items.
These items are major components of EAT and AE. The
following describes the tendency grasped.

Figure 4: Illustration
Paintings and illustrations are two-dimensional expressionsof art, but there is a difference in the category reflected in
respondents’ perceptions. Respondents selected “art asgeijutsu” more than “art as āto” for paintings andsculptures/objects; however, the converse was true for “art asāto” for illustrations. This suggests that “art as āto” isconsidered more familiar and commercial than “art as■ Visual Art
Regarding paintings (Figure 2) and sculptures/objects(Figure 3), most of the respondents chose “art as geijutsu” or
The questionnaire was prepared based on the perspectiveof phenomenological sociology proposed by A. Schutz [13].
2230International Workshop on Informatics ( IWIN2023 )geijutsu,” while “art as geijutsu” evokes a purer form of fineart, emphasizing “art for art’s sake.” This observation
highlights a difference in perception and usage between “artas geijutsu” and “art as āto” as Japanese terms.
“Entertainment/hobby” was the most common category forphotography (Figure 5), animations (Figure 6), movies, andManga (cartoons). These items can be regarded as visual art
in a broad sense. However, they are considered more familiar
than paintings and sculptures/objects. It is suggested by thefact that the majority (60-80%) of respondents chose“entertainment/hobby” for each item.
Figure 7: Classical Music

Figure 5: Photography
Figure 8: Pop Music■ Theater
In the domain of theater, “performing arts” was the mostcommon category for all items: dramas (124) (Figure 9),
Kabuki (150) (Figure 10), Noh play (140), Bunraku (Japanesepuppet theater) (136), and Manzai (comic dialogue) (140)(Figure 11).

Figure 6: Animation■ Music“Art as geijutsu” was the most common choice for classicalmusic (126) (Figure 7) and opera (134). On the other hand,“entertainment/hobby” was the most common category forpop music (152) (Figure 8), and other forms of popular music
such as music videos (125), jazz (126), and rock (138).
Unlike the visual art field, the “entertainment/hobby” ratio
is significantly high in music; the ratio of “art as āto” is low.
Even in classical music and opera, where “art as geijutsu” wasthe most common, the second most common category was“entertainment/hobby.” It may be inferred that musical andstage performances are considered both high art andentertainment.

Figure 9: Drama

Figure 10: Kabuki

231

International Workshop on Informatics ( IWIN2023 )Ballroom (Figure 14) and creative dancing were commonlychosen as “entertainment/hobby” (67 and 79, respectively).
The second most common category for ballroom dancing was“competition” (59), and for creative dance was “sport” (65).

Figure 11: Manzai
The Japanese word “geinō” means performing arts andrefers to both popular show business and classical performing
arts. Kabuki, Noh, and Bunraku are the foremost Japaneseclassical performing arts registered as World Intangible
Cultural Heritage. It is unlikely that they are completely
equated with general theatrical performances. Notably, themajor category chosen for both popular and classicalperformances is the word “geinō” in Japanese, indicating thewide-ranging scope of this term in Japanese.

Figure 14: Ballroom Dancing
In hip-hop (Figure 15) and break dancing, which are bothfamiliar to the younger generation, “entertainment/hobby”
was the most common category (126 and 97, respectively),while “sport” ranked second (50 and 63, respectively).

■ Dance
In the domain of dance, there was notable recognition of it
as a physical activity across various items. Dance wasperceived as closely related to sports or existing at theintersection of art and sports. Moreover, many dance items
were evenly spread across categories compared to other areas.
“Entertainment/hobby” was the most common category foralmost all individual dance items. However, the “art asgeijutsu” category was not commonly chosen. Even in ballet(Figure 12), a representative dance art, “sport” was the mostcommon choice (74), followed by “art as geijutsu” (54). In
ice dancing (Figure 13) “sport” and “entertainment/hobby”
ranked highest (both 66).

Figure 15: Hip-Hop
Traditional Japanese dance (Figure 16) had the largest
number of “art as geijutsu” (73) in all kinds of dance.
However, its most common category was “performing art(geinō)” (99). It is presumed that this is treated as a classicalperforming art, similar to Kabuki, Noh, and Bunraku.

Figure 12: Ballet

Figure 16: Japanese Dance
The university students surveyed were not majoring in arts
or sports and did not have specialized knowledge orexperience in these areas. Hence, the choices noted in the
results can predict the cognitive tendencies of the general
students’ perception of the arts and sports.

Figure 13: Ice Dancing

232

International Workshop on Informatics ( IWIN2023 )3.3.2 Perception of Therapy

an important aspect to carefully examine when discussing andimplementing EAT and AE approaches.

The respondents were asked to select one of the options thatmost strongly reminded them of the word “therapy.” The
results are shown in Figure 17.
The English word therapy refers to treatment or cure, whichrefers to medical practice or related fields. However, the word
most strongly associated with “therapy” as a Japanese word
written in katakana (serapī) was “relaxation” (87), followedby “healing” (66). While this perception might differ among
medical professionals and those studying medicine, itsuggests that Japanese people generally perceive theloanword serapī in a broader sense of relaxation and healingrather than strictly within the context of medical practice in
the Japanese language.

4

DISCUSSION
4.1 Outcomes

Using this questionnaire, we confirmed the diversity andambiguity of arts and therapy recognition. Overall, “art asgeijutsu” tends to be associated with things of a historical andtraditional nature (e.g., classical music and traditional
Japanese dance). “Art as āto” is linked to more modern orcommercialized items (e.g., illustrations).
“Entertainment/hobby” is commonly chosen for activities
related to self-expression and daily enjoyment (e.g.,
animation, pops). In the field of dance, there is a notableemphasis on the physicality of the expression and the broad
range of categories encompassing it.
Regarding “therapy,” respondents primarily associated itwith healing and relaxation rather than medical treatment,
regardless of strict definitions of the term.
EAT and AE practitioners should recognize the diversity
and tendencies of perceptions of arts and therapy in order toaccurately explain their activities without causing confusion
or misunderstanding. To address this issue and increase
public awareness and social recognition of their activities,
EAT and AE practitioners should recognize everyone has
their own unique perception of arts and therapy.
For many practitioners conducting EAT or AE activities
independently, ICT tools such as websites and social media
are useful for public relations. However, recipients of
information do not necessarily have the same perceptions of
the arts and therapy as practitioners. Therefore, it is importantto establish the basic policies and spread the informationconsidering people’s wide-ranged perception of arts andtherapy. These actions will assist EAT and AE practitioners
in achieving a better public understanding of their activities.
The survey results will provide clues of insights for a deeper
understanding of diversity of these terms and concepts.

[Question] Therapies utilizing arts and sports have recently
been increasing. Please choose one word that you moststrongly associate with the term “therapy”.

Figure 17: Word Most Strongly Associated with “Therapy”
Some EAT and AE practitioners perceive “therapy” ashealing and hope others will perceive it similarly. EAT
involves art activities incorporating therapeutic elements torefresh the mind and body and relieve stress. However, since“therapy” is often associated with medical treatment or cure,many EAT practitioners are cautious about using that term,as they are not engaged in psychotherapy.
The associations that Japanese people have with loanwords
written in katakana may not always align with the word’s
original meaning or its Japanese translation. Arts therapy as aloanword tends to evoke a more casual and accessible
impression than its Japanese translation geijutsu ryōhō. This
allows for a broader connotation and recognition among
Japanese people.
However, it is necessary to carefully consider whether it isappropriate for EAT practitioners to represent their activities
as “arts therapy” because it may be confused with arts therapy
as psychotherapy. This becomes a significant challenge for
EAT practitioners as they try to explain their activities
concisely and accurately. Most EAT practitioners are awareof these difficulties in describing their activities.
There is often a gap between loanwords from foreign
languages and their Japanese translations. Linguistic
discussions were beyond the scope of this study; therefore,
we did not discuss them in detail. While this study did notextensively delve into linguistic discussions, it is commonly
understood that there are discrepancies between the literaldefinitions and people’s perceptions of “arts therapy.” This is4.2 Future Issues
This survey has several limitations. The respondents werelimited to predominantly male university students (80%).
Detailed questions were not asked about the respondents’attributes, such as their social, economic, and cultural
backgrounds, which could have influenced their responses.
To promote the practice and dissemination of AE and EAT,
it is important to understand the circumstances of a morediverse population, including individuals of different ages
and genders. This understanding can inform the developmentof targeted public relations and marketing strategies. It would
also be beneficial to analyze the correlation between
respondents and their economic and cultural capital.
Despite these limitations, this preliminary survey shed lighton the diverse and ambiguous perceptions of arts and therapy,
providing useful insights for future research. Future studies
will address the unresolved issues that require further
examination.

233

International Workshop on Informatics ( IWIN2023 )REFERENCES[1] T. Okada, Psycho-educational Approach with
Application of Psychotherapy in Elementary School Trial and Results of the Technique: ‘Picture Drawing
Play’, Japanese Bulletin of Arts Therapy, Vol.40, No.1,
pp.43-51 (2009) (in Japanese).
[2] T. Fujii, A Postcard Artist’s Report on Cloth Collage
Workshops, Japanese Bulletin of Arts Therapy, Vol.41,
No.1, pp.70-80 (2010) (in Japanese).
[3] N. Okada, Mosaic, Reflections in Art Therapy, Japanese
Bulletin of Arts Therapy, Vol.50, No.1, pp.90-97 (2019),(in Japanese).
[4] M. Ishihara, “Nichijo ni yorisou art therapy - Kosodateshien toshite no katsudo-jirei ni art therapy no hattenkei
wo miru -”, K. Tokiko, and K. Nishi (eds.), Art Therapy
Saiko - Geijutsugaku to rinsho no genba kara -,Heibonsha, Tokyo, pp.232-253 (2013) (in Japanese).
[5] Headquarters for Vitalizing Regional Cultures, Agency
for Cultural Affair, Bunka ni kansuru yoron chosa -Wellbeing to bunka geijutsu katsudo no kanren hokokusho (2022) (in Japanese).
https://www.bunka.go.jp/tokei_hakusho_shuppan/tokeichosa/pdf/93858401_01.pdf[6] Shunsuke Tsurumi, Genkai-geijutsu-ron, Chikuma
Shobo, pp.14-15 (1999) (in Japanese).
[7] H. Kaneko, and M. Ishihara, A View of ‘Art Therapy for
Empowerment’, Journal of the Kobe University of
Welfare, Vol.19, No.1, pp.1-22 (2018) (in Japanese).
[8] M. Komura, and A. Hayashi, A Method of Assessment
for Empowerment Arts Therapy, IEICE Technical
Report, Vol.121, No.372, pp. 1-7 (2022) (in Japanese).
[9] M. Komura, M. Ishihara, and A. Hayashi, Proposal for
Continuous Improvement of Empowerment Arts
Therapy - A Method of Self-Assessment by Applying
Rubric -, Proc. International Workshop of Informatics(IWIN 2022), pp.121-126, Nachi-Katsuura, Japan(2022).
[10] M. Ishihara, M. Komura, A. Hayashi, and H. Kaneko,
Empowerment gata arts therapy no self-assessment nojitsuyoka ni mukete - Empower suru hito wo empowersuru tame ni -, The Crisis of the Mind and ClinicalKnowledge (Bulletin of Konan Institute of Human
Sciences, Konan University), Vol. 24, pp.1-21 (2023)(in Japanese).
[11] M. Komura, M. Ishihara, H. Kaneko, and A. Hayashi,
Effectiveness Applying a Rubric in Self-Assessment of
Empowerment Arts Therapy Practitioners, IEICE
Technical Report, Vol. 122, No. 282, pp. 33-39 (2022)(in Japanese).
[12] M. Ishihara, and H. Kaneko, Empowerment toshite noshisei no art therapy, The Crisis of the Mind and ClinicalKnowledge (Bulletin of Konan Institute of Human
Sciences, Konan University), Vol. 16, pp.105-130(2015) (in Japanese).
[13] A. Schutz, edited by M. Natanson, translated by H.
Watabe, H. Nasu, and, K. Nishihara, Alfred Schutz
chosaku-shu 2: Shakai-teki genjitsu no mondai II,
Maruju-sha (1985) (Japanese translation of A. Schutz,edited & introduced by M. Natanson, Collected Papers
I: The problem of the Social Reality - Part III, The
Hague (1973)).

234

International Workshop on Informatics ( IWIN2023 )Appendix: Questionnaire Results of Items Relating to Arts
Items

Dance

Drama

Music

Visual Art

Painting

Entertainment/Hobby
44

Competition
1

Category Selection
Education
Art asSport/PE'geijutsu'
3
7
144

Art as'āto'78

Perform-ing Art
0

Total

Graph

277

Fig.2

Illustration

133

5

2

4

31

99

3

277

Fig.4

Sculpture/Object

36

4

6

6

163

61

1

277

Fig.3

Movie

190

0

1

4

32

13

37

277

Photography

166

3

0

2

63

40

3

277

Animation

228

5

2

0

12

20

10

277

Manga

226

2

1

3

17

22

6

277

Fig.5

Calligraphy

39

17

4

66

98

34

19

277

Fig.6

Karaoke

256

4

3

1

3

2

8

277

Classical Music

90

5

3

14

126

18

21

277

Fig.7

Pop Music
Japanese PopularSong (Kayokyoku)
Music Video

152

5

7

5

41

18

49

277

Fig.8

122

3

3

12

44

11

82

277

125

3

2

5

46

31

65

277

Opera

58

8

6

14

134

12

45

277

Falk Song

62

8

3

47

83

15

59

277

Japanese Enka Song

96

5

4

9

56

11

96

277

Jazz

126

7

5

1

61

26

51

277

Rock

138

7

4

4

46

26

52

277

Drama

67

9

5

9

56

7

124

277

Fig.9

Kabuki

41

8

2

15

55

6

150

277

Fig.10Noh play

43

4

3

20

54

5

148

277

Bunraku

45

6

2

15

64

9

136

277

Manzai

113

11

4

2

6

1

140

277

Fig.11

Ballet

33

54

74

18

52

12

34

277

Fig.12Creative Dance

79

37

65

19

27

21

29

277

Ballroom Dancing

67

59

52

28

29

8

34

277

Fig.14

Japanese Dance

43

16

18

24

73

4

99

277

Fig.16

Folk Dance

69

38

49

22

38

12

49

277

Ice Dancing

66

34

66

12

40

12

47

277

Japanese Bon Dance

96

5

11

29

48

2

86

277

Hip-Hop

126

27

50

6

16

18

34

277

Breakdance

97

41

63

10

16

14

36

277

Modern Dance

102

29

59

7

22

19

39

277

Jazz Dance

99

34

51

11

27

19

36

277

Cheer Dance

76

46

85

18

12

9

31

277

Hula

106

32

48

14

22

14

41

277

235

Fig.15