IWIN2018
International Workshop on Informatics

Proceedings ofInternational Workshop on Informatics
September 9-12, 2018
Salzburg, Austria

Sponsored by Informatics Society

IWIN2018
International Workshop on Informatics

Proceedings ofInternational Workshop on Informatics
September 9-12, 2018
Salzburg, Austria

Sponsored by Informatics SocietyPublication office:Informatics Laboratory
3-41, Tsujimachi, Kitaku, Nagoya 462-0032, Japan
Publisher:
Tadanori Mizuno, President of Informatics SocietyISBN:
978-4-902523-43-0
Printed in Japan

Table of Contents
Session 1: Intelligent Transportation Systems( Chair: Yoshia Saito ) ( 9:10 - 10:30, Sep. 10 )(1)

Rightfulness evaluation of obtained data from traffic simulation for improving traffic flow······································································································· 3
Kazuki Someya, Masashi Saito, and Ryozo Kiyohara(2)

Study on state transition table based on context and risk degrees ·························· 9
Hirotaka Sakai, and Ryozo Kiyohara(3)

A method to synchronize movie and acceleration sensor data using directions of vectors
determined by corresponding points between video frames ·································· 13
Yosuke Ishiwatari, Takahiro Otsuka, Masahiro Abukawa, and Hiroshi Mineno(4)

A method for estimating degradation level of road markings by participatory sensing with
in-vehicle camera ·················································································· 17
Yoh Shiraishi, and Issei Sugaⅰ

Session 2: IoT Systems( Chair: Ryozo Kiyohara ) ( 10:40 - 12:00, Sep. 10 )(5)

A proposal of PC facilities utilization state management system in the office and other byusing the IoT technology ········································································· 25
Keiichi Abe, Kurika Kobayashi, Kouki Iizuka, and Masao Isshiki(6)

Proposal of IoT edge optimization model for balancing local system and total system andits simulation evaluation ·········································································· 33
Shinji Kitagami, Yuichi Tokunaga, and Norio Shiratori(7)

Time synchronization method for rice cultivation management system with IoT
specialized outdoor communication procedure ················································ 37
Koichi Tanaka, Mikiko Sode, Masakatsu Nishigaki, and Tadanori Mizuno(8)

Proposal of IoT system with SmartContract on BlockChain ································· 43
Tetsuo Furuichi, and Hiroshi Mineno

Keynote Speech 1( 13:30 - 14:10, Sep. 10 )(I)

Cyber-physical security in an age of digital transformation ···································· 51

Mr. Kazuhiko Ohkubo, Vice President / Head of NTT Secure Platform Laboratories

ⅱSession 3: Multimedia Systems( Chair: Katsuhiko Kaji ) ( 14:20 - 15:20, Sep. 10 )(9)

A method for video advertisement insertion on smartphone ································· 71
Yoshia Saito(10)

Experiments to investigate the effects of scents on vection perception ····················· 77
Aoi Aruga, Yuichi Bannai, and Takeharu Seno

(11)

A distributed multi-viewpoint Internet live broadcasting system with video effects ······ 83
Satoru Matsumoto, Tomoki Yoshihisa, Tomoya Kawakami, and Yuuichi Teranishi

Session 4: Network and Security( Chair: Kozo Okano ) ( 15:30 - 16:50, Sep. 10 )(12)

Detection of malware-infected host with data compression algorithm ······················ 91
Yasushi Okano, Kazunori Kamiya, Atsutoshi Kumagai, Taishi Nishiyama, Bo Hu,
Masaki Tanikawa, and Kazuhiko Ohkubo(13)

Detection of malware infection using traffic models based on the similarity between
malware samples ··················································································· 97
Masatsugu Ichino, Yuuki Mori, Mitsuhiro Hatada, and Hiroshi Yoshiura(14)

A method of lightweight secure communication considering reliability in IPv6 wirelesssensor network ···················································································· 103
Shunya Koyama, Yoshitaka Nakamura, and Hiroshi Inamura(15)

Effective countermeasure against biometric spoofing attacks using unsupervised one-classlearning ···························································································· 109
Vishu Gupta, Masakatsu Nishigaki, and Tetsushi Ohkiⅲ

Session 5: Data Models( Chair: Yoshia Saito ) ( 9:00 - 10:20, Sep. 11 )(16)

A proposal of deep learning training data generator for image recognition ··············· 117
Tsukasa Kudo, and Ren Takimoto(17)

Applying SAW to regression verification for C functions with recursive data structure 125
Kozo Okano, Satoshi Harauchi, Shin Maruyama, and Shinpei Ogata(18)

Occurring Frequency for Selecting Optimal Window Length in Motif Discovery ······· 133
Makoto Imamura, Mao Inoue, Tadao Yagou, and Daniel Nikovski(19)

Visualization of skills and techniques using big data analysis in vocational skill
development ······················································································· 141
Masaki Endo, Takuo Kikuchi, Shigeyoshi Ohno, Makoto Imamura, andHiroshi Ishikawa

ⅳSession 6: Systems and Applications( Chair: Ohki Tetsushi ) ( 10:30 – 12:10, Sep. 11 )(20)Interactive topic confirmation on decision support system for information dissemination······································································································ 149
Kei Utsugi Hajime Mori, Kiyohiro Ohara, Masahide Okamoto, and Yuuki Shimizu(21)

Initial attempt of bluetooth low energy water-level estimator······························· 157
Ryo Orihara, Shigemi Ishida, Masahiko Miyazaki, Shigeaki Tagashira, andAkira Fukuda(22)

Development of a self-guided web-based mental healthcare course for day-to-day practice······································································································ 163
Takeshi Kamita, Tatsuya Ito, Atsuko Matsumoto, Tsunetsugu Munakata, andTomoo Inoue(23)

Healthcare promotion system by stimulating in-group contribution mind and inter-groupcompetitiveness ··················································································· 169
Yuki Aso, Nobuyuki Ito, Katsuhiro Naito, Naoya Chujo, Tadanori Mizuno, andKatsuhiko Kaji

Keynote Speech 2( 13:30 - 14:10, Sep. 11 )(II)

Digital transformation towards Society 5.0···················································· 181
Dr. Shigetoshi Sameshima, Deputy General Manager, Center for Technology
Innovation / General Manager, Yokohama Research Laboratory, Hitachi,Ltd., Research &Development Groupⅴ

Session 7: Trust and Risk
( Chair: Tomoki Yoshihisa ) ( 14:20 - 15:20, Sep. 11 )(24)

A study of risk management method using quantitative proces management ············· 185
Akihiro Hayashi, Nobuhiro Kataoka, Yasunobu Kino, and Mikio Aoyama(25)

Trust representation under confusion and ignorance ········································· 191
Kazuhiko Ohkubo, Tetsuhisa Oda, Yuki Koizumi, Tetsushi Ohki, Masakatsu
Nishigaki, Toru Hasegawa, and Yoshinobu Kawabe(26)

Elliptic curves suitable for elliptic curve signature ··········································· 199
Masaaki ShiraseⅵMessage from the General Chairs

It is our great pleasure to welcome all of you to Salzburg, Austria, for the TwelvesInternational Workshop on Informatics (IWIN 2018). This workshop has been held annuallyby the Informatics Society. Since 2007, the workshops were held in Naples in Italy, Wien inAustria, Hawaii in the USA, Edinburgh in Scotland, Venice in Italy, Chamonix in France,
Stockholm in Sweden, Prague in Czech Republic, Amsterdam in Netherlands, Riga in Latvia,
and Zagreb in Croatia, respectively.
In IWIN 2018, 26 papers were accepted after peer reviewing by the program committee.
Based on the papers, seven technical sessions were organized in a single track format, whichhighlighted the latest research results in the areas such as Intelligent Transport System(ITS), Internet of Things (IoT), Trust and Risks, Mobile Computing, Networking, Data
Models and Multimedia Systems. IWIN 2017 will also welcome two keynote speakers: Mr.
Kazuhiko Ohkubo of Vice President, Head of NTT Secure Platform Laboratories and Dr.
Shigetoshi Sameshima of Deputy General Manager, Center for Technology Innovation andGeneral Manager, Yokohama Research Laboratory, Hitachi, Ltd., Research & Development
Group. We really appreciate their participation in the workshop.
We would like to thank all the participants and contributors who made the workshop
possible. It is indeed an honor to work with a large group of professionals around the world
for making the workshop a great success. We are looking forward to seeing you all in theworkshop. We hope you enjoy IWIN 2018.
September 2018
Hiroshi Inamura
Yoh Shiraishi

ⅶ

Organizing Committee
General Co-Chairs
Hiroshi Inamura (Future University Hakodate, Japan)
Yoh Shiraishi (Future University Hakodate, Japan)

Steering Committee
Hitoshi Aida (The University of Tokyo, Japan)
Toru Hasegawa (Osaka University, Japan)
Teruo Higashino (Osaka University, Japan)
Tadanori Mizuno (Aichi Institute of Technology, Japan)
Jun Munemori (Wakayama University, Japan)
Yuko Murayama (Tsuda University, Japan)
Ken-ichi Okada (Keio University, Japan)
Norio Shiratori (Chuo University / Tohoku University, Japan)
Osamu Takahashi (Future University Hakodate, Japan)

Program Chair
Yoshia Saito (Iwate Prefectural University, Japan)

Financial Chair
Tomoya Kitani (Shizuoka University, Japan)

Publicity Chair
Yoshitaka Nakamura (Future University Hakodate, Japan)

Program Committee
Keiichi Abe (Kanagawa Institute of Technology, Japan)

Masahiko Ishino (Bunkyo University, Japan)

Naoya Chujo (Aichi Institute of Technology, Japan)

Katsuhiko Kaji (Aichi Institute of Technology, Japan)

Chiaki Doi (NTT DOCOMO, Inc., Japan)

Tomoko Kaneko

Yu Enokibori (Nagoya University, Japan)(Information-technology Promotion Agency, Japan)

Hisao Fukuoka (Tokyo Denki University, Japan)

Masaji Katagiri (NTT DOCOMO, Inc., Japan)

Yusuke Gotoh (Okayama University, Japan)

Yoshinobu Kawabe (Aichi Institute of Technology, Japan)

Takaaki Hishida (Aichi Institute of Technology, Japan)

Tomoya Kitani (Shizuoka University, Japan)

Hiroshi Horikawa

Ryozo Kiyohara (Kanagawa Institute of Technology, Japan)(Mitsubishi Electric Information Network Corporation, Japan)

Minoru Kobayashi (Meiji University, Japan)

Toru Hoshi (Tokyo University of Technology, Japan)

Tsukasa Kudo

Yusuke Ichikawa (NTT Corporation, Japan)(Shizuoka Institute of Science and Technology, Japan)

Makoto Imamura (Tokai University, Japan)

Hiroshi Mineno (Shizuoka University, Japan)

Tomoo Inoue (University of Tsukuba, Japan)

Shinichiro Mori (Chiba Institute of Technology, Japan)

Hiroshi Ishikawa (Tokyo Denki University, Japan)

Hiroaki Morino (Shibaura Institute of Technology, Japan)ⅷ

Katsuhiro Naito (Aichi Institute of Technology, Japan)

Ken Ohta (NTT DOCOMO, Inc., Japan)

Yoshitaka Nakamura (Future University-Hakodate, Japan)

Masashi Saito (Mitsubishi Electric Corporation, Japan)

Masakatsu Nishigaki (Shizuoka University, Japan)

Fumiaki Sato (Toho University, Japan)

Kozo Okano (Shinshu University, Japan)

Masaaki Shirase (Future University Hakodate, Japan)

Tetsushi Ohki (Shizuoka University, Japan)
Hideyuki Takahashi (Tohoku University, Japan)

Hirozumi Yamaguchi (Osaka University, Japan)

Yusuke Takatori (Kanagawa Institute of Technology, Japan)

Tomoyuki Yashiro (Chiba Institute of Technology, Japan)

Yoshiaki Terashima (Soka University, Japan)

Takuya Yoshihiro (Wakayama University, Japan)

Hirosato Tsuji

Tomoki Yoshihisa (Osaka University, Japan)(Mitsubishi Electric Information Systems Corporation, Japan)

Takashi Yoshino (Wakayama University, Japan)

Hiroshi Yoshiura(The University of Electro-Communications, Japan)
Takaya Yuizono(Japan Advanced Institute of Science and Technology)ⅸSession 1:
Intelligent Transportation
Systems( Chair: Yoshia Saito )International Workshop on Informatics ( IWIN 2018 )Rightfulness Evaluation of Obtained Data From Traffic Simulation
for Improving Traffic Flow
Kazuki Someya*, Masashi Saito** , and Ryzo Kiyohara**

Kanagawa Institute of Technology, Japan
Kanazawa, Institute of Technology, Japan({s1521138@cco, kiyohara@ic}.kanagawa-it, msaito@neptune.kanazawa-it).ac.jp**

Abstract – As traffic jams in urban areas are becoming
worse, some cities are taking measures, such as increasing
the number of roads through the effective utilization ofunderground roadways and detecting increasing traffic
volume. Moreover, many cities can avoid traffic jams byencouraging the use of public transportation. However, inprovincial cities, public transportation services are poor,and many people use cars because of the small number oftrains, even if there are railroads. Therefore, traffic jams
continue to be a problem. One method of grasping the flowof traffic is to use the data of a road traffic census toinvestigate the traffic volume and the data of mobile spacestatistics expressed in the mesh region based on theposition information of users of mobile phones. A
simulation was performed to obtain the same data at thesame time, and the accuracy of the data was demonstrated.
Keywords: ITS, Traffic jam, Simulation, Open data .

1

a parking lot near public transportation, and a system thatmakes public transport available for long distances.
However, many people are not using P&R owing to low
parking lot capacity and high usage fees [4].
Road pricing is a system that collects fees whentraveling on a specific road, such as an expressway. As it
is limited to large cities that already have many public
transportation facilities, it will be ineffective in theprovincial cities with limited alternative transportation
methods.
The above considerations show that it is necessary to
improve the accuracy of the simulation and improve theaccuracy of the effect obtained when the road is improved.
For that purpose, it is necessary to know the origin–destination traffic volume [5] of the driver as aprecondition.
Using a personal trip survey [3] is the conventional
method. However, the cost of implementing a personal slip
investigation is large, and it lacks accuracy because it
requires data to be provided that depend on a person’s
memory. Thus, the frequency of such surveys is not high,and is often limited to weekdays in a city every few years
because of costs, so this method is not recommended.
However, in Japan, a road traffic census is conducted
every five years and it discloses traffic surveys as open
data. This is data obtained by manually counting thenumber of vehicles at an intersection or the like.
Moreover, mobile spatial statistics are available, which
disclose information on terminals existing in base stationsof mobile phones as open data.
In this study, we propose traffic flow simulation using
the road traffic census and mobile space statistics data. As
shown in Figure 1, the traffic flow movement is predictedfrom past data. As shown in Figure 2, simulation is
performed so that the same data as the road traffic census
and the mobile space statistics can be acquired, and thevalidity of the data is obtained for the area around
Kanazawa station in the Ishikawa prefecture. Because theexact number of cars and people cannot be known fromthe open data, the data cannot be used as is. Therefore, thesame acquisition method as for open data was
implemented using simulation.

INTRODUCTION

Recently, traffic jams have become increasingly serious.
In a developed city, traffic jams can be avoided whencommuting by using public transportation. Moreover,because of the increasing use of automated vehicles in the
future, people have greater concern about traffic jams
becoming even more severe.
In a city with financial strength, such as Tokyo, it ispossible to respond to this problem by using undergroundroads. However, not only in Japan but also in many
provincial cities, public transportation has not developed
as much, and movement using automobiles is mostcommon.
Therefore, reduction of traffic jams inprovincial cities is important. As a result of improving
roads, for cases in which the frequency of use is small, noeffect can be obtained at all. In provincial cities with little
financial power, there is no room to use low-impact roadmaintenance, thus it is necessary to implement it reliably
and effectively. Therefore, it is necessary to carry out asimulation beforehand to determine whether an effect canbe obtained.
Especially in the provincial cities of Japan developed
mainly around rivers, there are many cities having thecharacteristics of sea, mountains, and rivers. As a result,because roads over bridges and through mountain areas arelimited, a bottleneck exists that causes traffic jams becauseof the small number of roads that can pass, even though
the traffic volume is high.
Many cities have introduced park and ride (P&R)
systems [1][2] and road pricing [3] as measures to alleviatetraffic jams in these provincial cities. The P&R system has
3

International Workshop on Informatics ( IWIN 2018 )2

method by verifying its accuracy [5]. We verified theseparation between migrants and residents, but theestimation accuracy is not high.
We estimated the number of users by mode oftransportation, but, because it is not based on precise
techniques or automobile traffic survey data, the actual
traffic volume cannot be predicted and is a future topic forstudy.
NTT DOCOMO has been experimenting with the "nearfuture people forecast" [9], predicting movement several
hours ahead by using a real-time version of mobile spacestatistics and spatiotemporal variable online prediction
technology. NTT DOCOMO is currently developing an
artificial-intelligence taxi, which predicts the number oftaxis that will be used in 30 min. Forecasting the numberof people in the near future is a technique for predicting
the number of people in an area several hours ahead.
We propose a vehicle direction estimation method using
the Road Traffic Census and mobile space statistics for
Kanazawa and Nonoichi. If we assume that an error of±20% is tolerated, a regression equation at approximately
a 63% mesh is derived. Because the determination
coefficient R2 of the regression equation is 0.58 and theaccuracy is low, improvement of the accuracy is a problem.

RELATED WORKS

Several methods of forecasting the traffic volume havebeen proposed. Particularly, there is the method of using apersonal
trip
survey.
However,problemsofimplementation cost and the accuracy of the data havebeen pointed out for personal trip surveys. However, by
putting an application on a smartphone, one can extract it
as human behavior information and use it as data for a
personal trip survey [6]. However, it is related to the dataacquisition method of personal trip surveys, and it cannot
be used immediately because of the necessity of installing
applications.
Also, a method of estimating the origin–destination
traffic volume by means of transportation using retention
population data has also been proposed and implemented[7].
A "demand bus" operates in response to a demand. It is
like a taxi, and it is effective for rural areas with few bus
stops, because it does not run on a fixed route. As a result,the fee is expensive compared with the usual bus, and thetraveling time depends on other users. However, if thenumber of users increases, there is a problem that routecalculation becomes difficult to design in a timely manner.
Therefore, relaxing the calculation time required for path
planning using a hierarchical cooperative transport system
can eliminate the problem [8]. It also makes it possible touse this option in provincial cities with high traffic demand,such as regional core cities
There is also a system that combines the advantages ofdemand-type taxis and merged-type buses, the smart
access vehicle. We address the inverse estimation problem
of OD traffic volume by a transportation method using
mobile residence population data from mobile zone
statistics, the number of people getting on and off on
specific bus route, and individual traveling locus data of asmall sample from the viewpoint of easily obtained
observation data. The calculated traffic volume between
200 zones of a 500-m mesh in Hakodate in the Hokkaido
prefecture was used, and it shows the usefulness of the

3

PROPOSED METHOD

The following problems exist when using either roadtraffic census or mobile space statistics data alone.
① The road traffic census cannot grasp the traffic
volume of all roads, because it is implemented only
on specific roads.
② Because mobile space statistics acquire the position
information of the mobile terminal on a mesh unit
basis, it is impossible to determine whether theobject is a pedestrian or a car.
③ Additionally, because the data do not have IDs forrecognizing an item, it is not possible to find the
movement data of a specific car.

Figure 1. Traffic flow prediction

Figure 2. Around Kanazawa in Ishikawa
prefecture（Source: Google）
4

International Workshop on Informatics ( IWIN 2018 )Table 1. Example of Road Traffic Census (Partially Extracted)
Hourly traffic volume (count/hour)
7hour 8hour 9hour
5hour 6hour

id

date

up・
down

type

17300080010

20151008

1

1

550

337

327

17300080010

20151008

1

2

129

120

179

17300080010

20151008

2

1

360

318

17300080010

20151008

2

2

73

111

24 hour traffic
volume

66

274

5968

116

130

2274

331

38

168

5777

169

107

109

2376

~

Table 2. Example of mobile space statistics
date

day

hour

area

residenceage

genderpopulation

Kanazawa

2015/11/19

木

7:00

543665645

17201

-1

-1

7896

Kanazawa

2015/11/19

木

8:00

543665645

17201

-1

-1

8012

Nonoichi

2016/04/06

水

18:00

543654865

17203

-1

-1

1779

Nonoichi

2016/04/06

水

19:00

543654865

17203

-1

-1

1542

Hakusan

2016/02/10

火

9:00

543614465

17210

-1

-1

823

Hakusan016/02/10

火

10:00

543614465

17210

-1

-1

902

Thus, accuracy is low when trying to understand the
movement of a car with only one source of open data.
Therefore, in this research, we evaluate the validity of thesimulation data, which is the preliminary stage of deriving
the regression equation from the road traffic census andmobile space statistics data.
Instead of using actual data directly, we installed theobservation spots/mesh ranges where the road traffic
census and the mobile space statistics were executed in thesimulator, acquired the data, and carried out theexperiment.
In converting the information of the road traffic census
based on the mesh used in the mobile space statistics, weborrowed data from the research group and used it asshown in Figure 3.
The ratio of automobiles and pedestrians was changed,
and data were acquired. The ratios used were 10:0 and 5:5..

3.1

place/destination of automobile operation, purpose ofoperation, etc. The contents are 12 h of traffic between
roads, traffic volume of 24 h, average traffic speed duringcongestion, crowding at the peak traffic times in themorning or evening, and vehicle-type classification.
Details of the data are shown in Table 1.
The road traffic census data used in this study was for
2017, which is the most recent.

3.2

Mobile space statistics

Road Traffic Census

The road traffic census [11] is conducted to understandthe condition of roads and road traffic throughout Japanevery five years by the Ministry of Land, Infrastructure
and Transport Road Bureau. It investigates nationwide
road conditions, traffic volume, travel speed, departure

Figure 3. Road traffic census survey areaaround Kanazawa station
5

International Workshop on Informatics ( IWIN 2018 )that reason, there are two types of means of transport:
walking and cars. The maximum number of passengers
represents the number of people who can ride in one car.
In other words, in this research, only the driver is in the car.
The number of human and vehicle agents is based on thehourly traffic volume table of the Road Traffic Census,
based on the number 317,597.6, which is the average ofthe number of units of 7:00 to 12:00 a.m. that are needed
to carry out the simulation if the number of moving
vehicles is large. Because the time is enormous, it isrounded off to 1/100 and is set to 3176. Also, 34 patternsof human and vehicle agents are prepared so that they headto their destinations.
The experiment time was based on 6 h = 21,600 s from7:00 to 12:00 a.m. At the start of the simulation, there
were 0 cars, and they appeared exponentially. As a result,the number of vehicles increased with the passage of time,causing serious traffic jams and a situation in which
mobility could not be obtained. When falling into a state
where this mobility could not be acquired, it was abehavior that could not be realized, and the result of thesimulation became unexpected. Thus, the experiment time
was set to 21,600 s to 1/10, 2,160 s.
Examining the data of mobile space statistics, the mesharound Kanazawa Station is a 2-km mesh, so, to represent
the whole area, as shown in Figure 4, a 5 × 5 mesh with 2
km per side was created.
As shown in Table 5, the data acquired were based on ajudgement whether the moving object is a car or apedestrian, on the name of the current road, and on
position information on the x-axis and the y-axis. These
data were acquired at intervals of 360 s. Because the nameof the road was based on the data of the road traffic census,
in the case of walking, it was blank. The x- and y-axes
were used for conversion to a mesh. From the data of theroad traffic census, shown in Table 5, only the row inwhich the moving object was a car and the road name wereextracted. Originally, data on road traffic census did notdescribe the latitude and longitude of the observation pointthat replaces the x-axis and the y-axis. Therefore, in thisTable 3. Simulation settings
Human Agent
3,176
Vehicle Agent

3,176

Maximum number ofpassengers
Vehicle Speed

160km/hPedestrian Speed

3~5km/hExperiment Time

2,160sec

Mobile space statistics [12] are open data provided by
NTT DOCOMO. They periodically detect the mobile
phones that exist in each area of each base station. The
population during daytime and night can be roughly
predicted, and, depending on the time, it can be known
how many people commute to and drive in the central city.
Details of the data are shown in Table 2
Because the information based on the mobile spacestatistics data includes date, time, area, residence, age,gender, and population, and because the information isbased on cell phone contracts, a cell phone that a parent
bought for a child in that case is nominally a parent’s
phone, so the age is considered the parent's age, but
although obtaining precise information is not possible, thedata are sufficient as information for knowing overall
trends.
The population is the number of mobile phones. The
data of the area is cut in a mesh, and the area is confirmedby side 200 m - 2 km. Because the number of base stations
is large in urban areas such as Tokyo, the distance of one
side becomes short. Because age is based on informationthat can be contracted, such data are collected as thewritten age shown in Table 4, and they become rough data.
The determination of low age group and high age groupassumes that the contract does not necessarily belong to
the person himself/herself. It is based on the premise thatfamily registration is not necessarily done. With respect tothe problem of personal information, processing is done
without specifying an individual, and because the contract
data of the corporate name is removed, it is safe.

4
4.1

EXPERIMENT
Environment of Simulation

An experiment was conducted using "Scenarige" [13], amultiagent traffic simulator of the Space–TimeEngineering Company. The simulator was set as shown in
Table 3. The map data published in OpenStreetMap wereused [14].

4.2

Model

The human agent represents a person and treats the
person as a pedestrian when not on a vehicle agent. For

Figure 4. Allocation of mesh number
6International Workshop on Informatics ( IWIN 2018 )Table 4. Age classification
Actual age

15~19

20~29

30~39

40~49

50~59

60~69

70~79

Recorded age

15

20

30

40

50

60

70

X axis

Y axisTable 5. Example of acquired dataTime(sec)

How to
move

720

Car

Ishikawa Prefecture Route 146 Kanazawa Stop
Station South Line

179.102 2815.36

720

Car

road777

41.13 1982.31

720

Walk

Road name-324.44 3949.05

study, we obtained the position information of the car as
the x-axis and y-axis.
Experimental results were for automobile-to-pedestrian
ratios of 10:0 and 5:5, and the experiment was conducted
10 times. The average is shown as the result..

4.3

items that could acquire data of the road traffic census andmobile space statistics as a method for determining theflow of people The validity of the data was evaluated.
Automobile-to-pedestrian ratios of 10:0 and 5:5 showedthat the correlation coefficient exceeds 0.95 and that thedata are reliable.
Because pedestrian movement is the same as that of theautomobiles at the present stage, it was no problem to

Result250

5Road Traffic Census (count/sec)

We divided by mesh per hour and compared the roadtraffic census data obtained from simulation with the dataof mobile space statistics. When a correlation was obtainedwith actual data. because it was convex, the correlation
was derived by a quadratic polynomial as well.. For an
automobile, when the ratio was 10:0, R2 = 0.9504; when
the correlation coefficient R2 = 0.9688, the ratio was 5:5.
It was confirmed that both have considerably highaccuracy, indicating that there is no problem with the data.
The respective graphs are shown in Figures 5 and 6.
Moreover, if the road traffic census data and the data ofthe mobile space statistics were both 0 with the mesh at thesame time, they were clearly unnecessary data for an
accurate evaluation and were excluded.
The reason why 10: 0 is not R2 = 1 is considered to bethat road traffic census does not acquire data on all roads,
and excludes cars on narrow streets such as back streets
and residential areas.
Th reason why 5: 5 is not very different from thecorrelation coefficient of 10: 0 is that the road traveled by
car and the path walked on by foot are the same setting. It
is also considered that it moved to an area not covered in
the north.

200
150
100
50
0
0

100

200

300

400

500

600

700

Mobile space statistics(count/sec)

Figure 5. Correlation of ratio 10: 0
simulate at a ratio of 5: 5. However, in reality, because

Road Traffic Census (count/sec)

90

CONCLUSION

80
70
60
50
40
30
20
10
0
0

50

100

150

200

250

300

Mobile space statistics(count/sec)

In this research, as a preliminary step for finding
countermeasures to the increasing traffic congestion inregional core cities, a simulation was performed using

Figure 6. Correlation of ratio 5: 5
7

350

International Workshop on Informatics ( IWIN 2018 )people who do not work outside the home do not movefrom the residential area and students go to school a more
realistic evaluation environment is needed. Furthermore,various means of transport other than cars, such as buses.

ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant
number 16K00433

REFERENCES[1] K.Asai, “Dictionary for road and way,” NIPPON JITSUGYO
PUBLISHING, 2001 (Japanese)[2] S. Aoshima, S Suda, et al. “Analysis on Cost-Time Characteristics
of Park & Ride Trip and Necessary Conditions of the Parking in
Provincial Regions,” INFRASTRUCTURE PLANNING REVIEW, Vol.
16 (1999) P 863-868 (Japanese)[3] The Ministry of Land, Infrastructure, Transport and Tourism,“Report of the Fifth Person Trip Survey ” 2009 (Japanese)[4] K. Enomoto, M. Saito, R. Kiyohara,” Prediction of Traffic Flow inProvincial Cities by Open Data,” IPSJ 2016-MBL-80, 2016 (Japanese)[5] R.Kanamori, K.Mizuno, I. Noda, H. Nakashima,” Estimation ofMulti-Mode Origin-Destination Flows by using Staying Population Data
at Base-Stations,” IPSJ 2015-ICS-178 (Japanese)[6] H.Nakamura, K. Miyashita, E. Hato, T. Kishii,” The Estimation
Method of Transportation mode for Person Trip Survey using
Acceleration Sensor with Random Forest,” JSTE Journal of Traffic
Engineering Vol. 1 (2015) No. 5 p. 10-18 (Japanese)[7] H. Kobayashi, S. Hirata,” A Proposal of Evaluation Method forPeople with Mobility Difficulties using the Person Trip Survey Results,”
Journal of the City Planning Institute of Japan, Vol. 48 (2013) No. 3 p.
159-164 (Japanese)[8] K.Uehara, Y.Akamine, N.Toma, M.Nerome, S.Endo, “A Propose of
Hierarchical Cooperative Transport System Using Demand Responsive
Vehicles for Mid-size Metropolitan Area”, IPSJ Journal, Vol.57， No.1
[9] NTT DOCOMO"Near Future People Forecast"https://www.nttdocomo.co.jp/info/news_release/2017/09/20_00.html[10] M.Saito，D.Hatano， R.Kiyohara：A Proposal of Vehicle Driving
Volume Estimation Method using Spatial Statistics Data，IPSJ Journal，(Japanese)2018-ITS-073[11] The Ministry of Land, Infrastructure, Transport and Tourism,“Census of Traffic Volume” http://www.mlit.go.jp/road/census/h27/[12] T. Seike, H. Mimaki, S. Morita, ” RESEARCH ON THE
EVALUATION OF REGIONAL PECULIARITIES IN KASHIWA AND
YOKOHAMA BY “MOBILE SPATIAL STATISTICS”, ” AIJ Journal ofTechnology and Design,Vol. 21 (2015) No. 48 p. 821-826(Japanese)[13] Scenargie，https://www.spacetime-eng.com/jp[14] Open Street Map Japan，https://openstreetmap.jp

8International Workshop on Informatics ( IWIN 2018 )Study on State Transition Table Based on Context and Risk Degrees
Hirotaka Sakai* and Ryozo Kiyohara**

Kanagawa Institute of Technology, Japan{s1521077@cco, kiyohara@ic}.kanagawa-it.ac.jp
Abstract -There are many researches for autonomousvehicles in Intelligent Transport System (ITS) fields in theworld. These technologies are defined from driver support
system to driverless systems. However, the lifetime ofvehicles are very long. Therefore, there will be mixed
environment of autonomous and non-autonomous vehicles
which have to require the driver support system. In thispaper, we introduce the state transition table based oncontexts which are driver status, vehicle status, othervehicles status, road status and etc. Then, we defined therisk degrees by the experiment using driving simulator.

technologies for the driver are required.
There are many preventive safety technologies. Many ofthese technologies are event driven technologies. Most ofthese technologies are supported to avoid the accident or
keep safety. However, these support system are sometimes
too much or too early, because the timing to support andlevel to support is different for each drivers. Therefore, we
focus on the contexts which the driver, the vehicle, vehicles
around the vehicle, roads, weather and etc. Suitable
information or action which support to driver and reduce theaccidents for each driver depends on the contexts. There aremany accidents which reasons are shown in Table 2.These
reason is depend on the many contexts.
Moreover, distracted driving is the reason from otheraspects. Therefore, we have to understand how dangerous of
distracted driving, specially using smartphones.
In this paper, we defined these contexts and discuss about
driver’s contexts with risk degrees. Then we experiment
how dangerous the distracted driving with driving simulator.
Moreover, we discuss about the contexts from this result.

Keywords: Context, vehicle, ITS, Risk degree, State
Transition table.

1

INTRODUCTION

Recently, autonomous vehicle technologies are researchedand developed widely. National Highway Traffic Safety
Administration (NHSTA) defined the autonomous vehicle
technology level as shown in Table 1 [1].
Current level of vehicles in the market is from level 0 tolevel 2. The autonomous vehicle of level 3 is demonstrated
by many manufacturers. In the near future, the autonomousvehicle of level4 and level5 will be developed and shipped.
However, human operated vehicle will be driven more than
20 years. Therefore, many vehicles will be driven in themixed environment of the autonomous vehicles and humanoperated vehicles. It means the further preventive safety

2

WHAT IS CONTEXT

We defined that the contexts are status of the driver, thevehicle who the driver operates, other vehicles around thevehicle, road surface, and weather. Moreover, the status ischanged by some events (e.g. sudden braking of frontvehicle, traffic signal, called by hands-free phone and etc.)
Knowledge of location is also context. Because, thedriver’s distraction is depend on the knowledge of locations
or roads.
We think we can define the state transition table. The stateis changed by many events. This table might be too large totranslate for the program. However we can cover the all
status for guarantee of safety in any situation.

Table 1: Definition of the autonomous vehicle technology
level
Level

substance

operatorlevel 0
Human
operated

Driver operates all tasks

humanLevel 1
supported

System operates a sub-task ofcontrolling a vehicle

humanLevel 2
Partially
automated

System operates some sub-tasks ofcontrolling a vehicle

human

There are many studies for estimating the human status.
Sleepiness is one of the most important factors as a driver’s
context. In many case, the camera located on dashboard, on
review mirror or on the information devices records thelevel 3
Conditionallyautomated

System operates all tasks forcontrolling a vehicle except the
emergency

System andhuman

T able 2: Number of accidents in Japan in 2013

Level 4
Automatedhigh level

System operates all tasks forcontrolling a vehicle when theconditions are satisfied

systemLevel 5Fully
automated

System operates all tasks in anytime

system

3

RELATED STUDIES

ReasonMistaking of steering
Mistaking of pedals
Mistaking of braking
Others9

Number of accident
5,940
6,402
20,1179,346

International Workshop on Informatics ( IWIN 2018 )status of eye [4]. Therefore, we can estimate easily tosleepiness. Moreover, the drivers fatigue can be estimatedby Bayesian Network technologies [5].
The degree of driver’s vigilance is estimated by camera[6]. These technologies can estimate the driver’s status bycamera which has to be installed on the vehicle. There areother studies for estimating the drive’s status bysmartphones [7]. We think we can estimate the important
status of driver by these technologies.
The knowledge of location is constructed in the cognitive
maps. This cognitive map depends on many landmarks [8].
This status can be estimated from logs in navigation devices.
There are many studies for vehicle to vehicle or roadsideequipment [9] which can get many kinds of information
around the vehicle (e.g. other vehicles velocity, status ofroad surfaces, traffic accident information and etc.)
There are no studies which gather many kinds ofinformation and feedback to the driver to the dangerous
information for safety. Therefore, in this paper, we defined
status transition table which can be decided from many thesetechnologies.

4

There are too many states and too many events.
Therefore, we classified and categorized to degree the statetransition.

4.2 What is risk degree
We defined the risk degree corresponded for each state inTable 3. There are three kinds of risks which the driver has,other vehicles have and the driver gives other vehicles. We
defined the degree five levels temporarily. After experimentor evaluation, we have to redefine these levels. Because,
there are no bases of risk levels.

5

EXPERIMENT

We experimented to evaluate for definition of risk levels
by driving simulator.

5.1 Environment of experiment
The driving simulator which used is simple type ofsimulator as shown in Figure 1. The events are raised byscenario which can be edited to many types of events. We
prepared two types of scenario. One is sudden braking
scenario which cause the rear-end accidents. Anotherscenario is the entanglement accident for bikes. Each
operation and events are logged by 10ms intervals.

PROPOSED METHOD
4.1 State Transition of Context

We have studied by gathering many kinds of informationfrom many drivers by interviewing what events give them
the impact of their driving. We have classified the answers
to three types which are driver, the vehicle state andprediction. Then, we defined five context which weclassified more detail of three types as follows: Vehicles state
Velocity, braking and etc. which can be gotten fromvehicle network (on board diagnostic port)
 Driver’s state
Estimating from the data which can be gotten fromsensors or camera on the vehicle.
 Driver’s behavior
Getting from the operational logs from camera or
on-vehicles devices
 State around the vehicle
Getting from vehicle to vehicle (V2V) or vehicle toroadside equipment (V2R) communication
 State of destination or on the route getting from V2Vor V2R communication with multi-hopping
technologies
Driver’s sate can be refined to two types. One is location
context where driver know well or not. Location context isrelated to look the road carefully or not. Another is driver’s
mentalities which are fatigue, sleepiness, and etc.
Table 3 shows a part of state transition table. First, wecollect many contexts and categorize them. Left side of table
is wide categories. Three columns of right side are example
of events. In this table, these events are caused by humanbehavior. The events defined in this table are only caused
during driving.
There are many other types of events which are watchingnavigation, operating smartphone, front vehicle’s braking,jumping out in front of the vehicle, car horn and etc.

5.2 Scenarios
We prepared three scenarios for the two types of scenario.
Moreover, we prepared the scenario for preliminary
experiment. These are as follows:(1) Preliminary scenario
After 4 minutes driving, go to the mountain road andbikes move suddenly(2) Sudden braking scenario
 Route1:After driving 300 m, front vehicles suddenly
brakes.
 Route2: front vehicles suddenly brakes on thestraight road
 Route3: after waiting the traffic signal and start,frontvehicles suddenly brakes.

Figure:1 driving simulator
10International Workshop on Informatics ( IWIN 2018 )Table 3: Example of state transition table
Press theevents
Acceleration
pedal

status

Go straightHigh speed

Turn right
Turn left
Go straightnormal speed

Turn right
Turn left

Status ofvehicleStatus inmotion

Going
ahead

Go straight
Low speed

Turn right
Turn left
Go straightSlow speed

Turn right
Turn left

Decreasing
speed

Go straight

5.3 Experiments

6

We experimented for Six men and two women. 4
men drive frequently and one of them has a driver’s license
but no experience. After training of operation for driving
simulator, we experimented. We gathered various
information as follows: Simulation time Velocity of the vehicle
 Steering angle
 Acceleration
 Braking Location
 Distance of stop after braking

High speed,gostraight
High
speed,turn
right
High
speed,turn left
Normalspeed,gostraight
Normal speed,
turn right
Normal speed,
turn left
Slow speed,gostraight
Slow speed,
turn right
Slow speed,
turn left
Normal speed,
go straight

Releaseacceleration
pedal
Decreasing
speed, gostraight
Decreasing
speed, turn right
Decreasing
speed, turn left
Decreasing
speed, gostraight

Brinker for rightturnDecreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker

Decreasing
speed, turn right

Decreasing speed,
turn right, brinker

Decreasing
speed, turn left
Decreasing
speed, gostraight
Decreasing
speed, turn right
Decreasing
speed, turn left
Slow speed, go
straight
Slow speed, turnright
Slow speed, turnleft
Decreasing
speed, gostraight

Decreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker
Slow speed, turnright, brinker
Slow speed, turnright, brinker
Decreasing speed,
turn right, brinker
Decreasing speed,
turn right, brinker

EVALUATION AND DISCUSSION
6.1 How to evaluate

We evaluate by comparing two experiments. one is thedata of driving normally. Another is the data of driving with
operating the smartphone. We focus the data of the distanceof stop after braking. If the decision of the braking isdelayed, there is sudden braking. The distance of stop after
sudden braking is shorter than normal braking.
Figure 2 shows the response time of three routes insudden braking scenario. Table 3 shows the distance fromthe front vehicle and the distance for stopping after thesudden braking. These data is average data of all subjects.
Figure 3 shows the deference of the driver who is beginner
or not.

11

International Workshop on Informatics ( IWIN 2018 )4.50

6.2 Discussion

4.00

反応時間[s]Responsetime [s]3.50

We should evaluate the dangerous degree. Therefore, wehave to define the dangerous degree. Dangerous degree
should be the percentage of accidents in the situation. In
many cases, the percentage is very small. If the all vehicle
keep the rule of traffic, there are a few accidents. Moreover,accidents are raised by which both vehicles do not keep the
rule.
Then we define the dangerous degree with the percentageof the accidents of the area and driver’s status which we try
to define by this experiment.

3.00
2.50
2.00
1.50
1.00
0.50
0.00

ルート１
Route
1

Route 2
ルート２

Route 3
ルート３

Nomal
通常運転

2.16

1.44

1.42

ながら運転

4.19

1.84

2.31

With smartphone

7
Figure 2. Response time of sudden braking

We introduce the driver’s contexts and we experiment thedangerous degree is depend on the contexts. However, weshould experiment many cases and we get the true degree of
dangerous. So, we are planning to evaluate it by sensors
which measured the blood flow in the head of each part.

Table 3 distance of stopping and distance from front vehcile
Route1
NormalWith smartphone
Distance for
42.2
61.23
stopping [m]Distance from
15.71
35.92
front vehicle

NormalDistance forstopping [m]Distance fromfront vehicle

33.65

1.89

7.33

NormalDistance forstopping [m]Distance fromfront of vehicle

REFERENCES[1] NHTSA, https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety[2] Institute for Traffic Accident Research and Data
Analisys (ITARDA), http://www.itarda.or.jp/english/[3] Yuhei Ikeda, Ryota Horie, Midori Sugaya, “Estimating
Emotion with Biological Information for Robot
Interaction,” Procedia Computer Science, Procedia
Computer Science (2017)[4] Han, W., Yang, Y., Huang, G.-B., Sourina, O., Klanner,
F. and Denk, C. .Driver Drowsiness Detection Basedon Novel Eye Openness Recognition Method andUnsupervised Feature Learning, Systems, Man, andCybernetics(SMC),
2015
IEEE
International
Conference on, pp.1470-1475 (2015)[5] Yang, G., Lin, Y. and Bhattacharya, P. . A Driver
Fatigue Recognition Model Based on Information
Fusion and Dynamic Bayesian Network, Special Issue
on Intelligent Distributed Information Systems, Vol.
180, p.19421954 (2010).
[6] Ji, Q. and Yang, X. . Real-Time Eye, Gaze, and Face
Pose Tracking for Monitoring Driver Vigilance, RealTime Imagin, Vol.8, pp. 357-377 (2002).
[7] Lee, B. –G. and Chung, W.-Y. . A Smartphone-Based
Driver Safety Monitoring System Using Data Fusion,
Sensors, Vol, 12, pp. 17536-17552 (2012).
[8] Seiji Matsuyama, Yuichi Tokunaga and Ryozo
Kiyohara, “Recognizing User Location Context for Car
Navigation Devices,” IEEE International Symposiumon Consumer Electronics (ISCE) (2016)[9] Hiroto Furukawa, Ryozo Kiyohara, Yuich Tokunaga,
Masashi Saito, “Vehicle Control Method at TJunctions for Mixed Environments Containing
Autonomous and Non-Autonomous, “ IEEE
International Conference on Advanced Information and
Network Application (AINA) (2017)

Route2
With smartphone

28.05

Route3
With smartphone

29.05

43.62

9.55

22.87

6.00
反応時間[s]Responsetime [s]

5.00
4.00
3.00
2.00
1.00
0.00

ルート１

ルート２

(1)
通常運転(一般運転者)

1.21

1.46

1.38

(2)
ながら運転(一般運転者

3.41

1.04

1.39

(3)
通常運転(熟練運転者)

3.11

1.42

1.49

(4)
ながら運転(熟練運転者)

4.98

2.63

4.16

CONCLUSION

ルート３(1) Beginner drives normally(2) Beginner drives with operating the smartphone(3) Driver who has experience drives normally(4) Driver who has experience drives with operating the sma

12International Workshop on Informatics ( IWIN 2018 )A method to synchronize movie and acceleration sensor data using directions ofvectors determined by corresponding points between video frames
Yosuke Ishiwatari* **, Takahiro Otsuka*, Masahiro Abukawa*, and Hiroshi Mineno***

Information Technology R&D Center, Mitsubishi Electric Corporation, Japan**
Graduate School of Science and Technology, Shizuoka University, Japan{Ishiwatari.Yosuke@dr, Otsuka.Takahiro@dw, Abukawa.Masahiro@bx}.MitsubishiElectric.co.jp,
mineno@inf.shizuoka.ac.jp
Abstract - Data acquisition and analysis systems using
plural sensors are gaining popularity owing to the diversity,
ongoing miniaturization, and inexpensiveness of sensors.
These systems use two or more sensors to acquire sensor
data. It is therefore important to synchronize different sensor
data to analyze. In this paper, we propose a method forsynchronizing video data and acceleration data from amoving car. We evaluate the performance of our method byusing video data and acceleration data acquired using asmartphone and by extracting the intervals when a car turns
right or left as synchronization points. The error found is
1.12 frames using this approach. We intend to expand andfurther optimize our methodology by extracting data fromdifferent scenarios.
Keywords: multimodal, data synchronization, motion
estimation of a vehicle

1of time, as described below; however, our target is notsuitable for employing that method.

2 DETERMININGSYNCHRONIZAIONPOINTS USING IMAGE FEATURES AND
CHARACTERISTICS OF ACCELERATION
DATA
2.1

Our Target

In this paper, we aim to synchronize camera video andacceleration data, both which are recorded by sensors on acar. Each sensor is connected to different systems. We willthen perform synchronization after all data are acquired (Fig.
1). Each system’s time is approximately similar, and thedifference in time between the systems is not known.

INTRODUCTION

Data acquisition and analysis systems using plural sensors
is gaining popularity owing to the diversity, ongoing
miniaturization, and inexpensiveness of sensors.
Autonomous driving is one of the applications using thisapproach. In an autonomous driving car, different sensors
are used [1][2]. Even within common household vehicles,
some utilize dashboard cameras with a GPS receiver.
Vehicles that do not possess any sensors are therefore rare.
However, sensors used are not commonly connected into
one system. In this situation, multiple systems are therefore
used for acquiring data.
Synchronization between the acquisition times of different
sensor data is very important for analyzing the relationshipbetween the data to find correlation values. If sensors are not
set in one systems, a synchronization method is needed foranalyzing the relationship between the data. In general, a car
does not have such a synchronization method for a user, so
an extra system must be added.
A typical correction method is to synchronize system times,
but as each system clock is different, system times become
incorrect after multiple synchronizations over a long periodof time. Furthermore, an extra cost for using external time
for synchronization is added, for example, a cost for GPS
receivers for using GPS time.
In this paper, we propose a method for synchronizing
sensor data by extracting the data ranges of different car
motions through analysis of the characteristics of sensor
data while not recording time. In [3], a method used acorrelation value between sensor data without consideration

Figure 1: Our target
In this situation, the difference in time when sensor data
were acquired cannot be obtained. In addition, each system
clock is different; thus the differences in time between
camera and acceleration data are not always constant.
Therefore, if the synchronization is performed at one data
point, it does not necessarily mean that all data points can besynchronized in a similar way.
Camera data and acceleration data have different
characteristics, therefore same reference points related toboth characteristics of the two data types are important forsynchronization. Therefore, we propose a synchronization
method via the detection of car motion behavior using
sensor data and by matching the behavior ranges of these
data.

2.2

Target Car Motion Events

As described previously, our method uses plural
synchronization points. Thus, car motion events that13

International Workshop on Informatics ( IWIN 2018 )determine the synchronization points must be easy to detectand appear frequently while driving. In this paper, we wish
detect events wherein a car is turning right or turning left.
We will then determine the points at the beginning or end of
the events as synchronization points.

a method to synchronize camera images and accelerationdata.

2.5.1 Overview
Our method comprises two functions. The first is afunction that detects ranges of the frame that indicates the
car is turning right/left. The other is a function that
calculates the difference between the ranges and the range ofacceleration data by searching points that match the desired
data. To reduce the search range, as described in section 2.1,
the times of the systems are approximately similar and thedifference in time is not known; however, the start of thesearching point can be determined (the point that matches
the point of the other sensor if the difference is zero).

2.3 Detecting “Turning Right” and “Turning
Left” Events Using Image Features
We used the optical flows of image features for detecting
the behaviors of the car as “turning right” or “turning left”(“turning right/left”) from camera images because theoptical flows of image features that are on stationary objects
show vectors that are opposite to the vector of a moving car.
We can calculate the tendencies of the vectors from theoptical flows of image features using whole frames. These
image features are not solely on stationary objects; however,there are not many objects that move around the car, so thetendencies of the vectors that can be regarded as a vector are
the same as those of a vector that shows the movement of
the car. When a car is turning right/left, the optical flows
from the image features on stationary objects are opposite to
the direction in which the car is moving. If a camera isrecording in front of a car and when a car is turning right,
optical flows turn left. However, when a car is turning left,
optical flows turn right. Meanwhile, the vertical direction of
the optical flow varies with the position of each image
feature within the camera image. As described in Fig. 2 as
an example of “turning left,” each vertical direction of the
optical flow varies from each other.

2.5.2 Detecting the range of frames
As described in section 2.3, in camera videos, a “turning
right/left” event has characteristics such that the opticalflows tend to turn left/right. As such, we use thesecharacteristics for detecting the ranges of the frames. We
detect the ranges based on the functions as follows (Fig. 3):

Figure 3: Flow of matching time between
acceleration data and movie data
1) Obtain the optical flows of the image features between
successive frames2) Calculate the tendency of the vectors of the opticalflows by classifing the vectors into 16 bins based on
the direction of the vector, and select the bin that
includes the majority of the vectors
3) Calculate the range by counting the frames in which
the bin of the start frame is the left binor the right bin.
If the bin is near (within 2 bins) the former frame in acontinuous fashion

Figure 2: Optical flows of a “turning left” event

2.4 Detecting “Turning Right” or “turning
Left” Using Acceleration Data
In this paper, we hypothesize that we can acquire
acceleration data in the crosswise direction. When a carturns right/left, a driver operates the steering wheel to movein the crosswise direction. Thus, this operation is equal toaccelerating the car in the crosswise direction, so the start ofthis operation causes a significant change in acceleration.
This means that this operation can be detected using thedifference in acceleration. We then used the difference inacceleration to detect the start or the end of the operation by
detecting a peak or an inflection of the acceleration.

2.5

2.5.3 Correcting separated situations
In [4], we discussed how an incorrect detection of a turning
right/left event that was divided for more than one situation.
This was caused by another action occurring simultaneously
when a car is turning right/left, e.g., riding on a curb andheavy braking. If the number of frames between two of the“turning right/left” events is very small (a few frames) and if
these situations are the same (i.e., these situations are“turning right” and “turning right” and vice versa), thesesituations should be regarded as one situation (Fig. 4).

Detecting synchronization points

Following the ideas for detecting the “turning right/left”
events from camera images or acceleration data, we propose
14

International Workshop on Informatics ( IWIN 2018 )immediately before the point X, and the difference (Δd2)
between the point X and point X2, that point is the startframe of the range of the “turning right/left” event that
occurs immediately after than the correcting point X, areused. The time difference ΔX at point X is calculated asfollows:
ΔX =𝛥𝛥𝑑𝑑2 ∗�(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝑋𝑋)−(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝑋𝑋1 )�+𝛥𝛥𝑑𝑑1 ∗((𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑋𝑋2 )−(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝑋𝑋)).
(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝑋𝑋2 )−(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝑋𝑋1 )3 FUNDAMENTALOUR METHODEVALUATIONOF

We evaluated the fundamental accuracy of our method.
The data and our evaluation method is described as follows.

3.1

Figure 4: An example of separating one situation“turning right” into two situations

We used the camera video and the acceleration dataacquired by a smartphone in a car. The camera recorded thefront view of the car. The acceleration sensor recorded theaccelerations in three dimensions: the direction in which the
car moves, the crosswise direction of the car, and thevertical direction of the car.
For evaluation, the recorded time was also acquired forthese data (camera frames and acceleration data). The
difference after employing our method for these data isequal to the accuracy of our method.

2.5.4 Matching the range of frames into rangesof the acceleration data
After calculating the range of the frames, as describedabove, the matching points between the start/end frame ofthe desired range and the acceleration data are calculated.
As described in section 2.4, a peak or an inflection point of
the acceleration data is searched. A peak or an inflection
point of the acceleration data closest to the start point isregarded as the corresponding point of the start/end frame.
After calculating the corresponding point of the start frame(point C1) in the acceleration data (point A1) and thecorresponding point of the end frame (point C2) in the samedata (point A2), A1 and A2 are corrected to have the samerange of time between the range from C1 to C2 and the range
from A1 to A2. In detail, point A1 is moving to A1′ and point
A2 is moving to A2′. Therefore, “(time of A2′) − (time of A1′)
= (time of C2) – (time of C1)” and “(time of A1) − (time ofA1′) = −[(time of A2) − (time of A2′)].” This means that A1′
and A2′ are calculated as follows:Δtc = (time of C2) − (time of C1)
Δta = (time of A2) − (time of A1)
diff = (Δtc − Δta)/2
time of A1′ = (time of A1) − difftime of A2′ = (time of A2) + diffHence, (time of C1) − (time of A1′), i.e., D1, means thedifference in time between the camera video and theacceleration data at the start frame of the range and (time ofC2) − (time of A2′), i.e., (D2), means the difference in time
between the camera video and the acceleration data at theend frame of the range. D1 and D2 are not always identical.
Therefore, the difference value ΔE at time E (E is the time
between A1′ and A2′) is calculated as follows:
ΔE =

3.2

Data Preprocessing

To reduce noise in the acceleration data, a smoothing
process was employed. In addition, an interpolation process
was used to adjust the sampling rate of the acceleration data
to the interval between the frames of the camera video (30
fps).

3.3

Range of Searching the Matching PointAs described in section 2.5.4, a peak or inflection point in
the acceleration point is searched from the start. During
evaluation, the range of search was defined for 30 samples(equal to 1 s).

3.4

Evaluation Target Range

Using the method that is described in sections 2.5.2 and2.5.3, we obtained the ranges from the camera image. The
ranges included the “turning right/left” events, but someevents that were not regarded as the target event were
included. The desired target events were also not included.
For example, the event was a car avoiding an electric pole.
In this situation, the car was moving in the right/left
direction opposite to the pole after passing through the pole.
Events that were not the desired targets were manually
removed, and 20 events were used for evaluation.

𝐷𝐷1 ∗��𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝐴𝐴′2 �−(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝐸𝐸)�+𝐷𝐷2 ∗((𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝐸𝐸)−�𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝐴𝐴′1 �)�𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝐴𝐴′2 �−(𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑜𝑜𝑜𝑜 𝐴𝐴′1 )Data for Evaluation.

2.5.5 Synchronizing the range other than“turning right/left”

3.5

For synchronizing the point (point X) in the range otherthan a “turning right/left” event, the difference (Δd1)
between the point X and point X1, that point is the end
frame of the range of the “turning right/left” that occurs

Evaluation Results

We evaluated the 20 situations, as described above, by
calculating the difference values between the start timecamera frames and the synchronized time of the acceleration
15

International Workshop on Informatics ( IWIN 2018 )data during “turning right/left” events (after employing our
method).
The evaluation results are summarized in Table 1. In Table
1, a unit of error value is “frame” (1/30 s). The average oferror value is 1.12 frames, and the standard deviation is 0.81
frames.

that determine accuracy of our method, but we think
the interval time and accuracy is not necessarily inproportion, because if the interval time is large,
optical flows are mainly determined from thebehavior of car, so some noise, for example, verysmall motion of car or other objects, are lessinfluenced for optical flows. So, it is important todetermine appropriate interval time.
4) Expanding our method for other situations
Our method uses all the optical flows in the frame tocalculate the tendency of their direction. The “turning
right/left” event is an appropriate situation to be
detected using the method described above. However,
our method is not suitable for some situations, e.g.,“moving straight.” In this situation, all the opticalflows do not tend to turn in the same direction. The
direction is determined according to the point of theimage feature within the frame.
To expand our methodology, we can split a frame
into subframes and calculate the tendencies within
the subframes, followed by detection of the range
based on the characteristics of the tendencies.

Table 1 Evaluation results

No. Difference of frames
1
1
−1
2
1
3
−1
4
0
5
−0.5
6
0.5
7
−0.5
8
−0.5
9
2
10
1
11
−1.5
12
2
13
1
14
−2
15
−0.5
16
−0.5
17
0
18
−1
19
0
20

3.6

Absolute value ofdifference
1
1
1
1
0
0.5
0.5
0.5
0.5
2
1
1.5
2
1
2
0.5
0.5
0
1
0

4

CONCLUSION

In this paper, we propose a method to synchronize acamera video and acceleration data onboard a moving carthat are acquired from different systems. In our method, we
calculated the synchronization points by determining a“turning right/left” event from the camera image and theacceleration data. From the camera image, we use thetendency of optical flows of the camera frame to detect therange of the event by detecting the specific tendency
continuously. From the acceleration data, we detected thesituation by identifying the peak or inflection point of theacceleration.
We evaluated the fundamental performance of our method
using the camera image and the acceleration data acquired
from a smartphone in a car, and the average of error frame
was 1.12 frames. However, we have some issues to beresolved.

Discussion

In the 20 situations described above, the average of theerror frames is short, so it can be concluded that our
methodology accounts for the data adequately. However, wehave to consider some issues:
1) Handling the difference between Δta and Δtc
In our method, the difference between Δta and Δtc
is divided equally and employed for A1 and A2.This
is not always true as the difference between A1′ andA1 is not always the same as that between A2′ and A2.
We can potentially resolve this issue by matching thecorrelation value.
2) Problems associated with insufficient number ofimage features
Some frames do not possess multiple image features.
For example, the acquisition time is at night, andtherefore, the frame has dark features. A frame is also
occupied by the sky or the ground with no lines, signs,
and other objects.
3) Tradeoff between accuracy of our method and framerate
We calculated optical flows between successiveframes (interval time between those frames is
0.33sec). The interval time is one of key parameters

REFERENCES[1] “Autonomous long-distance drive”, Mercedes-Benz,
https://www.mercedes-benz.com/en/mercedesbenz/innovation/autonomous-long-distance-drive/[2] Google Self-Driving Car Project,https://www.google.com/selfdrivingcar/[3] S. Tanaka, et al., “Study of Time Synchronization
Method between Multiple Wearable Sensors”, the 13thof the S O F T K y u s h u C h a p t e r A n n u a l
C o n f e r e n c e . (2011).
[4] Y. Ishiwatari, et al., “A Synchronization Method
Between Movie and Sensor Data Using Directions of
Vectors Determined by Corresponding Points Between
Video Frames”, IPSJ SIG Technical Reports, 2018ITS-73. (2018).

16

International Workshop on Informatics ( IWIN 2018 )A Method for Estimating Degradation Level of Road Markings
by Participatory Sensing with In-Vehicle Camera
Yoh Shiraishi* and Issei Suga**

Graduate School of Systems Information Science, Future University Hakodate, Japansiraisi@fun.ac.jp

Abstract - The total road length in Japan is increasing
year by year. On the contrary, the budget for roadmaintenance is decreasing in recent years. Efficient roadmaintenance and management are required to maintain roads
with less financial resources. Due to the direct influence by
vehicles, road markings are extremely degraded. Currently,
the authorities concerned conduct visual examination or
investigation by dedicated vehicles to assess the degradationlevel of road markings. However, such methods are
expensive and inefficient. Accordingly, it is essential toestimate the degradation level of road markings at low costand efficiently. Existing studies estimate the degradationlevel of road markings from the road surface image capturedby the in-vehicle camera. However, these methods havehigh installation cost because the cameras used in thesestudies are not common.
Therefore, this paper proposes a method to estimate thedegradation level of road markings using a driving recorder
or a smartphone as an in-vehicle camera. Also, this study
adopts participatory sensing to collect road surface images
efficiently and widely. The proposed system scans thecaptured image, and extracts the part including a roadmarking based on the painted ratio. The method estimates
the degradation level of the extracted marking by comparing
the template image without degradation. We conducted theexperiments in order to examine the basic performance ofthe proposed method, and the experimental results suggested
the potential and availability of the proposed method.

due to the road damage in a short period. Also, in the snowy
region, road markings remarkably peels in winter due totraveling using the tires chains such as buses and trucks andthe repeated snow removal work by the dedicated vehicles.
Currently, road management operator visually inspects
such damage and peeling of road markings or inspects themby using the dedicated vehicle. However, such inspection
has high cost and inefficient in information collection fromthe view point of road maintenance. From the view point ofdrivers who use road, it is necessary to grasp road markings
or road signs in order to operate their vehicle safety andcomfortably. Information such as the position of the stop
line and the traffic classification by travelling direction isimportant on unfamiliar roads. However, it is difficult toconfirm the road marking when the road marking is peeledoff.
In this study, we adopt participatory sensing [1] as an
approach for collecting the degradation level of road surface
markings effectively, and share the information about roadmarkings provided lots of drivers. In this paper, we propose
a method to estimate the degradation level of road surface
markings as an indicator of how degree the road marking
deteriorates, by using the shot images from an in-vehicle
camera. We assume that many drivers can use the proposed
method by using a driving recorder or a smartphone whichbecomes popular as an in-vehicle camera.

2

As researches using in-vehicle cameras, the related works
for recognizing road markings [2, 3] and for estimating thedegradation level of road markings [4, 5, 6].
The works [2, 3] recognize road markings by transforming
the shot image to the bird’s eye view. When shooting theimage including multiple road surfaces by using the invehicle camera, the size of the road marking is different
depending on the distance from the shooting vehicle to thetargeted road sign. By this transformation, the size of theroad markings in the image becomes constant irrespective ofthe distance from the shooting vehicle. Consequently, it iseasy to apply various kinds of image processing forrecognizing road markings. We think we can apply thistransformation [2, 3] into our study.
The works [4, 5, 6] estimate the degradation level of roadmarkings with simple shape. These works use specificcamera such as a single lens reflex camera or a sphericalcamera. The work [4] shoots the image ahead of the vehicle,
and estimates the degradation level of line for road partition
by adapting the template matching mechanism to the shot

Keywords: Road surface marking, degradation level, invehicle camera, participatory sensing, Intelligent Transport
Systems.

1

RELATED WORKINTRODUCTION

Roads in Japan continue to grow year by year, and thetotal length of these roads exceeds 1,270,000 km as of April
1st, 2016. On the other hand, the road project cost is a peakaround 1998 and has been decreasing in recent years. It isrequested to maintain and manage more efficient roads inorder to maintain and manage many roads with limited
financial resources.
Roads have various kinds of markers such as road surface
makings and road signs, and various kinds of structure suchas bridges and tunnels. There are short-term deteriorationsuch as the paint peeling of markers on roads, and long-term
deterioration such as the collapse of the tunnel and the rustof the bridge. Especially, road markings are directly affectedby the tires running vehicles, and the deterioration is intense17International Workshop on Informatics ( IWIN 2018 )image. The line for road partition indicates center line inroad and roadway outside line. The work [5] installs a 4K
camera on the passenger side of the front bumper of thevehicle, and estimates the degradation level of roadmarkings. The work [6] installs a spherical camera at thebottom of the vehicle. These works need to estimate
complete state of the targeted road marking from the shot
image in order to calculate the degradation level of the roadmarking. It is easy to estimate only road markings withsimple shape such as road partitioned line. However, it isdifficult to apply the estimation of the degradation level ofvarious kinds of road markings that this study will tackle.
Also, the installation cost is high because it is necessary to
install the camera device not to fall on roads while vehicle
traveling.

3

effectively rather than using the dedicated vehicle for roadmaintenance.
For solving the Problem II, we focus on the rate of white
pixels in the extracted road area. We need to recognize
where the target road signs are on road, and extract the roadarea for the next image processing: identification of roadmarkings. Therefore, we slide the particular window along
the road, and calculate the rate for each window. If thewindow includes many white pixels (the rate is high), there
is high probability that the window contain a road marking.
For solving the Problem III, we use SIFT (Scale-Invariant
Feature Transform) feature quantities and BoVW (Bag ofVisual Words) model for identifying road markings. As theimage processing after recognizing road markings, we need
identify which road markings the extracted markings are.
SIFT is one of representative feature quantities for similarity
measurement among images, and can classify the target
images with high accuracy even if there are the difference insize and angle between the targeted image and template
images. We express each road marking as distribution
representation based on the calculated SIFT feature
quantities. The representation is a BoVW model thatconsists of vectors (visual words). The BoVW model isadaptation of BoW (Bag of Words) for image domain. We
can express a sentence as vectors in the BoW model. In thesame way, we can express an image as vectors in the BoVW
model.

PROPOSED METHOD
3.1 Research purpose

The purpose of this study is to estimate the degradationlevel of road markings by using the images from in-vehicle
camera. We use driving recorders and smartphones as invehicle cameras because these devices are becoming popular.
We tackle the estimation of the degradation level of various
kinds of road markings: line for lane partition, speed
limitation, lane classification by traveling direction andapproach of pedestrian crossing. In this paper, we focus onlane classification by traveling direction that is composed ofseveral arrows such as straight, right and left.

As an approach for solving the Problem IV, we compare
the rate of the painted occupancy for each road markings.
The related works [7, 8] can estimate the degradation levelof only simple road segmented line due to the characteristicsof the processing algorithm. In this study, we tackle on
estimation of the degradation level of various kinds of roadmarkings, and cannot adopt the previous methods [7, 8] forour targets. Therefore, we prepare the template for each roadmarking without damage (namely, whole painted), andestimate the degradation level of the targeted road sign bycomparing the extracted road marking in the camera image
with the template.

3.2 Problems and approaches
In order to archive our purpose described in the previous
section, we need to solve the following problems:
Problem I: Reduction of the installation cost of devices.
Problem II: Recognition of road signs
Problem III: Identification of road signs
Problem IV: Calculation of the degradation level of roadsigns
As an approach for solving the Problem I, we use a driving
recorder and a smartphone with the high penetration rate as
an in-vehicle camera. We need to collect the image
including the targeted road signs in wide area. If we use lotsof vehicles with in-vehicle cameras, we can collect theimage about roads in wide-area effectively. Recently,
general drivers often have smartphones and are setting
driving recorders in their vehicles. Specific cameras used in
the reference [6, 8] are not easy to set the vehicle because
these cameras are not designed for in-vehicle use. By
adopting smartphones and driving recorders as in-vehicles
cameras, lots of general drivers will participate with datacollection. Accordingly, the introduction of commerciallyavailable smartphones and driving recorders reduce theinstallation cost for the devices for collecting the images
with road markings. Participatory sensing is a powerfulapproach for collecting the road images in city scale

3.3 The details of the proposed method
3.3.1 An overview of the proposed system
The proposed system consists of the part for acquiring
camera images and the part for estimating the degradationlevel of road markings. The first part dedicates to shootimages about road markings by using a driving recorder anda smartphone set on dash-board in each vehicle. The second
part estimates the degradation level of road markings byrecognition and identification of the targeted road markingsfor the shot image. The image processing part consists of thefollowing procedure:(1) Pre-processing before recognizing road markings(2) Recognition and identification of road markings(3) Estimation of the degradation level of road markings
18

International Workshop on Informatics ( IWIN 2018 )If lots of drivers use the proposed system, we can estimate
the degradation level of road markings on many roads inwide-area. The estimated results are recorded in a databaseincorporating with location information. Consequently, we
can grasp the degradation level of road markings on a map,
and will be also useful for road maintenance.

Figure 3: The result of bird’s eye viewtransformation (This image is the transformation
result from the image of Figure 2)

3.3.2 Pre-processing before recognizing roadmarkings

3.3.3 Recognition and identification of roadmarkings

There are two kinds of pre-processing before recognizing
road markings. One is image binarization and another isbird’s eye view transformation. By image binarization, we
can expect to extract the area including road markings fromthe in-vehicle camera image because most of road markings
are painted with white color. Figure 2 shows the result of thebinarization of an example image shown in Figure 1.
We transform the binarized image to bird’s eye view. The
bird’s eye view transformation is to transform like an image
taken from directly above by expanding the pixels far fromthe shooting vehicle. If the original image includes multiple
road markings with different distance from the shooting
vehicle, the size of the markings will be different and theshape will be distorted. By using the bird’s eyetransformation, the size of road markings in the shot image
can keep the same. Also, it is easy to recognize roadmarkings because the transformation adjusts the distortion in
the position and angle of the in-vehicle camera. Figure 3
shows the result of the bird’s eye view transformation for
the image shown in Figure 2.

Our method recognizes road markings while performing
the sliding window processing over the image transformed
to bird’s eye view. Concretely, we extract the area that is
highly possible that it contains a road marking.
We extract the partial image from the transformed view byusing the sliding window, and search the position having the
maximum number of white pixels while sliding the window.
The position having the maximum number of white pixels is
the position where it is high possibility to exist roadmarkings.
Next, we calculate SIFT feature quantities from theextracted area including a road marking, generates BoVW
vectors and identify road markings based on the similarity
among the BoVW representations.

3.3.4 Estimation of the degradation level ofroad markings
We compare the ratio of the painted occupancy in order toestimate the degradation level of road markings. The ratio of
the painted occupancy is an index indicating how degree
painting of the road surface marking occupies the targetedarea.
First, we prepare the template image for all targeted roadmarkings, and calculate the ratio of the painted occupancy
ptmp for the template image in advance. The template imageis the image including complete road marking without
degradation. Next, it calculates the total number of pixels
pxall in the area where the road marking exists, and the white
pixel number pxwhite of the road marking. We calculate theabsolute ratio of the painted occupancy by using the total
number of pixels pxall and the number of white pixels pxwhite.
Finally, we calculate the relative ratio of the painted
occupancy prel by comparing the absolute ratio of thepainted occupancy pabs with the ratio of the painted
occupancy for the template image ptmp. In this study, weregard the relative ratio prel as the estimation results of thedegradation level of the road marking.

Figure 1: An example of images from in-vehicle
camera4

EXPERIMENTAL RESULTS

4.1 The experiment about identification ofroad markings
In this section, we show the result of the experiment ofidentification of road markings. In this paper, we focus ontraffic classification by traveling direction as the

Figure 2: The result of binarization for the imageof Figure 1
19

International Workshop on Informatics ( IWIN 2018 )representative road markings. As shown in Figure 4, there
are 5 types of arrows indicating the traffic classification.
The indication for left turn is Arrow L (Figure 4 (a)), theindication for straight and left turn is Arrow SL (Figure4(b)), the indication for straight and right turn is Arrow SR(Figure 4(c)), the indication for right turn is Arrow R(Figure 4(d)) and the indication for straight is Arrow S(Figure 4(e)).

largely degradation and Figure 5(c) shows the arrow withintensely degradation.

(a)Slightlydegradation(b)
Largelydegradation(c)
Intensely
degradation

Figure 5: Examples of road markings withdifferent degree in degradation

(a)
Arrow
L

(b)
Arrow
SL

(c)
Arrow
SR

(d)
Arrow
R

In this experiment, we use Figure (a) as the template image,and calculated the degradation level of the targeted roadmarking (Arrow R) described in Section 3.3.3. Table 2
shows the estimation result for the images (b) and (c) in
Figure 5. This result suggests the proposed method canadequately estimate the degradation level for road markings.

(e)
Arrow
S

Table 2: The result of estimation of the degradation level

Figure 4: Examples of road markings (arrows for
traffic classification) that we used in the
The SIFT feature quantitiesi for each image were calculated

Image type
Absolute ratio of
the painted occupancy
Relative ratio of
the painted occupancy
The degradation levelof the road marking

and these images were classified. Table 1 shows theclassification result. We used the leave-one-out method as
cross validation.
Table 1: The result of classification of arrows

(a)

(b)

(c)

5.85

4.15

2.41

(100)

70.9

41.2

(0)

29.1

58.8

(unit: percentage)
label
L
SL
SR
R
S
Precision

L
10
0
0
3
0
0.769

The Estimated value
SL
SR
R
1
2
7
15
15
0
14
18
3
1
12
7
0
0
1
0.484 0.383 0.389

S
0
0
0
7
29
0.806

Recall
0.500
0.500
0.514
0.233
0.967

5

CONCLUSION AND FUTURE WORKS

This paper proposed a method for estimating thedegradation level of road surface markings by using theimages from in-vehicle cameras. Our methods transform theimage to the bird’s eye view, and extract the area including
the targeted road markings by sliding window processing.
The degradation level of road markings is calculated on
based on the ratio of the painted occupancy. We conducted
the preliminary experiment by using representative (butsimple) road markings and discuss the potential of the
proposed method.
In the future works, we should collect many images
including road markings and estimate other kinds of roadmarkings. The effective extraction method for collecting thetargeted road markings is required. Currently, we calculate
the degradation level of road markings based on the ratio of
white pixel. In the real environment, there are several roadmarkings with colors except white (for example, orange).
We need extend the proposed method to apply such roadmarkings.

The F-measure is 0.541. The estimation result is not good
except the Arrow S. There are many mis-estimation between
the Arrow SL and SR.

4.2 The experiment about estimation of thedegradation level of road markings
In this section, we show the result of the experiment about
estimation of the degradation level of road markings. We
prepare road markings (Arrow R) with different degradationlevel as shown in Figure 5. Figure 5(a) shows the arrow withslightly degradation, Figure 5(b) shows the arrow with20

International Workshop on Informatics ( IWIN 2018 )REFERENCES[1] J. Burke, D. Estrin, M. Hansen, A. Parker, N.
Ramanathan, S. Reddy and B.M. Srivastava,
Participatory Sensing, Proc. of World Sensor Web
Workshop (WSW'06) (2006).
[2] M. Noda, T. Takahashi, I. Ide, Y. Mekada and H.
Murase, Recognition of Road Markings from InVehicle Camera Images using Generative Learning
Method, Technical report of IEICE, Vol.108, No.263(PRMU2008-93), pp.31-36 (2008).(in Japanese)[3] Y. Kemuriyama and K. Onoguchi, Lane Marking
Recognition for Constructing a Multi-information Map,
Transactions of SICE, Vol.49, No.1, pp.25-32 (2013).
(in Japanese)[4] T. Asada, S. Honda and S. Kameyama, The
Development of a Method for Estimating the Peeling
Ratio of Road Partitioned Line by Using Image Feature
Quantities, Journal of Japan Society of Civil Engineers
E1 (Pavement Engineering), Vol.67, No.1, pp.10-21(2011). (in Japanese)[5] S. Nishino, K. Wakayama and H. Kawanaka and K.
Oguri, Estimation of the Stripping Ratio from
Projection Transformed Images of Lane Markings
Using an In-Vehicle Camera, Technical report of
IEICE, Vol.114, No.508, pp.17-22 (2015). (inJapanese)[6] T. Kawazaki, T. Uchikoshi, T. Iwamoto, M.Matsumoto,
T.Yonezawa, J. Nakazawa and H. Tokuda, A Method
for Detecting Friction on Road Markings by Using
General-purpose Cameras and General Vehicles, IPSJSIG Technical Report, Vol.2015-ITS-60, No.2, pp.1-8(2015). (in Japanese)

21Session 2:
IoT Systems( Chair: Ryozo Kiyohara )International Workshop on Informatics ( IWIN 2018 )A Proposal of PC Facilities Utilization State Management System
in Company Office or Educational Institution by using the IoT Technology
Keiichi Abe, Kurika Kobayashi, Kouki Iizuka, Masao Isshiki
Kanagawa Institute of Technology, Atsugi City, Kanagawa, Japan
abe@he.kanagawa-it.ac.jp
Abstract –application software used within PC work time. Therefore, unused

At present time, information management systems that canapplications software which have been installed on the PC but are

survey and monitor an office, movement of people in an office,unused cannot be found. In addition, there remains a problem thatsituation of utilization of facilities and so on in real time have been

it is impossible to accurately grasp the usage situation of PC

developed by companies and others using the wireless sensor

facilities including software.

network and IoT (Internet of Things) technology.

Currently, there are many dedicated software products that

The reason is because information systems aimed at finding

monitor and analyze the operation status of PC applications

waste (electricity usage, unnecessary equipment, work etc.) andsoftware, but since those products acquire and manage information

improving work environments are being promoted by companies

retrieved from the log information managed by the OS, they are

and others. Therefore, in this study, we propose a system that adds

occasionally prevented from logging due to a communicationa function that can grasp detailed usage of various applications

interrupt or the like. Another problem is that the period of time

software used within PC work time to the PC management system

where the computer is in the ON state with no user present is

we developed in the past［1］. We evaluated the effectiveness of ourincorrectly counted as the PC work time. Therefore, with those

proposed system by comparison with a piece of commercial

commercially available application analysis software products, it isapplication analysis software.

impossible to accurately grasp detailed usage of the applications

Keywords: Personal Computer Management System, Smartduring the PC working time while accurately managing the work

Tap , Mat Sensor, IoT Technology

state of the PC user.

1

Therefore, in this research, we propose a system that adds a

INTRODUCTION

function that can grasp detailed usage of various applications

At present time, information management systems that cansoftware used within PC work time to the PC management systemsurvey and monitor an office, movement of people in an office,

we developed in the past. We also evaluated the effectiveness ofstate of utilization of facilities and so on in real time have been

our proposed system by comparing it with a commerciallydeveloped by companies and others using the wireless sensor

available application analysis software product (Manic Time).
This paper is organized as follows. Chapter 2 describes related

network and IoT (Internet of Things) technology.
The reason is because information systems aimed at finding

technologies. Chapter 3 gives an overview of the proposed system,

waste (electricity usage, unnecessary equipment, work etc.) andand Chapter 4 details the prototype development. In Chapter 5, we

improving work environments are being promoted by companies

will describe the evaluation results of the prototype. In Chapter 6,

and others.

this research is summarized.

Our research is to realize an information system that enables us

2

RELATED WORKS

to grasp the operation status of PC facilities in offices and findwaste of unused software by accurately monitoring the work

Examples of the conventional techniques for checking thesituation of PC workers inside offices etc. In our past research [1],presence and attendance of users in a PC practice room include an

we developed a system in which a smart tap and a mat sensor nodeattendance system which checks the entry and leaving times by IC

were installed per PC user, and these two pieces of sensor

cards, and an attendance management system which determines

information were acquired by wireless communication.

attendance of users in a lecture based on a database which holds
PC use histories [2] [3] [4] [5].

We showed that we can accurately calculate the four conditionsof PC work time, work time other than PC, wasted electric use time

There are also studies on automated systems for checking the(time when PC is running with no user present), and absence time.

entry and leaving times in a room, such as a system which employs

However, in the system we proposed in the past [1], we could

the iBeacon technology[6], a hands-free system for checking the

not grasp the operating time and frequency of use of eachentry, presence and leaving of users[7], and an attendance checking
system which uses a camera [8] [9] [10] [11].
25International Workshop on Informatics ( IWIN 2018 )These management systems can manage the total number ofThis system uses sensor nodes equipped with a wirelesshours of the presence in the office room, but it is not possible tocommunication function (Wi-Fi, IEEE 802.15.4, etc.). This allows

manage the work time etc of the people actually using PCs.

the system to be installed with a free layout. This makes it easierMoreover, it requires a large-scale construction at the time offor individuals with no technical skills to install the system and, as

introduction, and there is a problem that the introduction costa result, reduces its introduction cost .

increases.

Ⅷ. Contents of information on PC users to be managed
For the management of information on PC users in offices or

There is also an application that monitors the operating status ofsoftware using log data of a personal computer [12] [13] [14]. This

the like, this system should have the following functions:application can acquire both information such as usage time of(1) Grasp the number of PC users

various software and information on the ON / OFF information of(2) Manage the presence status of each PC user.

the power supply of the PC etc. However, this commercially(3) Distinguish between the PC work time and work time otheravailable analysis software cannot record data when an interruptive

than PC by the user, and find the actual use time of the personal

event such as an update program occurs during the PC operation; incomputer.

such a situation, the data-recording stops and it is impossible to(4) Count and manage the time when the personal computer isgrasp the use situation of the PC. Also, because it is not possible torunning with no user present and wasting electricity.

acquire the presence information and it does not always mean that(5) Record the usage time and frequency of use of each softwarea person is working on the PC even if the power supply of the PC

application used within the actual use time of the personalis in the on state, it is considered difficult to measure an accuratecomputerIn our previous system [1], we realized the above functions (1)PC work time.

to (4), but we did not fully deal with (5). In this paper, we describe

Therefore, in this paper, we propose a system that cana management method for (5).

distinguish between “PC work” and “work other than PC” and cangrasp and monitor the usage status of various software in detail

3.2

during the PC working time. We also propose a method that can
reduce the construction cost at the time of introduction.

3

Summary of this system

Section3.2 explains the outline of the proposed system for
managing the PC work situation by users in an office or the like.

SUMMARY OF PC Facilities Utilization StateManagement System

Figure1 shows configuration of smart tap and mat sensor node
installed per PC Worker. This time we developed a dedicatedapplication that detects the software application being used on the

3.1 Design concept

Ⅰ. Installation Locationclient PC. Thereby, The our system can be transmitted information

The system is supposed to be installed in a computer room suchon the application currently used by the user.

as company office or educational institution.

Ⅱ. Personal Computer to be Monitored
The use of desktop personal computers is assumed. Mobile
computers are excluded.

Ⅲ. Person to be monitored
PC users in the office and PC training room.

Ⅳ. Scale of data management

On the assumption that this system will be used in a PC practiceroom etc. in corporate offices and educational institutions, thesystem should manage the maximum number of about 100 people.

Ⅴ．Method of data collection
In this system, sensor data is collected from various sensor nodes
by wireless communication, such as Wi-Fi standards. One datacollection personal computer (host computer) also uses the same

Figure 1: Configuration of smart tap and mat sensor
node installed per PC.

wireless communication standard as the sensor node, and collects
data from various sensor nodes with a dedicated application.

Ⅵ．Number of sensor nodes installed

In order to monitor the operation status of each desktop personal

The number of various sensor nodes may be about 2 nodes forcomputer installed in an office or the like, the smart tap constantly

each PC user.

measures the power of the desktop personal computer being used

Ⅶ．System installation method

and sends the result to the host computer for data collection. Also,

26

International Workshop on Informatics ( IWIN 2018 )in each personal computer, a commercially available mat sensor

PC facilities in the our proposed system By using this system, it isdetects the presence of the PC user, and the detection result ispossible to make detailed breakdown of usage time etc of varioustransmitted to the host computer by wireless communication. A

applications used within PC work time in addition to the four majorcommercially available mat sensor equipped with a wirelessdivisions, i.e. the PC work time, work time other than PC, wastefulcommunication function is called a “mat sensor node” in this paper.

electricity use time, and absence time for each PC user. By doing

We have originally developed the smart tap and mat sensor nodethis, our system can monitor and manage the utilization situation of

for this system. In our proposed system, wireless sensor nodes such

PC facilities installed in a PC room etc of offices and educationalas smart taps and mat sensor nodes are used. A general-purpose

institutions.

wireless communication standard (Wi-Fi, IEEE 802.15.4, etc.) is
used for communication between these wireless sensor nodes andthe host computer for data collection. Therefore, when introducing
this system, its components can be freely laid out, so no wiring
work is required.
Figure 2 shows an situation used in a PC classroom such as an
office and an educational institution. The host computer collects
and manages presence information and PC power consumption
information from each client. In addition, this time we also collectdata on information on the application currently used by the user.

Figure 3:Outline of the utilization analysis image
graph of PC facilities in the our proposed system.
3.3 Calculation method of person's presence and PC working
time [1]In our past research, as a method to grasp details of the work
situation of users' PCs etc., we used the sleep function of the PC in
addition to the two pieces of sensor information acquired from thesmart tap and mat sensor node. By the method, we could accurately
calculate the four states of PC work time, work time other than PC,
From the two kinds of sensor data acquired with the smart tap andmat sensor, we can estimate the activity of the user in four ways as
shown in Table 1.

Table 1: Each sensor information and action estimateof the user.
Figure2:Overview of our system.
In the proposed system, the number of PC users is determined bycounting the user-detection signals from the mat sensor nodes. The
actual use time of each PC (hereinafter referred to as the PC worktime) and other related information are accurately calculated fromthe power-use signal from the smart tap and the user-detection
signal from the mat sensor node. This calculation method will be
described in detail in Section 3.3. This algorithm was implemented
in a dedicated application of the host computer.
In addition, a dedicated application for acquiring information on

Firstly, we understand the ON/OFF state of the PC from thethe application currently used by the user and sending it to the hostpower consumption information of the smart tap. An output of "1"

computer is implemented on each client PC.

This applicationfrom the smart tap shows that the PC is ON, and "0" shows that theenables us to analyze details of the use of the various applications

PC is OFF. In the system, the power consumption threshold is 15

on each PC within the PC usage time.

W. Therefore, if it exceeds 15 W, the output is "1" when the power
consumption exceeds 15 W and "0" when less than 15 W. On the

Figure 3 shows outline of the utilization analysis image graph ofother hand, an output of "1" from the mat sensor indicates a state27

International Workshop on Informatics ( IWIN 2018 )where someone is in the chair, and "0" indicates no one is in therouter. In Chapter 4, the prototype development of application on

chair. It is possible to manage one PC by the two pieces of sensor

client PC, application on host computer, etc. will be described in

information.

detail.
4.1 Application on client PC

When A = 0 and B = 0, the PC is in OFF state and no user isThis time, we developed a dedicated application that detects thepresent, hence "Absence". A = 0 and B = 1 indicates that the PC issoftware application being used on the client PC.

in the OFF state, but because someone is present, it indicates"Work time other than PC". When A = 1 and B = 0, the PC is in the

Figure 4 shows the operation flow of the dedicated application

ON state, but because no one is present, it indicates "Wasted

on the client PC. In this application, software information about

electric use time". When A = 1 and B = 1, since the user is present

which application is displayed on the forefront window is acquired

while the PC is ON, this state is judged to be "PC work time" infrom the OS (Windows) and sent to the data collection PC.
Here, we will describe a more detailed method of obtaining

this study.
When the output of the smart tap is A and the output of the mat

software information. Some software programs are executed as

sensor is B, the output W in the absence state, the output X in the

background processes on the PC. We only need information on theworking state other than the PC, the output Y in the waste electric

software currently operated by the user. For this reason, we decided

utilization state, the output Z in the PC working state, the following

to acquire software information only from the window displayed atformulae (3.1) to (3.4)are derived from Table 1.

the forefront of the computer screen in the application developed
this time. because the unused application may be got when multiple――

W = A・B―

X = A・B―

windows are displayed on the computer screen.

（3.1）

Therefore, there is a problem that application information which（3.2）

Y = A・B（3.3）

Z = A・B（3.4）

is not actually used is also acquired, in this Application of clientPC, only the application information of the window used at theforefront is acquired and sent to the host computer.

Our system defines the output Z of equation (3.4) as PC
working, but a person sitting in the chair is not necessarily doing
PC work but may actually be doing other work than PC. Therefore,
in this research, in order to distinguish between PC work and nonPC work, the sleep function of the PC is utilized in addition to thetwo kinds of sensor data acquired from the smart tap and mat
sensor node. More specifically, while the PC is ON, if the user
sitting on the chair does not perform any PC operation for a certain
time, the PC automatically goes to the sleep mode and its power isturned off. This indicates the beginning of the work other than PC.
According to our prototype system experiment result [1], if weshorten the sleep time of the PC, the difference from the correct
answer data recorded with the camera became small. The result
showed that the four conditions of PC work time, different work
time other than PC, useless electric usage time, and absence time
can be measured with high accuracy. However, setting a short sleep
time may be inconvenient for the user in PC work.

4

PROTOTYPE SYSTEM IMPLEMENTATION
Section 4 describes the prototype development of the system

Figure 4: Application operation flow of client PC.

proposed in this paper. In the prototype developed this time, inorder to monitor the situation of PC work by users, one smart tap

Figure 5 shows the application screen of the client PC actuallyand one mat sensor node were arranged for one PC as shown indeveloped. Information on the application in the forefront window

Figure 1. One host computer was installed to collect sensor

is acquired from Windows OS. As shown in Figure 5, the acquired

information from each sensor node. The communication between

information, including the date, time and application name, iseach sensor node and the host computer was performed throughdisplayed on the text box. This information is also transmitted to

Wi-Fi wireless connections via a Wi-Fi wireless communication

the host PC by wireless communication (Wi-Fi).

28

International Workshop on Informatics ( IWIN 2018 )5．EVALUATION EXPERIMENT OF
PROTOTYPE SYSTEM
5.1

Experimental method

A prototype of the system proposed in this paper was developedand experimental evaluation was carried out. The experiment site
was the E602 room of Kanagawa Institute of Technology C2, 6thfloor. The experiment period was about 1 hour from 16:31:21 to
17:41:12 on December 18, 2017. In this example, each node device

Figure 5: The application screen on the client PC
developed this time.
4.2

was arranged as shown in Figure 10 so as to be able to measure theoperating situation of PC for one person. In addition, one
application which can analyze software usage was installed on one

Application of host computer

server PC and one client PC. On the client PC, the sleep function of

The screen of the application on the host computer developed

the OS of the personal computer (sleep timeout set to 3 minutes)

this time is shown in Figure 6. The application on the hostwas used to measure the total PC use time and total PC sleep time,computer receives the information on the used software transmitted

and it was evaluated whether various software information can befrom the client-PC application, the information on the PC operationacquired accurately within PC use time.

state from the smart tap, and the presence information from the mat

In the experimental evaluation of this time, the software

sensor. For the management of the presence information and theoperation time was calculated from the video taken by the camera

operation status of the personal computer, the operation time andas the correct answer data. We evaluated the proposed system by

the like of the personal computer are calculated by the same

comparing the correct answer data with the result obtained by amethod as the conventional system, and the operation state iscommercially available software analysis application (Manic Timedisplayed and recorded on the application screen on the host/ Finkit doo) and the system we proposed this time. In order tocomputer side. The software information used in each client PC

evaluate whether the data acquired in our system were accurate,

was developed so that the information on the used software

two cameras were prepared as shown in Figure 7, one was used totransmitted from the client PC can be saved in a file in CSV

check the operation status of the software and installed in front offormat. The applications on the host computer and client PC were

the screen. The second was installed at the position shown indeveloped with Visual Basic 2015 language.

Figure 7 in order to confirm the presence of the user and PC
operation status. Regarding the PC operation status, it was judged
to be ON when the screen of the display was lit, and it was judged
to be OFF when the screen was dark. We evaluated this proposedsystem by comparing the videos taken with these two cameras (ascorrect answer data) with the data measured by this prototype
systemFigure 7:Experiment layout (Camera position etc.)
5.2

Results of the experiment

Table 2 shows the total time of PC work and the total time of

Figure 6: Data collection application screen on hostcomputer.

work other than PC, comparing the proposed system and camera(correct answer data). In the proposed system, both the PC work

29

International Workshop on Informatics ( IWIN 2018 )time and the work time other than PC had an error of about ± 3

order of frequency of use are shown in Figure 9. The graphicalminutes compared to the camera. The sum of the PC work time

representation as shown in Figure 9 enables easy distinction

and the work time other than PC was the same as with the camera.

between frequently used ones and less used ones.

Therefore, it is likely that the error of breakdown of PC work time

From the above results, we confirmed that the proposed system

and work time other than PC changes depending on the PC sleep

can analyze the usage time of various software applications usedtime.

during the PC work in detail. Moreover, it was confirmed that theoperation time of each software application can be measured with

Table 2：Measurement result of total PC work time
and total PC sleep timehigher accuracy than the commercially available application
analysis software Manic Time.

Table 3：Measurement result of operation time ofvarious applications by comparing camera and ManicTime

Figure 8: Analysis results of various software bycomparing camera, Manic Time and proposed system.

Table 4：Measurement result of various application
usage times by comparing camera and prototype
systemFigure 9: Analysis result of frequently used ones andlow ones of various software usage.

6.CONCLUSION
Table 3 shows the results of running time of various

In this paper, we described a system with which we can grasp

applications, comparing the commercial application analysis

various applications used during the PC working time in detail forsoftware (Manic Time) with the camera, and Table 4 shows thethe PC management system we developed in the past.

comparison between our prototype system and the camera.

In the experimental evaluation of the prototype of the proposed

Compared to the camera, the commercial software (Manic Time)system of this paper, the error was less than 7 seconds as compared

had errors of up to 29 seconds in operation time of various

with the correct data recorded with the camera. The error was 29

software. By comparison, in the proposed system, the error wasseconds at maximum in the case of Manic Time, which is one of
the commercially available application analysis software products.

smaller and within 7 seconds.
Next, a graph of various software analysis results used for PC

It was confirmed that this error occurred greatly when the PC goesworking time is shown in Figure 8. As shown in Figure 8, in theto sleep and returns from sleep. A likely reason for the large error

proposed system, there is no loss of software information and thewhich occurs in the case of the commercially available application

error is small. The results of graphing the analysis results ofanalysis software is because the application is given a low

various software of the proposed system and rearranging them inprocessing priority on the PC and is forced to wait for the log data30

International Workshop on Informatics ( IWIN 2018 )[10] H. Suda, S. Nakamura, M. Ogawa, H. Kumazawa, N.
Komdou, “Attendance Management System using Password
Distribution of NFC Tag with Electronic Paper(In Japanese)”,The 76TH National Convention of IPSJ, No.4, pp.853-854(2014).
[11] N. Nishimura, “New・Suddenly Attend(In Japanese)”,
Journal of Yasuda Women’s University, Vol.41, pp.371379 (2013).
[12] ManicTime https://www.manictime.com/[13] Microsoft “Software metering in System Center
Configuration Manager(In Japanese)”〈https://technet.microsoft.com/jajp/library/gg682005.aspx〉(2018).
[14] Hitachi,“JP1 Version 8 NETM / DM Installation and Design
Guide (for Windows (R)) Monitor the operation status ofsoftware(In Japanese)”,〈http://itdoc.hitachi.co.jp/manuals/3020/30203L36A0/DMD
S0011.HTM〉(2017).
[15] CQ publishing companies, “Web brain connection WiFi×3G/LTE of IoT(In Japanese)”, Transistor technology,
Vol.9, pp.35-93 (2016).

to be saved in the PC before being available for readout by theapplication.
Since our proposed system adopts the method of directly
acquiring data from the window information on the screen, unlike
the commercially available software, the processing time is not
required and it is possible to obtain accurate information on theusage situation of each software application.
Therefore, we found that the system we propose can accurately
grasp the usage time of each software application used during the
PC work time than the commercial application analysis software(Manic Time) which relies solely on the log data. In addition tothis, PC work hours, work hours other than PC, absence time etc
can be accurately grasped. It is expected that our system will bepractically applicable in the future as a system to monitor theutilization situation of PC facilities in office and educational
institution PC practice rooms in real time.

REFERENCES[1] Y.Takabayashi, Y.Kagami, H.Kawauchi, M.Isshiki, K.Abe,“Development of PC management system for the office and
the PC practice room using IoT technology(In Japanese)",International Workshop on Informatics(IWIN2017), pp.3-8(2017).
[2] N. Morita, “Development and Result of Attendance
Registration System Using IC cards(In Japanese)”, The
Institute of Electronics, Information and Communication
Engineers (IEICE)/Forum on Information Technology (FIT),
11(3), pp.65-68 (2012).
[3] H. Suda, S. Nakamura, M. Ogawa, H. Kumazawa, N.
Komdou, “Attendance Management System using Password
Distribution of NFC Tag with Electronic Paper(In Japanese)”,The 76TH National Convention of IPSJ, No.4, pp.853-854(2014).
[4] S. Kubota, S. Furukawa, Y. Soejima, R. Kawamura, and K.
Sugitani,”Presence-type Attendance Management System in
Educational PC Classrooms(In Japanese)”, The Institute ofElectronics, Information and Communication Engineers(IEICE)/ Forum on Information Technology (FIT), 8(4),pp.175-177 (2009).
[5] H. Matsumoto, S. Wada, S. Hara, N. Morita, “Development
and Evaluation of Attendance Registration System using
Suica(In Japanese)”, Proceedings of the School ofInformation and Telecommunication Engineering Tokai
University, Vol.1, No.2, pp.16-21 (2011).
[6] K. Tanaka, K. Shuwa, “Application of iBeacon in Laboratory
Room Management System Automation(In Japanese)”,Tokyo City University Yokohama Campus Journal of
Information Studies, Vol.4, No.16, pp.31–37 (2015).

[7] N.Takayama, M.Kitamura, “Hand Free Entry/
Attendance Management System(In Japanese)”, NEC
Technical Report, Vol.63, No.3, pp.60–63 (2010).
[8] S. Ono, H. Yasuda, “Proposal of Setting Management
System”, Proceedings of the 2014 IEICE General
Conference, p.184 (2014).
[9] K. Onozawa, K. Watanabe, K Suwa, “Room Management
System and Learning Situation Management System Using
Smartphone(In Japanese)”, Tokyo City University Yokohama
Campus Journal of Information Studies, Vol.4, No.13, pp.615 (2012).

31

International Workshop on Informatics ( IWIN 2018 )Proposal of IoT Edge Optimization Modelfor Balancing Local System and Total System and its Simulation Evaluation
Shinji Kitagami *, Yuichi Tokunaga** , and Norio Shiratori*****

Department of Management and Information Sciences, Fukui University of Technology, Japan**
Information Technology R&D Center, Mitsubishi Electric Corporation, Japan***
Research and Development Initiative, Chuo University, Japan

Abstract - In IoT-based system, there is trade-off
relationship between the optimization for total system in thecloud computing and the optimization for local system in theedge computing. That is, the result of the cloud optimization
may become a constraint condition for the edge optimizationand vice versa. In this paper, we propose an IoT edge
optimization model for balancing the total system and local
system. In this optimization model, the balance of both cloud
optimization and edge optimization is treated as Pareto
optimization problem. Further, the cloud optimization
process and the edge optimization process negotiate eachother by an intelligent protocol to find the optimal balancing
condition.

2

The IoT architecture has changed from the vertical
integration type to the horizontal integration type and the
distributed type [1] [2]. In this paper, we refer thehorizontally integrated IoT architecture to the cloud-centric
IoT computing model (CC-IoT model). Also, the distributed
IoT architecture is referred to as the edge-centered IoT
computing model (EC-IoT model) in this paper. The CC-IoT
model optimizes the system by utilizing various datacollected from many sensors. In this paper, optimization by
CC-IoT model is referred to as the cloud optimization. For
example, the CC-IoT model is suitable for optimizing theenergy supply-demand balance in the energy management
system (EMS) and optimizing the relaxation of traffic
congestion by ITS [4] [5].
However, recently, some problems such as an increase in
network load and a delay in feedback control have been
pointed out in the CC-IoT model [2] [3]. The EC-IoT model
proposed to solve these problems optimizes the local systemin the edge area such as buildings and vehicles close to thedata source. In this paper, the optimization by EC-IoT modelis referred as to the edge optimization. For example, the ECIoT model is suitable for optimization for the energy saving
and comfort in a building by EMS and automatic drive
control by V/C for comfort and safety of drivers [5].
In general, there is trade-off relationship between the CCIoT model and the EC-IoT model. In other words, the resultof the cloud optimization may be a constraint condition foredge optimization. Also, the result of edge optimization may
be a constraint condition of cloud optimization.

Keywords: IoT System, Cloud Computing, Edge Computing,
Optimization Problem

1

OPTIMIZATION IN CLOUD AND EDGEINTRODUCTION

IoT systems that connect many sensors and devices directly
to the Internet and provide various services without humanintervention are expanding in the industrial sector, the home
sector, and the social sector [1]. The conventional IoT system
is cloud-centric IoT computing model (CC-IoT model) basedon the cloud computing [2]. The CC-IoT model realizes
optimization of the system using various data collected fromsensors installed widely. On the other hand, edge-centric IoT
computing model (EC-IoT model) has been proposed to solve
the problem of the CC-IoT model such as increase of network
load and delay of feedback response [2][3]. The EC-IoT
model performs data processing at the edge server and the IoT
gateway close to the data source and the device to becontrolled for local optimization within the edge area.
However, in many IoT systems, there is trade-off relationshipbetween cloud optimization and edge optimization. That is,
the result of the cloud optimization may become a constraint
condition for edge optimization and vice versa.
In this paper, we propose an IoT edge optimization model
for balancing the cloud optimization and edge optimization.
In this optimization model, the balance of both optimizations
is treated as Pareto optimization problem. Also, the cloud
optimization process and the edge optimization process
negotiate each other by an intelligent protocol to find the
optimal balancing condition.

3

IOT EDGE OPTIMAZATION MODEL
3.1 Basic Concept

In this paper, we propose an IoT edge optimization model
to realize effective next-generation mobility services. Figure
1 shows the basic concept of the IoT edge optimization
model. In the figure, the vertical axis represents the cost by
ITS cloud optimization, and the horizontal axis represents
the cost by V/C edge optimization. Both of which are higher
optimization level as they are closer to zero. Also, each costhas an acceptable range. For example, there is an allowable
cost range of the degree of congestion in the cloud33

International Workshop on Informatics ( IWIN 2018 )optimization by ITS, and there is an allowable cost range ofthe driver's comfort in the edge optimization by V/C. In
addition, the curve in figure 1 is a Pareto optimal curve. In
the case where priority is given to the cloud optimization by
ITS, the cost of the edge optimization does not fall within
the allowable range. Conversely, when priority is given to
the edge optimization by V/C, the cost of the cloud
optimization cannot be within the allowable range.
In the IoT edge optimization model proposed in this paper,
the cloud optimization process and the edge optimization
process negotiate by an intelligent protocol and balance thecloud optimization and the edge optimization.

The IoT edge optimization model minimizes the objectivefunctions of cloud optimization, edge optimization, andsystem optimization by exchanging shared variable vectors
with the intelligent protocol.

3.3

In the formulation mentioned above, the cloud variable
vector 𝒗𝒄 used only in the optimization process on the cloudside cannot be referred from the edge side. Likewise, the edge
variable vector 𝒗𝒆 used only on the edge side optimization
process cannot be referred from the cloud side. For thatreason, in the IoT edge optimization model, shared variable
vector 𝒗𝒔 are exchanged between cloud and edge by theintelligent protocol. Figure 2 shows the intelligent protocol
for exchanging shared variable vectors 𝒗𝒔 and negotiating
between cloud and edge optimization processes. First, asshown in the figure, 𝒗𝒔 which minimizes the cloud

Formulations

3.2

The formulations of the IoT edge optimization model isshown as below.
Intelligent Protocol

Cloud Optimization

𝑀𝐼𝑁𝒗𝒔,𝒗𝒄 𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒄
u𝑛𝑑𝑒𝑟 𝑐𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡𝑠 𝒗𝒔 , 𝒗𝒄

Edge Optimization

𝑀𝐼𝑁𝒗𝒔,𝒗𝒆 𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒆u𝑛𝑑𝑒𝑟 𝑐𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡𝑠 𝒗𝒔 , 𝒗𝒆

Total Optimization

𝑀𝐼𝑁𝒗𝒔,𝒗𝒄,𝒗𝒆 𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒄 , 𝒗𝒆u𝑛𝑑𝑒𝑟 𝑐𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡𝑠 𝒗𝒔 , 𝒗𝒄 , 𝒗𝒆𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒄 , 𝒗𝒆𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒄

𝑘 ∗ 𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒆

Here,𝒗𝒔 : shared variable vector
𝒗𝒄 : cloud variable vector
𝒗𝒆 : edge variable vector
𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒄 : objective function of cloud optimization
𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒆 : objective function of edge optimization
𝑐𝑜𝑠𝑡 𝒗𝒔 , 𝒗𝒄 , 𝒗𝒆 : objective function of system optimization

Figure 1. Balancing Cloud Optimizationand Edge Optimization

Figure 2. Intelligent Protocol of IoT Edge Optimization Model

34

International Workshop on Informatics ( IWIN 2018 )optimization cost is transmitted from the cloud to the edge.
On the edge side, adjust the value of 𝒗𝒔 to minimize the edge
optimization cost and send it back to the cloud. By repeating
this negotiation, each optimization processes are balanced sothat the optimal cost of the cloud, the edge, and the entire
system are minimized.

4

simultaneously, its node cost becomes 6. After calculating thecost of all the nodes, The ITS notifies the largest cost of thenode to the V/C of all vehicles. If there are multiple of nodes
with the largest cost, the ITS selects a node randomly andnotifies it to the V/C.
Figure 3 shows the relationship between cloud optimal cost
and the edge optimal cost for the optimization phases 1 to 8
as a summary of the simulation results. As shown in the figure,if the cloud optimization cost is 3 or less, the edge
optimization cost will be 20 or more. In that case, although
the number of congested nodes is limited to about 2, thetraveling time of the car increases more than twice.
Conversely, optimizing the edge optimization cost to be 10 or
less, the cloud optimization cost becomes 6 to 10, and it turns
out that the number of congested nodes increases. Also, it was
found that when the cloud optimization and the edge
optimization are balanced in the state of the optimization
phase 5. In this balanced point, the cloud optimization cost is
4 and the edge optimization cost is 16.

EVALUATIONAs described in Chapter 2, to realize effective nextgeneration mobility service, it is necessary to balance theoptimization of cloud by ITS and the edge optimization by
V/C. Here, representative cloud optimization is to eliminate
the traffic congestion in the whole town, and representativeedge optimization is to maintain the comfort of the driver ofthe vehicle by reaching the destination quickly. When the
traffic volume on the road is low, it is easy to minimize theobjective function of both optimizations. However, when the
traffic volume increases on the road, it is necessary to balance
the ease traffic congestion and drivers' comfort. Applying theIoT optimization model in which ITS and V/C negotiate by
the intelligent protocol, it is expected to solve this problem.
To evaluate the effectiveness of the IoT edge optimization
model proposed in Chapter 3, we conducted a simulation inwhich the traffic congestion mitigation control by ITS and theautomatic operation by V/C cooperate. In this simulation, weused the iGraph package of R [6].
In this simulation, it is assumed that four vehicles
simultaneously depart from different starting points on theroad in a grid pattern including 25 intersections (nodes). The
objective function of edge optimization by V/C is the sum of
the moving times of all vehicles, and the objective functionof cloud optimization by ITS is the sum cost of each node.
The initial value of the weight of the path corresponding to
the moving time between nodes is 2 for the peripheral pathand 1 for the other internal paths. In addition, the cost of eachnode is set in response to the degree of traffic congestion, andthe weight of the surrounding paths are added according to
the cost of the node.
V/C determines the shortest path that the vehicle moves
from the start node to the destination node by the Dijkstra
method. If the cost value is set for the nodes on the path, thepath determination algorithm adds the value to the weight ofthe path around the node. The path weight means a movement
time among nodes. After determining the shortest path, V/Ctransmits the scheduled passing time, a cumulative value ofthe path weight, of each node to the ITS. If the node cost value
is transmitted from the ITS, V/C determines the shortest pathagain by adding the cost to the surrounding pass. Although it
is necessary to calculate the shortest path continuously whilethe vehicle is moving, we assume that V/C determines theshortest path before moving in this simulation.
The ITS presumes congestion degree of the node based on
the expected passing time of the node receiving from the V /C of vehicles in the system. That is if two vehicles passthrough at the same node simultaneously, the ITS adds 1 point
to the cost of the node. If two or more vehicles pass at thesame node simultaneously, the ITS adds the number of
combinations of the simultaneous passage to the cost of thenode. For example, if 4 vehicles pass at the node

5

DISCUSSIONAs shown in the simulation result of Figure 3, there is atrade-off relationship between the cloud optimization by the
ITS and the edge optimization by the V/C. In this simulation,negotiating between the cloud optimization process and theedge optimization process, we confirmed that it is possible tobalance both optimizations in the optimization phase 5. In fact,
whether the balanced state of two optimizations is valid
depends on the allowable range for each optimization. In the
proposed model, we can select various combinations of thecloud optimization and the edge optimization depends onstakeholders' requests. Since the simulation shown in thispaper was aimed at verifying the principle of optimization
balancing by IoT edge optimization model, the simulation
model was simplified. Furthermore, to perform a practicalsimulation, it is necessary to consider the difference in themoving start time and the moving direction of vehicles in thesystem. That is, it is assumed that congestion occurs in a node
when there are vehicles passing through the node at the same

Figure 3 Balanced Optimization of Cloud and Edge
35

International Workshop on Informatics ( IWIN 2018 )time in this simulation. However, in fact, the case such as topass each other in the opposite direction and not to becongested at the node shall be considered.
The IoT edge optimization model can balance the cloud
optimization and the edge optimization according to different
requirements of the stakeholders of the system. In the energy
management system, power companies are stakeholders on
the cloud side and the supply balance of electric power is oneof the objective functions of cloud optimization. Meanwhile,
power consumers, such as buildings, factories, homes, etc.,
are stakeholders of the edge side and the comfort of theresidents is one of the objective functions of edge
optimization [5].
In next-generation mobility services for vehicles,
municipalities and road managers are stakeholders on thecloud side and drivers of vehicles are stakeholders on the edge
side. As the objective function of the cloud optimization, it isconceivable that the traffic congestion mitigation, traffic
accident reduction, CO2 emission reduction, and so on.
However, in this paper, relaxation of traffic congestion isregarded as a main objective function of the cloud
optimization. Also, the shortest arrival at the destination was
defined as a main objective function of the edge optimization
as one of the comforts of the driver.
As shown in the simulation result of Figure 3, there is atrade-off relationship between the cloud optimization by the
ITS and the edge optimization by the V/C. In this simulation,negotiating between the cloud optimization process and theedge optimization process, we confirmed that it is possible tobalance both optimizations in the optimization phase 5. In fact,
whether the balanced state of two optimizations is valid
depends on the allowable range for each optimization. In the
proposed model, we can select various combinations of thecloud optimization and the edge optimization depends onstakeholders' requests. Since the simulation shown in thispaper was aimed at verifying the principle of optimization
balancing by IoT edge optimization model, the simulation
model was simplified. Furthermore, to perform a practicalsimulation, it is necessary to consider the difference in themoving start time and the moving direction of vehicles in thesystem. That is, it is assumed that congestion occurs in a node
when there are vehicles passing through the node at the sametime in this simulation. However, in fact, the case such as topass each other in the opposite direction and not to becongested at the node shall be considered.

6

ACKNOWLEDGEMENTPart of this work was carried out under the Cooperative
Research Project Program of the Research Institute ofElectrical Communication, Tohoku University.

REFERENCES[1] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M.
Aledhari, and M. Ayyash, “Internet of Things: A Survey
on Enabling Technologies, Protocols, and Applications,”
IEEE Communication Surveys & Tutorials, vol.17, no.4,
pp. 2347-2376, Jun. 2015.
[2] N. Shiratori, S. Kitagami, T. Suganuma, K. Sugawara,
and S. Shimamoto, “Latest Developments of IoTArchitecture,” The Institute of Electronics, Information
and Communication Engineers, vol.100, no.3, pp.214221, Mar. 2017 (in Japanese).
[3] P.G. Lopez, A. Montresor, D. Epema, A. Datta, T.
Higashino, A. Iamnitchi, M. Barcellos, P. Felber, and E.
Riviere, “Edge-centric Computing: Vision andChallenges,”
ACM
SIGCOMM
ComputerCommunication Review, vol. 45 issue 5, pp. 37-42, Oct.
2015.
[4] T. Suganuma, T. Oide, S. Kitagami, K. Sugawara, andN. Shiratori, “Multiagent-Based Flexible Edge
Computing Architecture for IoT,” IEEE Network,
Vol.21, Issue.1, pp. 16-23, Jan. 2018.
[5] T. Ogino, S. Kiatagami, and N. Shiratori, “Flexible IoT
Edge Computing Model and Its Applications,” IPSJ SIG
ZTechnical Report, vol.2018-cds-21 No.1, Jan.2018(in Japanese).
[6] Get started with R igraph, http://igraph.org/r/(accessed 14 May 2018).

CONCLUSION

In this paper, we proposed an IoT edge optimization modelthat balances the cloud optimization and the edge
optimization. Also, as an example of applying the proposedmodel, we showed simulation results for alleviating traffic
congestion in automatic driving. As future work, we will
extend the IoT optimization model for the time series
optimization.

36

International Workshop on Informatics ( IWIN 2018 )Time synchronization method for rice cultivation management system with IoT
specialized outdoor communication procedure
Koichi Tanaka†, Mikiko Sode‡, Masakatsu Nishigaki†, Tadanori Mizuno*†Graduate School of Science and Technology, Shizuoka University, Japan‡Global Information and Management, International College of Technology, Japan*Faculty of Information Science, Aichi Institute of Technology, Japan
Abstract - In the field servers for rice fields that use LPWA
technology, which require only batteries for their operation,
time synchronization is an important factor in reducing thepower consumption. Additionally, time synchronization is an
important technique for increasing line use efficiency. In thispaper, we describe a method of constructing a wirelessnetwork of an economical time-synchronized field serverusing LoRa for achieving low cost and describe the effect of
reducing the power consumption. The performance of thetime synchronization using GPS is accurate but considering
the increase in manufacturing cost because of increase in thecost of the GPS module and the number of parts, weconsidered that the introduction of GPS is difficult in a field
server for usage by a general farmer. Therefore, we propose
a time synchronization method without GPS. Fromexperimental results, we confirmed that the time wassynchronized and transmission and reception of data between
the master and the field server ensued normally. The power
consumption of the field server was 108.4mWh per day, and
it is theoretically possible to operate it for 691 consecutive
days. Continuous operation of the server for 691 days is
sufficient for monitoring rice cultivation work.

The time synchronization technique has been extensively
studied previously [4, 5]. The method for the timesynchronization using GPS has also been proposed [4]. To
use this method, it is necessary to install a GPS receiving
module in every field server, which leads to an increase ininitial cost at the time of introduction. There is also a problem
of increasing power consumption, therefore, it is difficult to
use it for the rice field servers, for which lowering the
introduction barrier is desired. Although TPSN [5] is
proposed as the time synchronization method, it requires along time for the time synchronization and is therefore
difficult to use for the field servers where the power
consumption demands are considerably high.
In this study, we propose the intermittent operation
communication protocol and the time synchronization
method to solve the aforementioned problems. In the
proposed method, after the field server system transmits thesensor data to the master unit system, the master unit systemon receiving the sensor data transmits the time correction
signal to the field server system, thereby performing the timesynchronization. In this paper, section 2 describes the system
configuration, section 3 discusses the communication
protocol and time synchronization, section 4 shows theoperational test and result, and finally, section 5 summarizes
the study.

Keywords: Agriculture, Field Server, Sensor Network, Low
Battery Consumption, Time Synchronization

1

INTRODUCTION

Owing to the advancing age of older farmers, it is necessary
to pass down the knowledge and techniques of farming
known to them, to the next generation [1]. Therefore, we aredeveloping a field management system that will help thisknowledge transfer [2]. It is reported that LoRa has low
power consumption and long communication distance;
therefore, it is suitable for communication in the field server
for the rice fields [3]. Therefore, we adopted LoRa which
does not require communication cost.
The field server needs to operate on the battery for six
months, starting from the rice planting to reaping, becausethat rice fields do not have a power supply because of cost. In
addition, it is difficult to install solar panels because solar
panels are big and get in the way of farming.
Therefore, low power consumption is important for the field
servers for rice fields. To operate for six months with nopower supply, the field server needs to turn itself off except
when sending or receiving the sensor data or othercommunication. This requires the intermittent operation
communication protocol and the time synchronization
method.

Figure 1: Field management system.

2

SYSTEM CONFIGURATION OF FIELDMANAGEMENT SYSTEM

The field management system is composed of the field server
system, master unit system, and cloud service. Figure 1 shows
the overall structure of the field management system. The field
server system is installed in the rice fields and receives thesensor data. Further, the data is sent to the master unit systemthrough the LoRa wireless network. The master unit system
integrates the sensor data from the field server system and37

International Workshop on Informatics ( IWIN 2018 )time correction signal cannot be received. The power ON/
OFF circuit is composed of the PIC microcomputer and the
FET; it controls power supply to the AVR microcomputer.
The PIC microcomputer controls the FET by outputting
HIGH/LOW at the GPIO pin. The time required for the power
supply control is calculated and controlled by using the timer
interrupt in the internal clock of the PIC microcomputer.

sends them to the cloud service through the 3G line or Wi-Fi.
The cloud services are provided by the smartphone
applications, tablet applications, and web pages. These
services provide data to farmers to alert them about waterlevels, propose a suitable work plan, preserve work records etc.
Communication between the field server system and themaster unit system using LoRa is capable of long-distancecommunication. LoRa has been found to have a practicalcommunication distance of 3,000 to 4,000 m as shown by thebasic communication characteristics survey [7]. The rice fieldof Ishikawa prefecture was assumed, and the linear distance
between the parent machine and the field server was within
3,000 m. For this reason, we adopted LoRa, which enables
direct communication between the field server and parent
machine.
Figure 2 shows the positional relationship between each field
and the office of the assumed the agricultural corporation in
the Ishikawa prefecture. A, B, C, D, E, F, and G represent theposition in each field, where the field servers are installed. P
represents the location of the office; the master unit is installedin the office. The linear distances between the field servers A
to G and the parent machine P are as follows: 397 m between
A and P, 923 m between B and P, 943 m between C and P,
684 m between D and P, 1,150 m between E and P, 1,440 m
between F and P, and 1,910 m between G and P.

Figure 3: Field server system configuration.

Figure 4: Master unit system configuration.
The configuration of the master unit system is shown in
Figure 4. The master unit system is composed of a Raspberry
Pi, LoRa module for transmission, LoRa module for reception,
and a 3G dongle. When the field server system is turned on for
the first time, the time is not held and the time is set after it istransmitted by the master unit system. Therefore, the master
unit system always maintains the reception state. It is desirable
that the master unit system can respond to communicationfrom the field server system when the field server is installed.
Further, the reception and transmission modes exist in the
LoRa module, and it takes time to switch the modes. Therefore,
by installing two different LoRa modules for receive andtransmit, it is possible to reduce the waiting time oftransmission and reception and maintain the reception state at
all times.

Figure 2: Position of each rice field and the office.

We will further explain the configuration of the field server
system installed in the rice field and the master unit systeminstalled in the office. Figure 3 shows the configuration of the
field server system. The field server comprises the battery,
power ON/OFF circuit, AVR microcomputer, LoRa module,
various sensors, and SD card module. The field server ispowered by the battery. To realize low power consumption,
the power ON/OFF circuit operates only for several tens ofseconds in one hour. The wireless modules and the sensors arecontrolled by the AVR microcomputer. Five types of sensors
are mounted to measure the temperature, humidity, water level,soil temperature, and soil moisture content. The sensor data isstored in the SD card together with the time stamp. This is afunction for reliably saving the data, considering the casewhere it cannot be transmitted to the master unit or where the3
3.1

COMMUNICATION PROTOCOL
Communication Protocol

The frame formats used for the communication are shown in
Tables 1, 2, and 3. Table 1 shows the common frame format,
consisting of the destination, the source, and the payload.
Table 2 shows the format of sensor data transmission. Since
38

International Workshop on Informatics ( IWIN 2018 )there are five types of sensors in use, the sensor data aredefined in the format of 1 to 5. The temperature, humidity,
water level, soil temperature, and soil moisture content areentered in that order from the sensor data 1 to 5. The acquired
five types of data can be stored in 2-byte units. Additionally,
when the number of types of sensors increases, 2 bytes areadded to the format of sensor data transmission. Table 3.3
shows the format of the time correction signal. The timestamp and the time correction signal transmitted from themaster to the field server are stored in the payload.
Table 1: Common frame format.
Destination
Source1Byte
1Byte

operating time = sensor data acquisition time +sensor data transmission time +time correction signal reception time +retransmission time(1)

PayloadVariable

Table 2: Format of sending sensor data (Payload).
SensorSensorSensorSensorSensorData 1
Data 2
Data 3
Data 4
Data 5
2 Byte2 Byte2 Byte2 Byte2 ByteFigure 5: Communication protocol sequence.
Table 3: Format of correction time signal (Payload).
Timestamp
Correction time
(UNIX Time)
4 Byte2 ByteThis operating time is written in advance to the AVR
microcomputer and sent to the PIC microcomputer each time
the AVR microcomputer is powered on. The PIC
microcomputer uses this value to calculate the restart time.
Here, the sensor data acquisition time is the time to measure
the sensor data. The sensor data transmission time is the timeto transmit the sensor data to the master unit. The timecorrection signal reception time is the time to receive thecurrent time from the master unit. The retransmission time isthe time to perform retransmission processing when sensor
data cannot be transmitted to the master unit. In the proposed
method, time synchronization is performed once every hour.
It has been confirmed from the measurement results that there
is an error of not more than ±10 seconds at the maximum inan hour [3]. Therefore, the error of acquisition time of sensor
data is also ±10 seconds or less. This error is a problem-freerange as the sensor acquisition time error for agriculture.

The field server system starts once every hour. It gets thesensor data and transmits it to the master unit system. Whentransmission is successful, the master unit system sends thecorrection time to the field server system. The field server
system further corrects its own internal clock for the timesynchronization. Later, when the field server system receives
the corrected time or go on operating time per hour ofdescribed later elapses, the power is turned off except for the
power control circuit.
Figure 5 shows an example of three field servers system inwhich resending mode does not occur in any of thecommunications. First, the field server system A (hereinafter,
FS-A) is activated. FS-A measures the sensor data andgenerates the sending packet according to the sensor data.
Further, the packet is sent to the master unit system. The field
server system has only one LoRa module, therefore it
switches from the sending to the reception mode. This
switching requires several seconds. After switching to thereception mode, the field server system waits until the set
timeout period. Further, it receives the corrected time signal
from the master unit system. When the master unit system
receives sensor data from FS-A, it sends the corrected time
signal to FS-A. FS-A corrects its internal time based on thereceived correction time signal. After the correction, FS-A
goes into sleep mode even during the resending possible time.
When the field server system fails to receive the corrected
time signal or the master unit system fails to send thecorrected time signal to the field server system it is necessary
to resend it along while the possible resending time. Field
server system B (FS-B) and field server system C (FS-C)
perform in the same sequence as A.

3.2

Operation of Resending Mode

When the field server cannot receive the time correction
signal and the reception waiting time has elapsed, the field
server performs the timeout operation. After the timeout, thefield server waits for random seconds from 0.1 to 5.0 s andthen retransmits. The following is the cause of the timeout.
1) When the master unit cannot receive the communicationfrom the field server due to the radio wave attenuation andthe noise cannot be demodulated
2) When a collision occurs in the transmission data due tooverlapping of the transmission times of a plurality of field
servers
The first one is that the cause of the noise is often temporary,so there is a high possibility that the problem will be solved
if the transmission process is performed with shifted time.
The second one can be prevented by accurate timesynchronization.
When the noise or the collision occurs, the master unit does
not send the time correction signal to the field server,

The operating time per hour can be obtained from equation
(1).

39

International Workshop on Informatics ( IWIN 2018 )therefore, the field server's reception standby timeout occurs.
The field server that has timed out executes retransmission,
but to prevent re-collision with the communication performedby the field server of the initial power-on, a random second
standby time is provided. After executing the retransmission,
the field server switches from the transmission to thereception mode and waits for reception. This operation iscontinued until the field server can receive the time correction
signal from the master unit. However, to avoid collision of
the communication with that of other field servers, the poweris turned off forcibly when the other field server's operation
is about to start.
The sequence operation in this case is shown in figure 6. In
figure 6, the field server C (FS-C) has timed out and isretransmitting. If it is within the possible retransmission time,
the processing of the transmission and reception standby isrepeated until transmission/reception is completed.

starting simultaneously with other field servers, the current
time is shifted appropriately.
A formula for calculating the start time is shown in equation(2). Regarding equation (2), each field server has a uniquely
assigned field server identifier (below), and the correction
time to transmit to the field server is obtained by subtractingfrom the current time. Depending on the power-on time of the
field server, the time becomes a negative value; however, in
this case, a value of 3,600 is added.
𝑡𝑡2 = 𝑡𝑡1 − 30・FSID(2)

The FSID can be used in the range of 0x00 to 0x77, andthe field servers are started in the ascending order of FSID.
By using FSID, simultaneous activation of each field serveris prevented, and transmission signals of the field server are
prevented from colliding. It became possible to transmit

once every 30 seconds and about 100 field servers are able
to connect one master unit.

Figure7: Time synchronization mechanism between master
unit and field server.
The PIC microcomputer controlling the power ON/OFF
circuit always counts 0 to 3,599 s with the internal clock.
When the time within the PIC microcomputer reaches 3,600(0) s, power is supplied to the AVR microcomputer. Upon
receiving the time correction signal from the master unit, thefield server corrects the time within the PIC microcomputer
to the correction time transmitted from the master unit andcontinues counting. This leads to the time synchronization
between the field server and master unit. Even during thesecond and subsequent runs, when the time within the PIC
microcomputer reaches 3,600 (0) s, power is supplied to the
AVR microcomputer.
If the master unit fails to normally receive data from the field
server due to a communication error etc., the master unit
maintains the reception standby state of the sensor data
without transmitting the time correction signal.

Figure6: Sequence diagram at sensor data collision.

3.3

Time Synchronize Signal

The field server can transmit sensor data at an arbitrary
timing because the master unit is always on standby forreception. The master unit performs the time synchronization
with the NTP server beforehand and acquires accurate time.
The master unit generates the time correction signal
according to the time, receives the sensor data from the field
server, and then transmits the time correction signal.
The format of time correction signal is shown in Table 3.
The time stamp is entered with 4-byte UNIX time. It is used
to write the sensor data to the EEPROM or the SD card of the
AVR microcomputer. Since the correction time is used forcorrecting the time within the PIC microcomputer, the time
shifted for each field server from the current time is set as 0
to 3599 s in 2 bytes. CRC etc. is used for detecting andcorrecting communication errors that are not defined in theformat because they are added by the LoRa communication
module.
Figure 7 shows the mechanism of the time synchronization
between the field server and master unit. The master unit
sends the correct time obtained by the NTP server to field
server in the corrected time format. Upon receiving thecorrection time, the field server transfers the correction time
via the AVR microcomputer to the PIC microcomputer in the
power ON/OFF circuit. To prevent the correction time from

4
4.1

EXPERIMENTAL RESULTS
Verification of Communication Protocol

We conducted the 7-day operation test to confirm
whether the proposed communication protocol works as
expected between the master unit and field server.
Following are the points for the verification:

40

International Workshop on Informatics ( IWIN 2018 )1) Confirm whether the master unit can return the timecorrection signal to the field server within the reception
waiting time of the field server.
2) The field server that received the time correction
signal confirms whether to shift to the sleep state
immediately.
3) Confirm whether each field server properly changes
the timing according to the time correction signal andstarts at the specified time.

We examined the difference between the assumed
startup time of the field server and the actual startup time.
The results are shown in figure 8. The result displays therepresentative pattern of eight days from the operational
test of two months. From this result, it is understood thatwhen the time correction is performed, the error issuppressed to about in tens of seconds. Moreover, it isunderstood that the error is suppressed to 0 s in mostcommunication between the master unit to field servers.

In the verification of the communication protocol, we
used seven field servers, which is the same number used in
our field. The distance between the field server and themaster unit is centered on the master unit and all field
servers are installed within a radius of 1 m. The sensor data
sent from the field server to the master unit was saved in
the verification cloud.
Figure 8: Time error of field server.

Table 4: Verification results of communication protocol.
Classification
Number ofcommunications
Send sensor data
1,185
Number of resending
9
Completion of time
9
synchronization (resend 1 time)

The time error of the PIC microcomputer is the
maximum at 6.512 s in an hour from actual measurement[6]. This error is accumulated without timesynchronization; however, in the field management
communication protocol proposed here, this error is within
the range in which collision with other field servers does
not occur. Therefore, it is confirmed that timesynchronization is effective in this communication
protocol, and it is possible to reduce the increase in timeerror, which is proportional to the usage time. As a result,it became possible to transmit once every 30 seconds, andbecame possible to connect about 100 field servers to one
master unit.

Table 4 shows the verification results of thecommunication protocol. The number of operational days
is seven. The field server gets sensor data at an hourlyinterval, which is further sent to the master unit. The field
server successfully sent the data 1,185 times. Of the 1,185
times, only nine were retransmission. Even when the firstcommunication failed, reception of sensor data was
successful from all field servers through retransmission.
We confirmed that the protocol works for seven days
without problems.
We implemented the designed communication
protocols and carried out the operational test for a periodof two months in an actual field. Figure 2 shows themeasurement result at point C. A master unit was installedat point A. In this experiment, we confirmed thatenvironmental data can be acquired every hour. The field
server was equipped with sensors that can measuretemperature, humidity, water level, soil temperature, andsoil moisture content. The height at which the field server
was installed was approximately 1 m from the groundsurface to accurately measure the temperature. The heightat which the master unit was installed was set toapproximately 0.5m. It was confirmed that themeasurement can be performed without problems, anddata can be transmitted to the master unit. Owing to thefact that there is a communication failure at the rate of
approximately 15.8%, the time correction may not be
performed. The time synchronization was carried out
when the fault was solved and it was confirmed that the
protocol was operating properly.

4.2

Evaluation of Power Consumption

Apart from the verification of the communication protocol,
we conducted an experiment to verify the power consumption.
The purpose of the verification is to obtain the power
consumption during the operation. First, we measured thevoltage, current value, and processing time for each operation
mode. Table 5 shows the measured results [7]. The datatransmission mode is the most power consuming. It can be
confirmed that the sleep time mode has the lowest power
consumption among all.
Next, we calculated the power consumption and number ofworking days. Equation (3) shows the power consumption W[mWh] . Here, 𝑉𝑉1 is the rated voltage [V] of the field server
system. 𝐼𝐼𝑎𝑎 is the electric current [mA] during the sensor
stabilization standby and the sensor acquisition. 𝑡𝑡1 is thetime[s] during the sensor stabilization standby and the sensor
acquisition. 𝐼𝐼𝑏𝑏 is the electric current [mA] during thetransmission of sensor data. 𝑡𝑡2 is the electric time [s] during
the transmission of sensor data. 𝐼𝐼𝑐𝑐 is the electric current [mA]during the mode switching. 𝑡𝑡3 is the time [s] during the modeswitching. 𝐼𝐼𝑑𝑑 is the electric current [mA] during the standby
reception. 𝑡𝑡4 is the electric current [mA] during the standby
reception. 𝐼𝐼𝑒𝑒 is the electric current [mA] during the data
41

International Workshop on Informatics ( IWIN 2018 )reception. 𝑡𝑡5 is the time[s] the during the data reception. 𝐼𝐼𝑔𝑔 isthe electric current [mA] during the system sleep state.

5

CONCLUSION

We proposed a new communication protocol, constructed a
local wireless network, and conducted the experiment. In thefield servers for the rice field using the LPWA technology,
which require only batteries for operation, the proposed timesynchronization is an important technology for the purpose of
reducing the power consumption. Additionally, the proposed
time synchronization is an important technique for increasing
the line use efficiency. It was seen from the experimental
results that the power consumption of the field server is108.4mWh per day. Therefore, it was confirmed that themethod can continuously work for 691 days based on our
calculations. The time synchronization is effective and wasable to decrease the timing error in direct proportion to theoperating time. This protocol is valid for the rice cultivation
management systems because the field server is stable andcan operate for a long time. Therefore, it meets farmers'
expectation to utilize a reasonable field server.

𝑊𝑊 = (𝑉𝑉1 {(𝐼𝐼𝑎𝑎 ∙ 𝑡𝑡1 ) + (𝐼𝐼𝑏𝑏 ∙ 𝑡𝑡2 ) + (𝐼𝐼𝑐𝑐 ∙ 𝑡𝑡3 ) + (𝐼𝐼𝑑𝑑 ∙ 𝑡𝑡4 ) + (𝐼𝐼𝑒𝑒 ∙𝑡𝑡5 )} + 𝑉𝑉1 ∙ 𝐼𝐼𝑔𝑔 {3600 − (𝑡𝑡1 + 𝑡𝑡2 + 𝑡𝑡3 + 𝑡𝑡4 + 𝑡𝑡5 )} )/ 3600(3)

Table 5: Measurement results of power consumption.

We derived the number of operating days theoretically. The
power consumption is 4.52 mWh per hour; the consumption
being 108.4mWh per day. Therefore, theoretically, the field
server can operate for approximately 691 days, assuming theelectric quantity of the portable battery charger to be 75000
mWh. Although the number of operating days has thetheoretical value, it seems that the field server is able tooperate for six months, which is the requirement of the
agricultural corporation.

REFERENCES[1] Ministry of Agriculture, Forestry and Fisheries of Japan,
Statistics of agricultural labor, Accessed on 2017-6-2.
[Online]. Available:
http://www.maff.go.jp/j/tokei/sihyo/data/08.html.
[2] Kiyokazu Kurosawa, Isamu Iizima, Yoshiki Amemiya,
Shunya Yamamichi, Masaharu Toyota and Mikiko Sode
Tanaka, “Development of Operational Control System
for Rice Cultivation Equipped with Activity History
Function”, IEICE Technical Report, vol. 116, no. 346,
CS2016-55, pp. 59-64 (2016).
[3] Yuta Kawakami, Masaharu Toyota, Keitaro Terada,
Keiko Matsumoto and Mikiko Sode Tanaka, “A studyof the optimal agricultural field communication using
Sub-GHz wireless technology”, IEICE Technical
Report, vol. 116, no. 382, NS2016-138, pp. 107-112.
[4] Hao Guo and Peter Crossley “Design of a Time
Synchronization System Based on GPS and IEEE 1588
for Transmission Substations”, IEEE Transactions on
power delivery, vol. 32, no. 4, (2017).
[5] S. Ganeriwal, R. Kumar and M. B. Srivastava, “Timingsync Protocol for Sensor Networks”, Proceedings of the
1st ACM Conference on Embedded Network Sensor
Systems (SenSys'03), Los Angeles, California (2003).
[6] Keitaro Terada, Masaharu Toyota, Tadaaki Hirata, Yuya
Takada, Keiko Matsumoto and Mikiko Sode Tanaka,“Proposal of communication protocol for fieldmanagement using LoRa”, DICOMO2017, pp. 16711678, 2017/6/28-30.
[7] Yumeto Kojima and Mikiko Sode Tanaka, “Current
value measuring device for field server of fieldmanagement using LoRa”, IEICE Society Conference
2018, 2018/9/11-14.
[8] IEEE Standard for Local and metropolitan area
networks--Part 15.4: Low-Rate Wireless Personal Area
Networks (LR-WPANs) Amendment 1: MAC sublayer[Online]. Available:
https://ieeexplore.ieee.org/document/6185525/.

Figure 9: A field server system in a rice field.
We conducted the operational test in actual rice fields
using the 7 field servers of figure 2. The picture of the field
server system installed in the rice field is shown in figure 9.
We confirmed that the field server system works correctlyfrom rice planting to rice reaping.
In IEEE 802.15.4e [8], two types of time synchronization
methods, Beacon and Channel Hopping are defined. In thetime synchronization defined in both methods, it is requiredthat all nodes belonging to the network always synchronize thetime within an error of ±1 ms, thereby realizing the time
division access method. On the other hand, in the proposed
method, time synchronization is performed between themaster unit and each field server, but time synchronization
between the field servers is not performed. Therefore, timesynchronization accuracy of about ±10 seconds is sufficient, it
is not necessary to hold hardware for special timesynchronization and it is easy to put into practical use.
42

International Workshop on Informatics ( IWIN 2018 )Proposal of IoT System with SmartContract on BlockChain
Tetsuo Furuichi *, Hiroshi Mineno***

e-Cloud Computing&Co. / Graduate School of Informatics, Shizuoka University, Japanfuruichi.tetsuo.15@shizuoka.ac.jp**
Faculty of Informatics, Shizuoka University, Japan
mineno@inf.shizuoka.ac.jp

Abstract – In recent years, the demand for information
security for the IoT system has been increasing. For
information security, we made an IoT system using
BlockChains and evaluated the performance as an IoT system
in several types of node connection. And we propose theconfiguration method with the best performance. Specifically,by using SmartContract which is one function of theBlockChain as IoT Gateway, we implemented theauthentication of IoT module and data transfer function,
realizing the integrity and availability of information security.
Also, in systems using BlockChains already in practical use,the overhead associated with data transfer delay and mining
cycle time is disadvantageous, but from the evaluation results,
we have developed a realistic IoT system configuration using
BlockChains, and propose the requirements of BlockChainsspecialized for IoT.
Keywords: IoT, BlockChain, SmartContract

1edge side so far. However, specifications have becomecomplicated, and development time have been increasing.
Many IoT frameworks have been used as methods to decrease
those problems.

1.2 IoT security threat
As the number of IoT devices is increases, many
information security incidents including leakage andtampering of IoT data utilizing the vulnerability of thesedevices are reported much. In addition, there are various
attack methods using the vulnerability of the IoT system [2],not only functions and performance but also information
security measures including attack detection function andupdate function are important elements of IoT systemdevelopment. In the future, from the beginning of the design
it is necessary to practice "security by design" method tocounter information security.

1.3 BlockChain technology

INTRODUCTION

Encryption technology has evolved and abundant hardware
capable of executing cryptographic algorithms at practical
speed has become commonly available on PCs. By
implementing these cryptographic techniques, BlockChainsthat realized a distributed ledger system, which is one of the
distributed processing technologies, have received attention
in recent years. In the BlockChain, we have a mechanism
whereby mutual checking and sharing of each transaction in
the BlockChain cannot tamper with transaction information
without centralization. Many virtual currencies using
BlockChains have already been announced. Besides using
BlockChains in the finance field, application to primaryindustries including agriculture is also being considered. Foodtraceability in the agricultural sector is expected to utilize
BlockChains, and some consortium activities have alreadybeen started overseas as well. Tamper-proof reliable
information such as the place of production, producer,
production process and distribution route of agricultural
products is useful information for consumers as well. In
particular, we focused attention on functions that cannot betampered with as an information security system ofBlockChains.

The Internet of Things (IoT) has already been used forfactory and industrial products, and effects in agriculture havealso been reported. Recently, it has begun to be widely used
as consumer products targeting home and healthcare. By
connecting an embedded device to a server via a network, it
has become possible to acquire, accumulate and analyze
information that has not been used before. This information isuseful for improving our living environment and revitalizing
industrial activities and further development is expected in the
future [1]. However, due to the rapidly widely used IoTtechnology, some problems have arisen. In this section, we
explain situations where the IoT request is becoming
complicated due to diversification, and the problem that theIoT device is the next attack target of PCs and smartphones in
information security. We introduce an approach using
BlockChain technology which is one of the countermeasures
for these problems.

1.1 Component structure of IoT module
Many current IoT modules consist of simple parts of sensors
and microcontrollers for lower price and lower power
consumption. In recent years, with the development of smart
phones and tablet mobile devices, SoC (System on Chip)capable of running Linux has become a practical price, andmodules using them can be used as IoT devices. These IoT
oriented SoC have the merit of being able to execute advanced
processing with high load which could not be done on the

1.4 Application of BlockChain to IoT field
As requirements items for the IoT system, functions,performance, information security can be assumed.
Performance depends on the type of IoT, and it is necessary43International Workshop on Informatics ( IWIN 2018 )to consider the latency time of data acquisition and data
bandwidth. Also, at the same time, the IoT system needs totake measures against information security.
In recent years, IoT is also being used for agricultural
products production described in the previous chapter, anddata reliability is desired which cannot be tampered with byother persons or even operators even in history information
related to IoT.
Information security measures against an unspecified large
number of attacks can be implemented in various ways.
However, in a conventional IoT system, it is difficult toimplement a tamperproof mechanism including an operator.
In the prevention of tampering with all subjects, BlockChaintechnology is one of effective means for ensuring
completeness. In addition, although many IoT configuration
modules are composed of sensors, IoT Gateways, and cloudservers, each module may cause malfunction individually foreach reason, and availability measures of each module arealso important. In particular, since the IoT Gateway
constituted by the SmartContract of the BlockChain does not
limit the hardware to be executed, it is possible to realize not
only the function of preventing tampering inherent in the
BlockChain but also availability.
The confidentiality preservation of the IoT data on thecommunication line is a function necessary to prevent dataleakage. Because confidentiality cannot be secured in the
BlockChain, it is necessary to respond in another way.

2

security for the IoT system, confidentiality, integrity, andavailability, which are generally said three important subjects,
are important. In order to deal with the malware for IoT
mentioned earlier, securing access rights, prevention oftampering, and response to emergencies are becoming
important.

2.2 BlockChain
The BlockChain is a system capable of tracing alltransaction histories by a distributed consensus building
mechanism with network participants. So far, many kinds ofvirtual currency using BlockChain are distributed. Here we
introduce Bitcoin, the most famous BlockChain, andEthereum, a BlockChain system that can be applied toapplications.
Bitcoin was developed based on the BlockChain technology
posted by a person named Satoshi Nakamoto in 2008. It
started operation in 2009, and is a famous BlockChain forvirtual currency [3]. Bitcoin is a system composed of a
BlockChain node called Bitcoin client and a Bitcoin network.
The transaction information issued by the Bitcoin client issent as a transaction to the Bitcoin network, and minor, which
is a kind of Bitcoin client, miners, so that the block isgenerated. And that block is approved from multiple nodes of
the Bitcoin network.
Ethereum, proposed by Viralik Buterin's white paper in
November 2013, is a BlockChain and makes it possible tobuild applications by SmartContract [4]. While Bitcoin is
specialized in moving ownership of cryptographic currency,
Ethereum is characterized by being able to create and execute
distributed applications called SmartContracts as well as
moving cryptographic currencies [5] [6].
A BlockChain can be said a distributed database thatrealizes a "distributed ledger" that distributes and manages
transactions as exchange information on a distributed network.
We manage and operate a list of sequential data called"blocks" that summarizes those transactions on multiple
nodes. Moreover, the validity of the block is secured by themining processing using the distributed consensus algorithm.
PoW (Proof-of-Work) is mainstream in the current distributed
consensus algorithm. Under the agreement of theconfiguration node of the network, difficulty values are set
and have a mechanism to adjust the mining time. In addition
to PoW, a distributed consensus algorithm is studying
methods that do not spend processor resources or power
called PoS (Proof-of-Stake) or PoI (Proof-of-Importance).
Whereas Bitcoin has a mechanism specialized for virtual
currency trading, there is a BlockChain that can handle
SmartContracts, which is a type of program shared on the
BlockChain, as well as virtual currency transactions.
Ethereum is one of them, and each node can access virtual
currency transaction, mainly to execute virtual program.

PREVIOUS RESEARCH AND TASKWe describe IoT technology and BlockChain technology inthis section.

2.1 IoT
The growth of smartphones has reduced the cost of sensors
and the cost of infrastructure has declined due to thedevelopment of cloud computing. And due to promotion
from German Industrie 4.0 and various countries including
Japan, the IoT market is further expanding. IoT, which has
become popular, has begun to create new value by connecting
everything including people and things to the network [1]. In
particular, the IoT fields are diverse. 1) Health / medical
monitoring, 2) Machine monitoring, 3) Natural /environmental monitoring, 4) Human motion detection, 5)
Machine motion detection / control, etc. There are various
required real-time performances depending on the purpose.
The IoT system consists of 1) Sensor/actuator, 2) Preprocessing, 3) Communication, 4) Accumulation, 5) Analysis,
6) Inference, etc. The standard IoT system was specialized insensors, communication. For low power consumption,
miniaturization, low cost and high reliability, these areconfigured using one chip microcontrollers, and in manycases, they are implemented without OS or with RTOS (RealTime OS). The SoC having low power consumption and highperformance can be used at low cost, and the advanced IoT
system running Linux is used. As the system becomes moresophisticated, it has become possible to execute advanced
processing that is executed on the cloud server side and that
requires additional processor load. Concerning information

2.3 IoT + BlockChain
Focusing on the convenience of distributed management ofBlockChain and the characteristics of the virtual currency, thedegree of expectation for adaptation to IoT is increasing. In
the field of electric power systems, there are cases where
BlockChain efficiently perform IoT updates on an ongoing44

International Workshop on Informatics ( IWIN 2018 )basis [7]. To cope with IoT, a mechanism is developed tocover a BlockChain client program with a wrapper, and byusing a network different from the BlockChain, a weak datatransfer of the BlockChain is handled [8]. According toresearch to use IoT in Smart Home, the merit of information
security is larger than the overhead of processing
BlockChains [9]. In order to easily manage the configurationof the IoT device as a research concretely using
SmartContract, an IoT system that has the mechanism of RSA
key management in Ethereum's SmartContract has beenreported [10].

33.2 Sensor data flow
Each Node having a unique IP address forms a BlockChainby mutually connecting with the geth program executed on
Node. Each node has an address that is an account foraccessing the BlockChain, and communication among Nodes
is performed among addresses as transactions. This system
has IoT management API function to SmartContract which is
positioned as middleware existing in the BlockChain and tooperate as the IoT system throughout the BlockChain network.
Figure 1 shows the flow of processing of one example of thesystem using the BlockChain IoT module developed this time.
Data is sent from a node having a sensor (Sensor Node) to a
cloud server via a BlockChain network via a node having a
gateway function (Gateway Node).
Specifically, at Sensor Node, 1) IoT Application reads
information from the connected sensor via Sensor Library. 2)
The IoT Application passes the data to the SmartContract on
the BlockChain via the BlockChain Client (geth) and the
network. 3) Upon receiving the data, the SmartContract sends
an event to the IoT Application on the GateWay Node side
via the network of GateWay Node and the BlockChain Client(geth). 4) The IoT Application on the GateWay Node side
reads the sensor data stored from the SmartContract, and 5)
sends the data to the cloud server using the IoT Library.
Characteristically, in this connection, the Sensor Node and
the Gateway Node are able to exchange information with theunique address of the BlockChain to be connected and theinformation of only the address of the SmartContract to beconnected. That is, both Nodes can communicate even if they
do not know IP address each other. And the SmartContract isvirtually executed by hardware to be mined. In this example,
Mining Node becomes the execution machine of
SmartContract. At least one Mining Node is required.

ABOUT THE DEVELOPED IOT SYSTEM

In the IoT system, a registration of IoT devices and deliveryof data in a specific network is an important function. In thisresearch, we focused on the information security of theBlockChain and used geth which is Ethereum's BlockChainclient program of virtual currency which has already been put
into practical use. We worked with geth, developed an API
with the basic function of the IoT module interface, andimplemented it as a BlockChain IoT module in the embedded
system.

3.1 BlockChain IoT module
Requirements for a practical IoT system that meets thiscondition are flexibility, connectivity, extensibility andsecurity, for example. Then, the following five points were
defined as basic requirements. 1) use of general-purpose
network, 2) use of general-purpose hardware and OS, 3) preregistration of connectable sensor modules, 4) prompt
response at failure, abnormality, 5) secure access to the
network. In particular, in consideration of requirements of 3)to 5), this prototype system was constructed as a
SmartContract for management using virtual currency in the
BlockChain.
This hardware is based on embedded Linux. The basic
specifications are ARM-CPU, 1 GB of the main memory and
16 GB or more of the external storage, with network function
and sensor interface. This time, we prepared a module basedon Raspberry Pi 3b.
Ethereum's geth program running with this module is
implemented in Go language and executes on Linux / Unix
environment. The system is connected via a TCP/IP based
network, and its physical communication method may be
wired or wireless. Each device (hereinafter "Node") needs toassign a unique IP address. Node has made it possible to run
JavaScript API and application script which has interface ofgeth program and sensor and communication function withexternal cloud server. And handling of sensors andcommunication authority to an external network was set as aseparate policy from Node-to-Node connection using
BlockChain, and flexibility of connectivity as an IoT device
was secured.
The connection with the BlockChain is the jurisdiction of
the upper hierarchy without hardware dependency. Therefore,
when the corresponding module fails, even if the alternative
hardware has different performance, it can connect to theBlockChain.

Figure 1: Data flow between SmartContract and node

3.3 IoT Gateway by SmartContract
In this system, IoT module’s authentication function anddata delivery function are implemented in SmartContract of
the BlockChain, and this method is different implementationfrom the IoT device interface so far.
By connecting to the BlockChain via BlockChain client
software, the account address becomes the mutually approved45International Workshop on Informatics ( IWIN 2018 )be expected to prevent unscrupulous data access without
permission.

unique ID. Then, the data exchanged by that account ishandled as a transaction, and it is handled in the same way assigning with the account address. In consequence, theinterface in the BlockChain can ensure the integrity of
information security.
This IoT Gateway implemented in the SmartContract is a
distributed management API for connecting with the virtual
network for IoT, and it is registered and authenticated in the
BlockChain as a transaction in advance. And the BlockChain
information to be connected is preset in the BlockChain IoT
module developed this time. This module can physicallyconnect to the TCP/IP network and join a configured
BlockChain to synchronize with the active BlockChain and tocommunicate with its registered SmartContract.
As shown in Figure 2, this IoT Gateway roughly has twofunctions. The first one is a sensor management function andthe second one is a sensor data delivery function. The IoT
Gateway is registered as a transaction in the BlockChain withthe preset Owner Node account address. The account addressof that Owner Node will be the privileged user of theregistered the IoT Gateway and will have administrator
privileges. And sensor registration is executed with its Owner
Node account address.

Figure 2: IoT Gateway by SmartContract

4 PROTOTYPE IMPLEMENTATION AND
EVALUATION(1) Registration of Sensor Node
One of the IoT management functions is registration ofconnectable Sensor Nodes. This setting is proceeded by
Owner, the SmartContract creator. Owner Node sets theaccount address of a connectable Sensor Node as an argumentof the registration function and registers with the IoT
Gateway by generating a transaction.

To clarify the usage conditions of the configuration model
discussed in Chapter 3, we combined the IoT system with theexisting BlockChain and the evaluated characteristics as asystem. Especially we focused on the sensor data transfer
latency time from a viewpoint of one IoT system, andevaluated the dependency relationship with the logicalnetwork configuration and the number of mining nodes.

(2) Activation of Sensor Node
The Sensor Node registered in the IoT Gateway uses his
account address to access the IoT Gateway and activates its
own sensor data channel. Activation is programmed to spend
virtual currency as fees. Then it is possible to set the numberof data that can be buffered and set the fees for sensor dataexchanges.

4.1 Evaluation environment and method
Figure 3 shows the configuration of the environment used
for this evaluation. Ethereum Private Net was constructed by
implementing Ethereum 's client program geth on the server's container, Note PC and IoT module. Furthermore, IoT
SmartContract, which is an IoT API, is implemented in the
BlockChain net and it was created in advance as a transactionin the BlockChain.
Three types of Nodes were prepared for evaluation. The firstone is a general Node which is a contract Owner. The second
one is a Sensor Node that uploads sensor data to theBlockChain. The third one is a Gateway Node that takes
sensor data from the BlockChain. Each node assigns anaccount address of the BlockChain.
The evaluation script sends test data to the SmartContract
and measures the time the event returns from the
SmartContract. These sending and measurement are executedfor the specified number of times. Mining Node for blocking
transactions in the BlockChain is a Node running on a PC ora container connected to the network. And the execution
instruction of the Mining process was manually performed.

(3) Sending sensor data
The virtual currency is also required when Sensor Noderegisters data received from sensor device to IoT Gateway as
fees. This specification can expect the effect of suppressing
unnecessary data registration to the IoT system. Since we areassuming small capacity data delivery this time, we decided
to treat direct data as a transaction in the BlockChain. Whenthe sensor outputs a large amount of data, since the load of thetransaction in the BlockChain is large, it is necessary to havea separate data delivery mechanism.

In this implementation, when sensor data from Sensor
Node is stored in a preset number of buffers in the IoT
Gateway, this IoT Gateway asserts an event to noticeoutside. The Gateway Node is set to receive the event via
the geth program and receives data from the SmartContract
when it receives the event. This access requests fees of thevirtual currency of the amount set beforehand by the
Sensor Node at the time of activation, and this setting can46

International Workshop on Informatics ( IWIN 2018 )Figure 3: Overview of Logical nodes connection

4.2 Evaluation contents and results
In this research, two kinds of data transfer latency times
were evaluated.
(1) Data transfer latency time by physical connection
This evaluation measures the data transfer latency time in"Node - SmartContract - Node" by changing the physicalconnection form of Node in the BlockChain. We made 100
data accesses in each physical connection form.
Six types of physical connection were prepared. Figure 4
shows these connection modes. 'Y' in the green box is a Nodewith sensor data. 'Z' in the blue box is Mining Node. Pattern1 indicates that the constituent Nodes A to D are mutuallyconnected. In Pattern-2, configuration nodes A - D areconnected in an annular shape. Pattern-3-1 to Pattern-3-4 areconnection embodiments in which the number of Mining
Nodes is changed from Node having sensor data. The yellow
arrows indicate the expected direction of propagation of theissued transaction. The blue dotted arrows indicate the flowof the transaction after Mining.
Figure 5 shows the latency time results for each connection
pattern. There was no significant difference between theminimum value and the average value of the latency time inany of the physical connection patterns, but as for the
maximum value, the deflection becomes larger as the physicaldistance becomes farther.

Figure 4: Various Connection Patterns with 1-Miner

Figure 5: Latency time of each Pattern with 1-Miner(2) Data transfer latency time by mining number
Next, we measured the latency time when changing thenumber of mining nodes with the same net structure. Figure 6
shows the pattern of the BlockChain network evaluated. As
well as the evaluation of (1), the latency time of data transfer
was measured with a pattern in which Mining Node wasincreased from 1 to 4 patterns.
Figure 7 shows the latency time in each mining pattern. The
variance of the latency time when the mining number = 1 iswide, but as the mining number increases, the variation of thelatency time is contained.

Figure 6: Various patterns of Mining Nodes
47

International Workshop on Informatics ( IWIN 2018 )privileges is necessary to stop illegal acquisition of virtual
currency.
For SmartContracts, there are improvements in the methodof calling it. In the current BlockChain, the SmartContract iscalled with an address indicated by a number. However, due
to a malfunction or improvement of the SmartContract itself,
when version upgrade is done, it is necessary to change theaddress of the new SmartContract in some way. In order tosolve this problem, it is necessary to call a SmartContract
name or a secure delivery method of a SmartContract address.
By using the BlockChain, although overhead of theBlockChain itself occurs, it is thought that the feature thattransaction cannot be tampered can be effectively used in the
IoT system as a method of "Security by Design".

REFERENCES[1] Dave Evans, The Internet of Everything How More
Relevant and Valuable Connections Will Change theWorld, Cisco IBSG, (2012).
[2] Afreen Fatima Mohammed, Security Issues in
IoT, IJSRSET
Volume3
Issue8,
http://ijsrset.com/paper/3369.pdf, (2017).
[3] Satoshi Nakamoto, Bitcoin: A Peer-to-Peer Electronic
Cash System, https://bitcoin.org/bitcoin.pdf, (2008).
[4] Vitalik Buterin, A NEXT GENERATION SMART
CONTRACT & DECENTRALIZED APPLICATION
PLATFORM,
http://blockchainlab.com/pdf/Ethereum_white_papera_next_generation_smart_contract_and_decentralized_
application_platform-vitalik-buterin.pdf, white paper,(2014).
[5] Ethereum home page,https://www.ethereum.org/ , (Accessed on 06/03/2018)[6] Ethereum Homestead Documentation,
http://www.ethdocs.org/en/latest/ , (Accessed on
06/03/2018)[7] K. Christidis and M. Devetsikiotis, Blockchains andsmart contracts for the internet of things, IEEE Access,
4:2292--2303 (2016).
[8] M. A. Walker, A. Dubey, A. Laszka, and D. C. Schmidt,
PlaTIBART: a platform for transactive IoT blockchain
applications with repeatable testing, in 4th Workshop on
Middleware and Applications for the IoT (M4IoT),
December 2017.
[9] Ali Dorri, Salil S Kanhere, Raja Jurdak, and Praveen
Gauravaram, Blockchain for IoT security and privacy:
The case study of a smart home, In Pervasive
Computing and Communications Workshops (PerCom
Workshops), IEEE International Conference, 618–623(2017).
[10] S. Huh, S. Cho, S. Kim, Managing IoT devices using
blockchain platform, 2017 19th International
Conference on Advanced Communication Technology(ICACT), pp. 464-467 (2017).

Figure 7: Latency time on various Mining Patterns

5

CONCLUSION

In this prototype, by implementing the function of IoT
Gateway on the SmartContract of the BlockChain, we wereable to operate normally as an IoT system. However, wefound that the latency time of IoT data transmission may takemore than one minute. The result of the evaluation shows thatthe scope of application of this IoT system is not general
purpose and limited. For example, this system is suitable formonitoring natural environment, health, etc., which does notexpect real-time reaction, and it is not suitable for machine
control and warning detection expecting quick response.
However, since the propagated data inherits the basic functionof the BlockChain, no one can tamper with it.
Moreover, when using existing BlockChain as the IoT
system, each node has a logical connection link with plural
nodes, a stable mining cycle can be realized by preparing aplurality of mining nodes, and It is possible to reduce
variations in latency time of data propagation. Although thisevaluation used Ethereum, it is BlockChain which is used forordinary virtual currency transactions, and it is a system
assuming operation at PC and server, but it was able tooperate comfortably in IoT system as well. However, thecomputer resources to use seems to be slightly heavy in thecurrent IoT system, but future hardware evolution will reduce
those problems.
There are improvements in mining and SmartContract
implementation when assuming BlockChains specialized for
IoT. In mining, control of the mining cycle is necessary to
reduce the power consumption of the IoT system. With thisfunction, if data transmission does not come for a while,
mining can be stopped. In addition, granting of mining node48

Keynote Speech 1:
Mr. Kazuhiko Ohkubo(Vice President / Head of NTT
Secure Platform Laboratories)

International Workshop on Informatics ( IWIN 2018 )Cyber-physical Security in an Ageof Digital Transformation
September 10th, 2018
Kaz Ohkubo
NTT Secure Platform Laboratories
Copyright©2018 NTT corp. All Rights Reserved.

NTT Secure Platform Laboratories
Creation of security technologies contributing to new businesses,
resisting intensifying cyberattacks and using diverse data

Communication
services
New security
technologies for OT/IoT

Corporate
systems

Corporate
solutions

Defensive security

Newbusiness

Offensive security

Resistingincreasingly cleverand large-scale
cyberattacks

Malware prevention
Secure system platform
blacklist
Connected car
Cyberattacks prevention technology
securitycollaboration platform (LRR)

Effective technologies
supporting total security
operation

Security CoE
Security HR Development

Realizing safe and securecommunication and data use

Encryption technology
Advanced encryption applied systems
Secure computation

Integrated risk management
NTT-CERT Activities
Copyright©2018 NTT Corp. All Rights Reserved. 2

51

International Workshop on Informatics ( IWIN 2018 )Table of Contents / Amid environmental changes involving Cyberspace

1. Cat-and-Mouse
2. Non-IT ”IoT/OT” fields
3. Protecting critical infrastructure
4. Data utilization in society
5. Singularity (2045 problem)
Copyright©2018 NTT Corp. All Rights Reserved. 3

Olympic Destroyer
 For the PyeongChang Winter Olympics in February 2018, targeted attacks on Olympicsrelated organizations occurred from the beginning of the year, and several incidents caused
damage around the time of the opening ceremony…
・ Public Web site down・ Stadium wireless LAN down

・ Press center network connection down
・ Drones unable to fly, etc.

 NTT Laboratories obtained and analyzed the malware used in the attack⇒ Cyberattacks objective was clearly to “disrupt the event”. The techniques used andother information also suggested a high probability that Russia was involved

Machine A

Destructive
behavior

MalwareInitialcontaminationroute is unclear

exe1

Credential fraud
Copy of defrauded
credentials added
to self

exe2
exe3

Self copy
Remote copy and execution

More credentials
gained with each
infection

Machine B
MalwareCopyright©2018 NTT Corp. All Rights Reserved. 452

International Workshop on Informatics ( IWIN 2018 )Opposing Increasingly Clever and Large-scale Cyberattacks
Private company
defenseBusiness
MDR(Managed Detection & Response)

IOC

MSS

Network operation(Managed Security Service)

ISP

Malicious server list

Vulnerability/Attack
analysis products

BotnetConfig. Server groupWeb server attack
Domain reputation
NTT labs
Taint analysis
detection
elemental
Usable privacy
Vulnerability
and security
technologies
Analysis
Malicious file
detection

Forensics

ThreatInfection

Botnet estimation
by xFlow analysis
High speed DDoS
Detection using
xFlow analysis

Bot profiling

Infected Web server False/Attacksite
C2 Server
attackterminal

Vulnerability

Corporate/Home
Networks

Endpoints

Botnet Overview

ISP Networks

Backbone Networks

Being
monitored
Terminal

Web

Proxy

IPS/WAF

DPI

DNS

xFlow router
Copyright©2018 NTT Corp. All Rights Reserved. 5

Endpoint Security (Generating highly accurate IOC for MDR)
Customer

Detect overlookedcontaminated terminals!
Conventional

B
A

E
H

D

G
J

C

F

IOC

IOC

Highlyaccurate

I

Distribute IOC
Endpoints

Administrator
Remotehandling

MDR Service

Security Operations Center

DetectIOC
Distribution

Taint analysis
Identify malware‐specificbehavior

Symbolic execution
Extract environment‐
dependent behavior

Collect malware
・ Analysis request from customer
・ Malware in circulation

Advanced malware analysis technology

Taintanalysis

Symbolic
execution

IOC

Generate highly

accurate IOC
Copyright©2018 NTT Corp. All Rights Reserved. 6

53

International Workshop on Informatics ( IWIN 2018 )Usable Privacy and Security

User

Human Computer
Interaction (HCI)

System Security

SystemNetwork Security

UsablePrivacy and Security

Networking

SystemSystem• Research Approach (1): Discover privacy and security gaps and make system
improvements• [User aw areness/behavior]: W ork on observational experiments using real users to understand thecause of the user ’s unawareness or inappropriate operation• [System behavior]: Research attack technology (side-channel/vulnerability attack, etc.) andlook for potential security/privacy violations in real systems• Research Approach (2): Improve all types of countermeasure technology, based on
security gaps
• [End users]: Novice recognition, behavior styles, understanding principles used to deceive, andapply them in advanced honeypots that mimic human behavior• [Security Operators]: Understand expert analytical know-how and offer “security intelligence” byquantifying importance and procedures and formalizing knowledge• [Attackers]: Understand behavioral principles, like how targets are selected and use in attacker
identifying/counterattack technology as decoy systems/information that can fool attackers
Copyright©2018 NTT Corp. All Rights Reserved. 7

Table of Contents / Amid environmental changes involving Cyberspace

1. Cat-and-Mouse
2. Non-IT ”IoT/OT” fields
3. Protecting critical infrastructure
4. Data utilization in society
5. Singularity (2045 problem)
Copyright©2018 NTT Corp. All Rights Reserved. 854

International Workshop on Informatics ( IWIN 2018 )Mirai Attack Mechanism - Analysis by NTT Laboratories -Scan receiverDistributor
Receive scan results, download Mirai binary
to successfully logged in terminals

TargetBullet‐proof host (VPS)
Attacker protected by DB server
on hosting service

Database

CNC
Bot controller giving attack
instructions, alive/deadmonitoring, etc.

High‐speed
telnet port scanBrute‐force(dictionary) attack

https://krebsonsecurity.com/2016/10/source-code-for-iot-botnet-mirai-released/

Copyright©2018 NTT Corp. All Rights Reserved. 9

IoT Ransomware?! - Coming Soon -https://www.reddit.com/r/Bitcoin/comments/56m0qu/the_internet_of_ransomware_things/
Copyright©2018 NTT Corp. All Rights Reserved. 1055International Workshop on Informatics ( IWIN 2018 )New Security Technologies for IoT/OT(OT environment)
Field
Network

Authentication/
Authorization

Control
Network

IT in OT
Network(IoT environment)
IoT
Network

Next generation authentication
technology

Outside Target
DomainAuthenticity judgment

Configuration
management

IoT configuration mgmt. technology
IoT device anomalous behavior detection technology

DetectIndustrial protocol
attack monitoringand defense
technology

HandleIoT Orchestration technology

Copyright©2018 NTT Corp. All Rights Reserved. 11

Next Generation Authentication Technology■ Conventional method Issue 1: Passwords. Operation with simple passwords,
management of authentication data on authentication server■ Conventional method Issue 2: Certificates. Cost of issuing certificates and operations

Authentication client
ID
Password or
certificate notneededEncryption
operation
Various OSs
Secret data

Authentication protocol
ID, Authentication info.

Authentication server
No need to
manageauthentication
information foreach device

Authentication
resultProvisioning(send private data to
device)Initial registration
serverxGateway

Case (1):Case (2):Gateway authenticates devices
Cloud authenticates gateway

Copyright©2018 NTT Corp. All Rights Reserved. 1256

International Workshop on Informatics ( IWIN 2018 )IoT Configuration Management and IoT Security Orchestration
Technologies
Servers unregisteredin white-list

Operation
CenterServer underattackC&C
Server

Legitimate
server

Security Orchestration
 Generate communication white list
from IoT device communication
characteristics
 Control anomalous communicationfrom attacks, etc. (alerts, interception)xGW

IoT Configuration Management
 Accurately identify/estimate devices byanalyzing commonly used ARP frame
output characteristics and using noise
cancelling, even under harsh operating
conditions in a LAN environment, tounderstand the device configuration in
the LAN.
 Use graph theory, etc. to detect traffic
not from usual counterparts
 Discover IoT devices with vulnerabilities
from device characteristic information

Devices

Copyright©2018 NTT Corp. All Rights Reserved. 13

Industrial Protocol Attack Monitoring and Protection Technology
 InteRSePT® is composed of “Real-time detection/handling” and “Security integration management”
 Sensor and other data on the network is comprehensively monitored to detect malicious cyberattacks
using control commands that were difficult to deal with using earlier technology
 Security rules can be changed in real time for each operational state of the devices handled, to detect
anomalies quickly, handle unknown cyberattacks quickly, and maintain system availability.

(4) Comprehensively monitor theoverall behavior of the control
system and detect anomalies notdetected by the specificoperating state rules

InteRSePT®

Security integration
mgmt. equipment(1) Check sensor data &determine operating state
Rules for specificoperating states(2) Based on operating stateand detected anomalies,change communication control
rules for the real-time
detection/handling equipment

Power plant

Security control
equipment(3) Based on rules for
specific operatingstates, analyze
packets and pass orfilter themSensordata

Real-time detection/
handling equipment

Various sensors
Handled devices(sensors, actuators)

Real-time detection/
handling equipment
Various sensors
Handled devices(sensors, actuators)

http://www.ntt.co.jp/news2018/1804/180425b.html

Develop Market
Increase applicability
to industrial devices- Decrease cost and save
space by using general
purpose products- Reduce processing timeincreasing dataprocessing speed

New transportsystemChemical plant
Copyright©2018 NTT Corp. All Rights Reserved. 14

57

International Workshop on Informatics ( IWIN 2018 )Table of Contents / Amid environmental changes involving Cyberspace

1. Cat-and-Mouse
2. Non-IT ”IoT/OT” fields
3. Protecting critical infrastructure
4. Data utilization in society
5. Singularity (2045 problem)
Copyright©2018 NTT Corp. All Rights Reserved. 15

Cyberattack Targeting Critical Infrastructure
Dec, 2015

2009-2010

Jan, 2016

Oct, 2016

https://the01.jp/p0001741/
http://gblogs.cisco.com/jp/2016/03/syber-attack-in-ukraine/
https://eset-info.canon-its.jp/malware_info/trend/detail/160308.html
http://www.independent.co.uk/travel/news-and-advice/united-airlines-flight-delays-cyber-attack-heathrow-a7360641.html
Copyright©2018 NTT Corp. All Rights Reserved. 16

58International Workshop on Informatics ( IWIN 2018 )IC Passports: Significance and Authenticity Checking
An IC passport has a plastic card with a contactless IC(Integrated Circuit) chip built into the center of thepassport booklet. It stores basic passport information
including the passport holder’s name, nationality, birthdate, and passport number, as well as a facial image
read from the photo in the PDF of the passportapplication.
-----------------------------------------------------------The introduction of IC passports has made it easy to
discover forgeries that have substituted the facial image
by just comparing with the data recorded on the IC chip.
As devices able to compare the facial images recordedon the IC chip with the face of the person presenting it
are gradually introduced at ports of entry/exit in many
countries, it should be effective against attempts toimpersonate other people in the future.
Excerpt from: http://www.mofa.go.jp/mofaj/toko/passport/ic_kaishi.html
Copyright©2018 NTT Corp. All Rights Reserved. 17

Detecting System Falsification with Authenticity Checking
Technology• Build a chain of trust (trust reference points) and reliably detect any system falsification
occurring on a large-scale system, system-wide and from startup through operation
Ordinary software(control/communicati
on system, etc.)
Check

Trust
OriginAuthenticity

check
OS

Original-info.
Mgmt. center

Cloud server

Operation center

Trp

Server device
Trp

Hardware (Security chip)

Communication Network
Trp

Trp

Trp

Maintenancevendor

Trp

Control device“Trust reference points (Trp)”

Building the “chain of trust”

Realizing strong protection of basic data
used to check authenticity using asecurity chip to the latest international
standard (TMP2.0) and encryption
technology

Realizing a chain of trust able to checkauthenticity of entire facilities composedof (hundreds or thousands of) servers
with hundreds of thousands of files perserver.
Copyright©2018 NTT Corp. All Rights Reserved. 18

59International Workshop on Informatics ( IWIN 2018 )Anomaly Detection for IoT Devices and Control Communication• Implement technology to monitor the state of IoT device traffic using IoT-gateways placed in
the IoT network, and perform anomaly detection and cause classification using deeplearning. Also implement detection technology able to handle the particularities of control
communication.
Original-info.
Mgmt. center

Cloud Server

Operation center
Analysis
ServerServer Device
IoT GW

Communication Network
IoT GW

IoT GW

MaintenancevendorIoT GW

IoT GW

Control device

IoT Anomaly Detection Technology

Real Time Detection and Processing Technology

Implement detection that can adapt automatically todiversifying IoT devices and handle unknown
attacks by using AI technology (unsupervised deeplearning).

Implement detection that can handle particularities
of control communication, with signals from multiple
devices having unique packets overlapping withinseveral milliseconds.
Copyright©2018 NTT Corp. All Rights Reserved. 19

Next Generation Mobility Security
Countermeasure
in Cloud

Various Kinds
of Servicers

Car SOC

Service

Car Makers/
Suppliers

ServiceThreatVulnerability

Attack
Detection
in Cloud

False Positive

Attackers

ECU
Update- Interception of unauthorizedcommunication- Correction of ECU defect- Update of normal model Advanced Attack Detection in Cloud
Detection of unknown attack and presage by
massive-data analysis of various traffic/logs,
unification analysis of plural-vehicle data and
NW traffic correlation analysis

DCM

Attack
Detection
in Vehicle

 Quick Attack Detection in Vehicle
Quick detection of attacks that threaten safety
by analyzing important traffic/logs including
vehicle control messages in real time

GW

ECU

ECU

ECU

ECU

Countermeasure
in Vehicle

Copyright©2018 NTT Corp. All Rights Reserved. 20

60International Workshop on Informatics ( IWIN 2018 )Table of Contents / Amid environmental changes involving Cyberspace

1. Cat-and-Mouse
2. Non-IT ”IoT/OT” fields
3. Protecting critical infrastructure
4. Data utilization in society
5. Singularity (2045 problem)
Copyright©2018 NTT Corp. All Rights Reserved. 21

Treasure-trove Story
1. RealWorld

2.

IT information
cloud

Data

Photo Data

Video
data

Sensordata

Search data

Data

3. Analysis,

SNS posts

CameraimagesPublicdata

IoT

Learning

Location data

4. Feedback to the real world

https://codezine.jp/article/detail/9095
Copyright©2018 NTT Corp. All Rights Reserved. 22

61

International Workshop on Informatics ( IWIN 2018 )Anonymized Data Concept■ If personal data is anonymized, it can be provided to third parties without consent■ It can also be used for purposes beyond the provider’s intentions

Business gathering personal
information (provider)

Individual

Third Party (recipient)

Not agreeing toprovision to 3rd parties
Personal
data

Processing

Anonymized
data

Anonymized
data
Provision to 3rd parties(sold, etc.)

Processed so that personal
data cannot be reproduced

Re-identification prohibited

Copyright©2018 NTT Corp. All Rights Reserved. 23

Creating Anonymized Data
Anonymized data is “Data regarding people that has been processed so that a
particular person cannot be identified from the personal information”. In creating it,
all of all regulations 1 to 5 stipulated in Article 19 of the enforcement regulations *1
must be met.
Processing examples*2[1] Delete descriptions which can identify a specificindividual

Delete names, birthdates, street addresses.
and replace names of cities, states, etc.

[2] Delete personal identification codes

Delete biometric data (face, iris, fingerprints, palm
prints, etc.) converted to digital form, as well as
passport numbers, driver’s license numbers, etc.

[3] Delete linkage codes which link personal
information and obtained information

For service member data, decentralize managementof names and purchase histories, and delete IDs if
linked to an administrative ID.

[4] Delete idiosyncratic descriptions

Delete information about very expensive purchases,
the very elderly, etc.

[5] Take appropriate action considering theproperties and differences between descriptions
in personal information

If there is data regarding elementary school students
over 170 cm tall, replace it with “150 cm or taller”.

*1 Personal Information Protection Act Enforcement Rules (Oct. 5, 2016, Personal info. protection commission rules No. 3)
https://www.ppc.go.jp/files/pdf/290530_personal_commissionrules.pdf*2 Source: R, Osumi,K. Takahashi: “Personal Data Anonymization and Use,” (Seibunsha)
Copyright©2018 NTT Corp. All Rights Reserved. 24

62International Workshop on Informatics ( IWIN 2018 )Simple Anonymization Method● Simple deletion of names, addresses, etc. is not sufficient toprotect privacy
Name

Address

Sex

Age

Occupation

Name

Address

Sex

Age

Occupation

Sato

Shinjuku,
Tokyo

M

45

Company
Employee

Sato

Shinjuku,
Tokyo

M

45

Company
Employee

Suzuki

Mitaka,Tokyo

41

Company
Employee

Suzuki

Mitaka, Tokyo

M

41

Company
Employee

Abe

Shinjuku,
Tokyo

F

37

Homemaker

AbeShinjuku,
Tokyo

F

37

Homemaker

Nagasawa

Shinagawa,
Tokyo

F

35

Homemaker

Nagasawa

Shinagawa,
Tokyo

F

35

Homemaker

Funabashi,
Chiba
Chiba City,
Chiba
Kashiwashi,
Chiba

M

51

Self-employedM

57

Self-employedM

59

Self-employed

Yamamoto
Kobayashi
UchidaFunabashi,
Chiba
Chiba City,
Chiba
Kashiwashi,
Chiba

M

DeleteM

51

Self-employedM

57

Self-employed

Yamamoto
Kobayashi

M

59

Self-employed

UchidaThere is a risk that individuals could
be identified by combining attributes
and comparing with other data
Copyright©2018 NTT Corp. All Rights Reserved. 25

Advanced Anonymization Method (k-Anonymization)●Data processing based on “k-anonymity”, a typical index of safety–

k-anonymization: Data rounding (generalization) ⇒ Useful?

Name

Address

Sato

Shinjuku,
Tokyo

Suzuki

Mitaka,Tokyo

AbeShinjuku,
Tokyo

F

Shinagawa,
Tokyo

F

Nagasawa
Yamamoto
Kobayashi
UchidaFunabashi,
Chiba
Chiba City,
Chiba
Kashiwashi,
Chiba

Sex
M
M

Age

Occupation

Name

Address

Sex

Age

Occupation

45

Company
Employee

Sato

Tokyo

M

40s

Company Employee

41

Company
Employee

Suzuki

Tokyo

M

40s

Company Employee

Abe

Tokyo

F

30s

Homemaker

37

Homemaker

Generalize
Nagasaw
M

35
51

Homemaker

a

Tokyo

F

30s

Homemaker

Self-employed

Yamamot
o

Chiba

M

50s

Self-employed

Chiba

M

50s

Self-employed

Chiba

M

50s

Self-employedM

57

Self-employed

Kobayashi

M

59

Self-employed

UchidaCannot be narrowed down to lessthan k people with the same
information (k people with this data)
Copyright©2018 NTT Corp. All Rights Reserved. 26

63International Workshop on Informatics ( IWIN 2018 )Anonymization Method Developed by NTT (Pk-Anonymization)●Data processing based on a new index of safety with safety
theoretically equivalent to k-anonymization called Pk-Anonymization–

Pk-Anonymization: Data rewriting (randomization)

Name

Address

Sato

Shinjuku,
Tokyo

Suzuki

Mitaka,Tokyo

AbeShinjuku,
Tokyo

Nagasawa
Yamamoto
Kobayashi
UchidaShinagawa,Tokyo
Funabashi,
Chiba
Chiba City,
Chiba
Kashiwashi,
Chiba

Age

Occupation

Name

Address

Sex

Age

OccupationM

45

Company
Employee

Sato

Shinjuku,
Tokyo

M

57

Company
EmployeeM

41

Company
Employee

Suzuki

Mitaka,
Tokyo

M

41

Self-employed

Abe

Funabashi,
Chiba

F

37

Homemaker

Shinagawa,
Tokyo

M

35

Homemaker

M

51

Company
EmployeeM

45

Self-employedF

59

Self-employed

Sex

F
F

37
35

Homemaker
Homemaker

M

51

Self-employedM

57

Self-employedM

59

Self-employed

RandomizationNagasawaYamamotoKobayashi
UchidaShinjuku,
Tokyo
Chiba City,
Chiba
Kashiwashi,
Chiba

Maintains safety equivalent to
k-anonymization and preserves
data usability
Copyright©2018 NTT Corp. All Rights Reserved. 27

What is Secure Computation?
Compute the average of three peoples’ test scores,
without revealing the individual scores.

Ms. A(1) Partitioned
transmission
52
28

Mr. B

Randompartition so
the sum is 80

Computer 1

A: 52
B: 107
C: 12

107
-33
12

Ms. C
83

Scores of the threesubjects are neveravailable during thecomputation(52+107+12)/3=57(2) Computation using
partitioned data

(3) Compute result

57+26=
A: 28
B: -33
C: 83(28-33+83)/3=26
Computer 2
Copyright©2018 NTT Corp. All Rights Reserved. 28

64International Workshop on Informatics ( IWIN 2018 )First Implementation of Genome Analysis using Secure
Computation
NTT and Tohoku University Tohoku Medical Megabank (ToMMo) have for the first time,
implemented a method able to accurately analyze for correlations between human genome
variations and diseases, using secure computation technology (Fisher’s exact test)--July 12, 2016 press release--

Genome data

Medicalfacility A

1. Collect genome data
without releasing it toeach other

2. Implement
Fisher’s exact test

Test results
Medicalfacility A

Fisher’s exact test
Secure computationengineResearch
facility B

4

Research
facility B

Research
facility C

Research
facility C

1
3. Get only result ofanalysis, still encrypted.
Secure computation system®
Copyright©2018 NTT Corp. All Rights Reserved. 29“ SANSHI ”

Table of Contents / Amid environmental changes involving Cyberspace

1. Cat-and-Mouse
2. Non-IT ”IoT/OT” fields
3. Protecting critical infrastructure
4. Data utilization in society
5. Singularity (2045 problem)
Copyright©2018 NTT Corp. All Rights Reserved. 30

65International Workshop on Informatics ( IWIN 2018 )Singularity (2045 problem) - Kurzweil’s Law of Accelerating Change By 2045, a $1,000 computer will have performance of approximately 10 peta FLOPs, which is ten
billion times that of the human brain and a sufficient base for AI to reach a technical singularity

Source: Wikipedia “Technilogical singularity”
Copyright©2018 NTT Corp. All Rights Reserved. 31

Appearance of AI at the World’s Largest Security Hacking
Conference (2017)…

Research on automation of hacking, AI hacking, etc.
has already begun. The fact that Mayhem placed 14that DEFCON, solving problems without human help in
a contest with the best hacking teams in the worldshows how far R&D has progressed…
https://the01.jp/p0003285/
https://roboteer-tokyo.com/archives/5734

Copyright©2018 NTT Corp. All Rights Reserved. 32

66

International Workshop on Informatics ( IWIN 2018 )AI Hacking-related Technology (Identifying vulnerable points)
Computing similarity
Use a similar-text search algorithm to compute similarity between machine
language instructions and find replicated vulnerabilities…
Machine language
instruction sequencein an executable file being
searched

Machine language
instruction
sequence
at a vulnerability

push REG
mov REG REG
mov REG VAL
call MEM

Compute similarity

・・・

mov REG REG
push REG
mov REG REG
push REG
push REG
mov REG MEM
mov REG MEM
lea REG MEM

Similarity
xx%

・・・

Procedure for identifying vulnerabilities
1. Disassemble, normalize

2. Compute
similarity

3. Determine whether
vulnerability exists
Copyright©2018 NTT Corp. All Rights Reserved. 33

AI Hacking-related Technologies (identifying attack processes)
Potential API call extraction
Use static analysis to extract API call data (API names, parameters)from the parts of the original code not executed during dynamic
analysis…
Symbolically execute function internals
and extract API calls

Executed code blocks
markedExtract
unexecuted
functions

CreateFile(“a.txt”)

WriteFile("bbb")

Copyright©2018 NTT Corp. All Rights Reserved. 34

67

International Workshop on Informatics ( IWIN 2018 )Legal System using ML/AI
Technical development of AI will bring great change and diversification in human thought and behavior, andassumptions regarding societal structures, so research on legal systems, which function as societal standards
is also becoming crucial.

From development of AI to implementation in society

Regulations for AI vehicles and machinery
Resolving problems occurring in operation
Under what sort of system can personaldata, including medical/genome data, be
used by AI, etc. to contribute to society(2) Check current laws with knowledgeof AI and analyze individual concerns

Current laws

(1) Assuming AI will be applied, anticipate its
effects on people, society, and industry andrelationships among them(3) Propose a new legal system for
the AI age to create new legislationin the future・Protecting personal
information
・Privacy
・Infringement on other rights
etc.

Pros and cons of using AI for
evaluation and profiling of people．．．

AI

Related activity in Japan:
・Japanese Society for Artificial Intelligence “Ethical guidelines”
・MIC “AI Networked Society Promotion Council”

Promoteimplementation of an
AI society byrealizing a legal
system for the future*RIKEN Center for
Advanced Intelligence
Project (AIP) and NTT
Labs are dong joint
research in this area
http://ai-elsi.org/archives/471
http://www.soumu.go.jp/main_sosiki/kenkyu/ai_network/
Copyright©2018 NTT Corp. All Rights Reserved. 35

Thank you for your kind attention!

Moving Forward Together

Copyright©2018 NTT Corp. All Rights Reserved. 36

68Session 3:
Multimedia Systems( Chair: Katsuhiko Kaji )International Workshop on Informatics ( IWIN 2018 )A Method for Video Advertisement Insertion on Smartphone
Yoshia Saito**

Faculty of Software and Information Science, Iwate Prefectural University, Japany-saito@iwate-pu.ac.jp

Abstract –comfortably without feeling of interruption of their video
viewing by the video advertisement insertion. However,there are two issues in the previous method. The first issue
is that the previous method does not apply to various video
sharing services. This is because it needs special kind ofviewers’ comments with playback time such as commentson the niconico. The second issue is that the previous
method has room for improvement of viewers’ experience.
This is because it does not personalize the timing of video
advertisement insertion in spite of difference of the righttimings for each viewer.
In order to solve these issues, based on the fact that the
number of users watching videos on smartphones has
increased rapidly, it is worthwhile considering a method
using smartphone sensor information. In this research, we
use acceleration information from the accelerometer of thesmartphone at the time of watching video contents. We try
to find relationship between the acceleration information
and appropriate timings for video advertisement insertion.
Utilizing the relationship, we propose a method which
estimates an appropriate timing to insert a video
advertisement for each viewer.

In this research, we propose a method for video
advertisement insertion on smartphone. The method have analgorithm which estimates a comfort timing to insert a video
advertisement using the acceleration data at the time ofwatching a video. To create the algorithm, we formulated
three hypotheses; (1) Viewers who watch video contents on
smartphones sitting on chairs change their posture when
they take a short breath, (2) The postural change can bedetected by analyzing acceleration data from theaccelerometer of the smartphone, (3) The timings of the
postural changes correspondent with timings of a scenechange on the video content and one of the timings is anappropriate timing to insert a video advertisement. We
conducted a preliminary experiment and found thesehypotheses were true. On the basis of the finding, we
proposed an algorithm which estimates a comfort timing
using the acceleration data to detect a viewer’s posturalchange. The evaluation results showed accuracy rate of the
proposed algorithm was 86% and useful in terms of practicalusage.

Keywords: Video Advertisement, Insertion Algorithm,
Smartphone, Accelerometer

1

2

RELATED WORKINTRODUCTION
2.1

In recent years, video sharing services introduce abusiness model which inserts video advertisements in their
video contents in the same manner as TV. The businessmodel becomes popular with increase of smartphone users.
Major video sharing services such as YouTube [1] andniconico [2] provide smartphone applications to users andthe smartphone applications insert video advertisements tomake a profit. Therefore, many smartphone users have towatch video advertisements on the video sharing services.
There are three types of video advertisements, which are
pre-roll, post-roll and mid-roll. The pre-roll video
advertisement inserts a video advertisement before video
start. The post-roll inserts a video advertisement after video
end. The mid-roll inserts a video advertisement viewing avideo content like TV commercials and becomes popular inrecent years. Adobe reports the mid-roll video advertisement
is engaging commercials which have high completion
rate [3]. The mid-roll video advertisements will be usedeven further in the next several years.
We have proposed a video advertisement insertion
method which does not interfere with video viewing to makeviewers accept the video advertisements [4]. In the previous
work, it analyzes characteristics of viewers’ comments forthe video content. It enables viewers to watch videos moreVideo Advertisement Insertion

There are studies of interactive advertising to provide
interactivity to the advertising [5-8]. The interactive
advertising allows selecting appropriate ads according to theviewers and changing video length and display methods.
Our research is regarded as one of technologies forinteractive advertising. Tao Mei et al. [9] proposed a schemeof appropriate video ad insertion for online videos. In thisresearch, the appropriate timing for video advertisement
insertion is determined detecting an unattractive video shotboundary. The unattractive video shot boundary is detectedby importance of the scene audio-visually. Since thisresearch analyzes video image and audio in detail, theprocessing cost will be high when it applies to a large
number of videos on the video sharing services. Our study
aims to find other approaches which estimate theappropriate timing for video advertisement insertion without
heavy audio-visual processing.

3

HYPOTHESES FORMULATIONIn this section, we indicate possibility that there is arelationship between human motion and degree of interest.
We also describe existing techniques of sensing for human71International Workshop on Informatics ( IWIN 2018 )motion to choose what sensor is appropriate for estimating
human motion on smartphones. Then, we formulate
hypotheses in order to create a new method.

3.1

and one of the timings is an appropriate timing toinsert a video advertisement.
If these hypotheses are true, we can create a new method
which estimates an appropriate timing for each individual toinsert a video advertisement by analyzing acceleration datafrom the accelerometer of the smartphone.

Human Motion and Degree of Interest

There are several techniques for estimating human motion
by sensor devices on the smartphone. There are also somestudies which show a relationship between human motion
and degree of interest.
The relationship between eye motion and degree ofinterest is well-known [10-13]. The data of eye motion canbe acquired by eye-tracking techniques. However, highaccurate eye-tracking techniques require special devices foreye tracking or strict restrictions of the measuring
environment. It is difficult for smartphone users to prepare
for the special devices and strict restrictions force
inconvenience upon the users. For these reason, the eye
motion is not suitable to estimate viewers’ degree of intereston the smartphone.
The relationship between posture and degree of interest is
mentioned [14]. People change their posture at intervals
from 15 to 20 minutes at the time of sitting because of
fatigue [15, 16]. Meanwhile, their postural change hardly
occurs when they are interested on something. We can apply
this knowledge to a method which estimates an appropriate
timing to insert a video advertisement for each viewer if we
can detect viewers’ postural change by smartphone sensors.
There are a lot of techniques to estimate body motion
using sensors. Visual analysis using video data taken bycamera devices is one of the techniques. However, usage ofcamera devices causes large power consumption and it is a
disadvantage especially on the smartphone. Visual analysis
is not appropriate to estimate body motion. Estimation of the
body motion using acceleration information is popular andlight-weight techniques [17, 18]. Most of smartphones haveaccelerometer and many researchers study estimating body
motion of the smartphone users from the acceleration
information. These research shows various states ofsmartphone users such as sitting, standing, walking, running,
going up and down the stairs and so on can be discriminated.
Usage of accelerometer to detect postural change ofsmartphone users is reasonable.

3.2

4

We conduct a preliminary experiment to reconfirm if
there are difference of right timings to insert a video
advertisement for each viewer and test the hypotheses.

4.1

2.

3.

Methodology

We select 5 videos at random from several videos whichwere used in the previous work. We ask 3 participants forcooperation, who are in their twenties and thirties and haveused some video sharing services. At first, we explain about
the experiment to the participants. They watch the 5 videos
in random order on a smartphone sitting on a chair. We
shoot a video to record their watching situation. In thesmartphone, an application which records acceleration datafrom the accelerometer is running. After watching thevideos, we carry out a questionnaire survey to ask theparticipants top 3 comfort timings if a video advertisement
is inserted. We explain the participants the length of thevideo advertisement is about 15 seconds. Then, weinterview the participants showing the recorded video.
Table 1 shows the overview of the experiment.
Table 1: Overview of the experimentParticipants

3 people in 20s and 30s who have used video
sharing services

Smartphone Nexus 5VideosVideo 1: 3D action game [19]Video 2: 2D action game [20]Video 3: 3D action game [21]Video 4: 2D action game [22]Video 5: 3D action game [23]Condition

Sitting on a rotary chair with backrest (seatheight: 30 cm)

Procedure

1. Receive an explanation about the overview
of the experiment
2. Watch the 5 videos in random order
3. Answer top 3 comfort timings if a video
advertisement is inserted
4. Take an interview about the comfort timings.

Hypotheses

Our previous work shows appropriate timings for video
advertisement insertion corresponded with timings of scenechanges on the video content. The viewers may change their
posture in scene changes on the video content. We formulate
three hypotheses as follows.
1.

PRELIMINARY EXPERIMENT

Viewers who watch video contents on smartphones
sitting on chairs change their posture when they takea short breath.
The postural change can be detected by analyzing
acceleration data from the accelerometer of thesmartphone.
The timings of the postural changes correspondent
with timings of a scene change on the video content

4.2

Results

At first, we reconfirm whether there are differences ofright timings to insert a video advertisement for each viewer
or not. Tables 2-6 show the top 3 comfort timings in the 5videos for each participant. These results show there aredifferences of the right timing for each participant. We
found the participants had characteristic features to select72International Workshop on Informatics ( IWIN 2018 )the comfort timings. The participant A tended to select the
1st comfort timing in early scenes. The participant B tended
to select the 1st comfort timing in late scenes. We
reconfirmed that it was necessary to personalize the timingof video advertisement insertion because there wasdifference of the right timings for each viewer.
We also checked the recorded video. The participantschanged their posture in the scene change and the timings ofthe postural change matched their comfort timing. The
postural changes rapidly increased the resultant accelerationof 3-axis. These results show monitoring the rapid increaseof the resultant acceleration can detect postural changes ofthe viewers and estimate one of their comfort timings toinsert a video advertisement in the video content.

3-axis. On the basis of the finding, we create an algorithm
for video advertisement insertion.

5.1
Algorithm for Video Advertisement
Insertion
Figure 1 shows a flowchart of the algorithm for video
advertisement insertion using an accelerometer on thesmartphone. In the algorithm, it calculates a test statistic
based on acceleration values for outlier detection whichmeans occurring the viewer’s postural change. The test
static Ti can be calculated using Accelerationt, Et, SDt attime t. Accelerationt denotes the resultant acceleration of 3axis at time t. Et denotes the average of the resultant
acceleration from the video start to time t. SDt denotes thestandard deviation of the resultant acceleration from thevideo start to time t. Tt is calculated by the following
equation.

Table 2: Comfort timings for each participant in video 1
Video 1
Participant A

Participant B

Participant C

1st comfort timing

01:47

07:00

01:08

2nd comfort timing

00:09

06:43

01:47

3rd comfort timing

06:30

01:47

06:43

Tt = | ( Accelerationt - Et ) | / SDt
The outlier can be detected when the following inequality
is completed.

Table 3: Comfort timings for each participant in video 2
Video 2
1st comfort timing

Participant A

Participant B

Participant C

01:00

09:49

01:23

2nd comfort timing

01:41

10:41

10:41

3rd comfort timing

09:41

05:45

09:41

Tt > 2 * SDt
If the outlier is detected, the algorithm waits for reachinga next shot boundary. In the next shot boundary, the video
content is stopped and a video advertisement starts. Aftercompletion of the video advertisement, the video content
restarts and the algorithm terminates the process of the video
advertisement insertion.

Table 4: Comfort timings for each participant in video 3
Video 3
1st comfort timing

Participant A

Participant B

Participant C

01:23

10:50

07:09

2nd comfort timing

02:19

11:11

10:50

3rd comfort timing

07:09

06:14

11:11

Start

Calculate the following values at fixed intervals.
Resultant acceleration of 3-axis at time t :Accelerationt
Average of the resultant accelerationfrom the video start to time t :EtStandard deviation of the resultant accelerationfrom the video start to time t :SDtTable 5: Comfort timings for each participant in video 4
Video 4
Participant A

Participant B

Participant C

1st comfort timing

01:39

07:10

01:39

2nd comfort timing

01:44

09:38

07:10

3rd comfort timing

07:10

10:16

03:45

Test statistic at time t :Tt = |(Accelerationt – Et )| / SDt

Tt > 2 * SDt

Table 6: Comfort timings for each participant in video 5
Video 5
Participant A

Participant B

1st comfort timing

03:41

10:51

05:44

05:44

06:50

06:50

3rd comfort timing

06:50

11:15

11:15

5

false

true

Participant C

2nd comfort timing

false

t++

Video
end

true

Wait for reaching a shot boundary

Stop the video content and play a video advertisement

Restart the video content

PROPOSED METHOD

The preliminary experiment shows the possibility of
estimating a comfort timing for each viewer to insert a video
advertisement utilizing their postural change which can bedetected by the rapid increase of the resultant acceleration ofEnd

Figure 1: The flowchart of the algorithm for video
advertisement insertion73

International Workshop on Informatics ( IWIN 2018 )5.2

we explain about the experiment to the participants. They
watch the 5 videos in random order on a smartphone sitting
on a chair. We shoot a video to record their watchingsituation. In the smartphone, an application which records
acceleration data from the accelerometer is running. Afterwatching the videos, we carry out a questionnaire survey to
ask the participants top 3 comfort timings if a video
advertisement is inserted. We explain the participants thelength of the video advertisement is about 15 seconds.
After getting data of acceleration and comfort timings, weestimated a comfort timing for each viewer by using the
proposed algorithm. We also estimated a comfort timing byusing the previous algorithm which used viewers’ commentson the niconico. Then, we compared the result of the
proposed algorithm with one of the previous algorithm.

System Design

We assume the algorithm for video advertisement
insertion is implemented on the smartphone as anapplication for video viewing. A video advertisement andacceleration data are input to the algorithm. The algorithm
outputs less than one comfort timing for each viewer in thevideo content.
Figure 2 shows the system design of the proposed
method. The proposed system have two servers, which arefor video sharing and video advertisement. Video uploaders
submit their video contents to the video sharing server.
Advertising sponsors provide video advertisements to thevideo advertisement server. Video viewers havesmartphones with a smartphone application for video
viewing which has the algorithm for video advertisement
insertion. The smartphone application plays a video content
from the video sharing server. Shot boundaries of the video
content are detected by existing techniques for shotboundaries detection and it lies outside the scope of ourresearch. While the video viewer is watching the video
content, the algorithm for video advertisement insertion
monitors acceleration data of the smartphone. The
smartphone application stops the video content temporarilyand starts a video advertisement from the video
advertisement server when the algorithm estimates thetiming is comfort for the viewer. After that, the video
content restarts without any more video advertisement in theviewing. Note that we suppose the algorithm works only
when the video viewer is sitting on a chair and the viewer
watches the video holding the smartphone without video
skip. In case of other conditions, new routines should be
added to the algorithm.
Video
contents
Video
uploadersVideo ads
AdvertisingsponsorsVideo
sharing
serverVideo
advertisement
serverVideo
content

Video adSmartphone
application forvideo viewing
Algorithm forVideoadvertisement
insertion

6.2

Table 7 shows the result of comparing an estimatedtiming of the proposed algorithm with top 3 comfort timings
for each viewer. Table 8 shows the result of comparing an
estimated timing of the previous algorithm with top 3
comfort timings for each viewer. In these tables, “1st”
means the estimated timing coincides with the 1st comfort
timing for the viewer. The same applies to “2nd” and “3rd”.
“n/a” means the estimated timing does not coincide with any
top 3 comfort timings. We regard the estimated timing is anappropriate timing if it coincides with one of the top 3
comfort timings.
From Table 7, the proposed algorithm could estimate 43
appropriate timings of the 50 chances. The accuracy rate of
the proposed algorithm was 86%. On the other hand, theprevious algorithm could estimate only 22 appropriate
timings of the 50 chances as shown in Table 8. The accuracy
rate of the previous algorithm was 44%. Comparing theseresults, the proposed algorithm improved the accuracy rate
more than 40%. Personalization of a timing to insert a video
advertisement contributed the improvement of accuracy rate.

Play
video

Insertvideo adResultsVideoviewerTable 7: Result of comparing an estimated timing of the
proposed algorithm with top 3 comfort timings for eachviewerFigure 2: System design of the proposed method
Participant A

6

EVALUATION

We evaluate the performance of the proposed algorithm
for video advertisement insertion using an accelerometer on
the smartphone. Comparing the previous algorithm, we
verify the proposed algorithm estimates a better comfort
timing for each viewer.

6.1

Methodology

The evaluation is conducted in the same manner as thepreliminary experiment. We use 5 videos which are same
videos in the preliminary experiment. We ask 10participants for cooperation, who are in their twenties andthirties and have used some video sharing services. At first,
74

Video 1

Video 2

Video 3

Video 4

Video 5

1st

2nd

1st

2nd

n/a

Participant B

1st

1st

2nd

n/a

n/a

Participant C

1st

3rd

1st

1st

1st

Participant D

3rd

1st

3rd

n/a

1st

Participant E

1st

1st

2nd

1st

3rd

Participant F

1st

1st

1st

2nd

2nd

Participant G

2nd

3rd

1st

1st

n/a

Participant H

1st

2nd

1st

n/a

1st

Participant I

1st

n/a

3rd

2nd

3rd

Participant J

2nd

1st

1st

1st

1st

International Workshop on Informatics ( IWIN 2018 )Table 8: Result of comparing an estimated timing of theprevious algorithm with top 3 comfort timings for eachviewerParticipant A

7

Video 1

Video 2

Video 3

Video 4

Video 5

n/a

n/a

n/a

1st

n/a

Participant B

n/a

n/a

1st

n/a

3rd

Participant C

n/a

n/a

2nd

1st

n/a

Participant D

n/a

n/a

3rd

3rd

n/a

Participant E

n/a

3rd

1st

3rd

n/a

Participant F

n/a

2nd

3rd

n/a

2nd

Participant G

3rd

n/a

1st

2nd

n/a

Participant H

n/a

n/a

1st

n/a

2nd

Participant I

n/a

n/a

n/a

3rd

n/a

Participant J

3rd

3rd

1st

n/a

n/a

[6] Jong Woo Kim and Stephen Du: Design for an
Television
Advertising
System,Interactive
Proceedings of the 39th Annual Hawaii International
Conference on System Sciences, Vol. 2 (2006).
[7] J. Lloyd: I-Ads - a new approach, European
Conference on Interactive Television (2003).
[8] Panagiotis Giotis, George Lekakos: Effectiveness ofInteractive Advertising Presentation Models, EuroITV'09, pp.157-160 (2009).
[9] Tao Mei: VideoSense-Towards Effective Online Video
Advertising, ACM Multimedia'07, pp.1075-1084(2007).
[10] Jeffrey Heer and Stuart K. Card: Efficient userinterest estimation in fisheye views, CHI '03 Extended
Abstracts on Human Factors in Computing Systems,
pp. 836-837 (2003).
[11] Anthony Santella and Doug DeCarlo: Robust
clustering of eye movement recordings forquantification of visual interest, Proceedings of the2004 symposium on Eye tracking research &applications, pp. 27-34 (2004).
[12] Tina Walber, Chantal Neuhaus, Steffen Staab, Ansgar
Scherp and Ramesh Jain: Creation of individual photoselections: read preferences from the users' eyes,
Proceedings of the 21st ACM international conference
on Multimedia, pp. 629-632 (2013).
[13] Vanessa Georges, François Courtemanche, Sylvain
Senecal, Thierry Baccino, Marc Fredette and PierreMajorique Leger: UX Heatmaps: Mapping User
Experience on Visual Interfaces, Proceedings of the2016 CHI Conference on Human Factors in
Computing Systems, pp. 4850-4860 (2016).
[14] Selene Mota and Rosalind W. Picard: Automated
Posture Analysis for detecting Learner’s Interest Level,
Computer Vision and Pattern Recognition Workshop(2003)[15] P.Branton: Behaviour, Body mechanics andDiscomfort， Ergonomics, Vol.12, No.2 (1969).
[16] Hidetoshi Watanabe, Masao Ando and Takashi
Takahashi: Transition of sitting posture over time, J.
Archit. Plann. Environ. Eng., AIJ, No. 474, pp. 107114 (1995).
[17] Toshiki Iso and Kenichi Yamazaki: Gait analyzer
based on a cell phone with a single three- axis
accelerometer, ACM the 8th Conference on Human
Computer Interaction with Mobile Devices andServices (MobileHCI2006), pp. 141-144 (2006).
[18] Ronit Slyper and Jessica K. Hodgins: Action
capture with accelerometers, Proceedings of the 2008
ACM SIGGRAPH/Eurographics, pp. 193-199 (2008).
[19] niconico, http://www.nicovideo.jp/watch/sm5457137[20] niconico, http://www.nicovideo.jp/watch/sm6979644[21] niconico, http://www.nicovideo.jp/watch/sm2750853[22] niconico, http://www.nicovideo.jp/watch/sm4895582[23] niconico, http://www.nicovideo.jp/watch/sm8481759

CONCLUSION

In this paper, we proposed a method for video
advertisement insertion using acceleration data on
smartphone. From the preliminary experiment, we got 3
findings; (1) viewers who watch video contents on
smartphones sitting on chairs change their posture when
they take a short breath, (2) the postural change can bedetected by analyzing acceleration data from theaccelerometer of the smartphone, (3) the timings of the
postural changes correspondent with timings of a scenechange on the video content and one of the timings is anappropriate timing to insert a video advertisement. Then, we
created an algorithm for video advertisement insertion anddesigned its system. From the evaluation results, we foundthe proposed algorithm improved the accuracy rate morethan 40% comparing with the previous algorithm. This
result showed the effectiveness of the proposed algorithm.
For the future work, we will try to study other algorithms
even if the viewer does not sit on a chair. We also study amethod which switch mid-roll to pre-roll or post-roll video
advertisement when there are not clear scene changes in thevideo content.

REFERENCES[1] YouTube，http://www.youtube.com/[2] niconico，http://www.nicovideo.jp/[3] 2012 adobe digital video advertising report,
https://blogs.adobe.com/primetime/files/2013/11/Monetization-Report_FINAL1.pdf[4] Yoshia Saito: A method for video advertisement
insertion with audience comments on action game
videos, International Workshop on Informatics(IWIN2017), pp.153-158 (2017).
[5] K. Risden, M. Czerwinski, S. Worley, L. Hamilton, J.
Kubiniec, H. Hoffman, N. Mickel and E. Loftus:
Interactive advertising: patterns of use andeffectiveness, SIGCHI, pp. 219-224 (1998).

75

International Workshop on Informatics ( IWIN 2018 )Experiments to investigate the effects of scents on vection perception
Aoi Aruga*, Yuichi Bannai**, and Takeharu Seno***s*

Graduate School of Engineering, Kanagawa Institute of Technology, Japan
Department of Information Media, Kanagawa Institute of Technology, Japan***
Faculty of Design, Kyushu University, Japan
Emails: S1885007@cce.kanagawa-it.ac.jp, bannai@ic.kanagawa-it.ac.jp, seno@design.kyushu-u.ac.jp**

Abstract - Vection is an illusion that gives the feeling of
motion in the absence of bodily movement. Thisphenomenon may occur with presentation of a screen
displaying patterns with optical flows. In an immersive
environment that uses a head mounted display (HMD),
including virtual reality systems, vection is frequently
induced. Additionally, it has been reported that certain
scents can affect the perception of bodily movement and arethought to have some influence on vection. We aimed toinvestigate the effect of scents on vection perception.
Subjects were seated in front of an olfactory display while
wearing an HMD and were presented with moving images
to induce vection under several conditions: scentpresentation, sound presentation, and presentation of no
additional stimuli. We found that the scent stimulus did notaffect the perception of vection. However, there is atendency that there is a difference in the stimulus affected
depending on the vection direction. Furthermore, it appears
that lavender may promote the perception of vection.

Additionally, it has been reported that certain scents affectthe movement of the body and are thought to influencevection, where the stimulus causes the sense of motion. An
experimental example has been reported, where elderly
people, aged 65 years or older, in nursing homes, were
presented with the scent of lavender for 12 months. The
proportion of people who fell and the incidence rate perperson in these participants was lower than those who were
not presented with lavender. Moreover, the CohenMansfield Agitation Inventory score, which is used as anindicator of dementia, was also lower in the group that waspresented with lavender [4]. Sakamoto et al. (2012)
hypothesize that the scent of lavender may reduce thelikelihood of falls and agitation in nursing home residents.
However, few studies have investigated the relationshipbetween vection and olfactory stimulation.
Subjects were seated in front of an olfactory display while
wearing an HMD and were presented with two types ofmoving images (expansional and contractional optical flow)
under the conditions of with and without scents. During themoving image presentation, we measured the latency andduration of vection. After finishing the stimuli presentation,
the subjects evaluated the strength of the vection that theyexperienced using subjective values. This allowed for theexamination of the relationship between vection andolfactory stimuli.

Keywords: vection, sense of smell, scent, sense of sight,
HMD, olfactory display

1

INTRODUCTION

Vection is the visually induced illusion of self-motion [1],which may be felt when viewing a screen that is displaying
patterns with optical flows. It is known that vection is
induced not only by visual stimulation but also by auditoryand somatosensory stimulation. Sakamoto et al. (2004)
reported that sound images with movement from front toback or back to front induces linear self-motion perception
and that the self-motion direction is influenced by the
direction of the motion of auditory stimuli [2]. This showsthat auditory information also has a great influence on selfmotion. Vection is also caused by somatosensory sensation.
Murata et al. (2014) performed experiments in which
participants wore eye masks and were presented with white
noise through a pair of earphones. A horse riding machine
was used to produce bodily movement in the participant.
Almost all the participants in this condition felt a forward
motion sensation on presentation of a constant stream of air
to their front [3].
In recent years, many trials have been conducted in which
scents are displayed in conjunction with movies using
virtual reality (VR) systems. In an immersive environment,such as a VR game using a head mounted display (HMD),
vection is frequently induced.

2

METHODOLOGY
2.1 Olfactory Display

We used the Fragrance Jet 2 olfactory display (Figure 1).
This display uses the techniques of an ink-jet printer toproduce a jet, which is broken into droplets with a small
hole in the ink tank. Bubbles are formed in the ink by
instantaneous heating, and ink is ejected due to the pressure
created by the bubbles. The display can set up one ejection
head, which can store three small tanks and one large tank;thus, this display can contain a maximum of four kinds ofscents. There are 127 minute holes that are connected to thesmall tank and 256 minute holes in the head that areconnected to the large tank. Since the display can emit scentfrom multiple holes at the same time, the ejection quantity
can be set from 0 to 127 (in the small tank) or 0 to 256 (inthe large tank). We denote the average ejection quantity at
one time from each hole as the “unit average ejection
quantity” (UAEQ), and the number of minute holes that are
emitting at one time as “the number of simultaneous77International Workshop on Informatics ( IWIN 2018 )Table 2: Result of olfactory detection thresholdmeasurement
LavenderBananaAverage
2.67
3.67
SD
1.97
2.07
Maximum
5
7
Minimum
1
1

2.3

Figure 1: Olfactory Display: Fragrance Jet 2

To induce vection, two kinds of moving images were used.
The expansion stimulus was used to induce a feeling of
forward movement which is frequently experienced ineveryday life and VR games. When a dot reached the edgeof the screen and disappeared, a continuous stimulus
presentation could be obtained by rearranging the dot fromthe center of the plane. The contraction stimulus was used toinduce a feeling of backward movement. A dot was
repositioned from the edge of the plane until it reached thecenter of the plane and disappeared. In each stimulus, a totalof 2400 dots were displayed on the screen. The size of thedot on the screen was changed to be physically constantaccording to the distance change simulation. As the dots did
not have a density gradient and did not give a static depthcue, the depth cues were only provided by motion. The
speed of each dot was approximately 3.7° per second at theviewing angle. Figure 2 shows a still image of the vectioninducing visual stimulus, where arrows indicate movementof points.
The visual stimuli were presented using Oculus Rift DK2.
The viewing angle was set at 110° in the diagonal direction
and 90° in the horizontal direction and the motion stimuli
that induced vection was displayed throughout the screen.

ejections” (NSE), which we refer to as the ejection level.
The UAEQ from one minute hole in the small tank is 4.7 pland 7.3 pl in the large tank. The reproducibility of thesevalues was confirmed without depending on the residual
quantity of ink on examination. As the emission occurs 150
times per unit time (100 ms), the ejection quantity (EQ) in
the small tank was calculated as follows:
EQ (pl) = 4.7 (pl)(UAEQ) × (from 0 to 127)(NSE) × 150(times)Additionally, the display was equipped with a fan and 9phases of wind velocity control in the range of 0.8-1.8 m/sec
were used.

2.2

Scent Stimuli

Two kinds of scents were used in this experiment:
lavender (oil of lavender) and banana (iso-amyl acetate).
These scents ware diluted to 5% with ethanol and water, andthe component ratios of each perfume are shown in Table 1.
Table 1: Scent stimuli
ScentsLavenderScent (%)
5Component
Ethanol (%)
65
ratiosWater (%)
30

Vection Stimuli

Banana575
20Expansion

To determine the ejection level at which the subjects could
sense the scent, we measured the detection threshold, whichis the minimum detectable concentration of scent. Thismethod is based on the two-point comparison method thatwas proposed by the Japanese association of odor
environment. The initial ejection level was set to five. If thesubject correctly identified the scent twice at the initial level,
we reduced the level by two according to the descentmethod and ended the measurement when the subject could
not identify the scent. When the subject was unable tocorrectly identify the scent at the initial level, we adopted
the rising method and increased the level by three. The
measurement was complete when the subject identified thescent correctly twice. Based on this result, the ejection level
to be used was determined. Table 2 shows the measurement
values of six participants (between the age of 20 and 30
years, male).

Contraction
Figure 2: Still image of vection stimulus

2.4Sound Stimuli

As a control condition, sound stimuli using a pure tone at
the frequency of 440 Hz and 0.1amplitude were presented toinvestigate the influence of olfactory stimuli on vection
more clearly. For the sound presentation, a set of sealed
dynamic headphones (SE-MJ 522, Pioneer) was used, andthe playback volume was set to the sound that would
normally be heard through a speaker.
78

International Workshop on Informatics ( IWIN 2018 )2.5

Experimental Conditions

All experiments were performed while the subject wasseated on a chair with their jaw positioned on a chin rest.
The subjects wore HMDs and headphones throughout theexperiments. The distance from the ejection exit of the headof the olfactory display to the nose was fixed at 225 mm.
The olfactory display was set up with the presentation port
facing diagonally upwards, and the height of the jaws was
adjusted by each subject so that the wind from the display
hit their face firmly. Figure 3 shows the actual experimental
condition. The wind speed was set to level 9, which is the
maximum setting (on a scale of 1 to 9). The measured windspeed under these conditions was approximately 1.8 m/sec.
During the experiment, a fan was constantly in operationeven when no scents were being emitted. Thus, the risk of achange in perception due to the presence or absence of wind
was eliminated.

2.6

Figure 3: The experimental condition

LavenderBananaReferencevalue 2
21
29

Qualification of Subjects
interval of 1.3 s was repeated for 70 s.
After the stimuli presentation, subjects evaluated theirperception of the continuity of scent ejection at the fourstages outlined in Table 4 and their perception of theintensity change described in Table 5. We determined apresentation condition that was suitable for this experiment.
The selection of the condition was based on obtaining an
average value of two or more, regarding the continuity, andat a level where no subjects selected zero regarding thestrength.
Measurements were made with a 1-min break every 70 s
and the presentation was repeated four times for each of thescents. Taking the order effect into consideration, we
determined the first and second scents randomly andadjusted them, so each order was presented approximately
an equal number of times. Between the presentations of eachscent, there was a 5-min break. Due to the nature of theexperiment, it was assumed that the subjects could easily
feel the scents; therefore, prior to the first measurement, we
confirmed verbally whether each subject could feel theejection of scents twice. Next, we conducted the experimentwith five subjects (between 20 and 25 years of age, male)using the reference value 1, and with five subjects (between
20 and 30 years of age, male) using the reference value 2.

Prior to performing the olfactory experiment, we
conducted a test to confirm that the subjects had a generalolfactory sense. In this test, we used odor panel analyses toqualify five standard odor solutions and tested whether eachodor could be identified. The standard odor for panel
selection was prepared on the basis of the T & T
Olfactometer, which is used in national examinations todetermine olfactory measurement operator [5]. Using odor
solutions at the concentration determined by the Ministry of
the Environment, the test was conducted according to the 52 method. Only subjects who passed the test were taken as
subjects.
It is suggested that approximately one in 20 people do notexperience vection (vection blind). Therefore, subjects werescreened before the experiment was conducted. We asked
subjects to observe expansional and contractional moving
images. Subjects were then asked to evaluate the vection
they felt. In cases where subjects answered that they did not
feel vection (intensity 0) in more than half of these tests,
they were excluded from the experiment.

3 EXPERIMENT
1:PRESENTATIONMETHOD OF OLFACTORY STIMULI

3.2

It is necessary to examine the ejection method so that the
subjects can continue to feel the scent without adaptation for
70 s, including 30 s before the presentation of the vection
stimuli. Therefore, a preliminary experiment was conductedwith reference to a previous study [6].

3.1

Table 3: Ejection levels
Detection
Reference
thresholdvalue 1
2.67
11
3.67
15

Experimental Results

Table 6 shows the average evaluation values, standard
deviation, and minimum values in the case of reference
values 1 and 2 for each scent. As the average value of eachcell was 2 or more, it can be stated that the subjects wereable to smell the scents without adaptation for 70 s with anejection duration of 0.3 s and an ejection interval of 1.3 s.
However, with reference value 1, the minimum value of thecontinuity value of lavender and banana was 0, whereas inreference value 2, the minimum value of the continuity andintensity of lavender was 0. The results of this experiment
revealed that some of the subjects could not sense the scentfor 70 s.
When we interviewed the subjects who rated thecontinuity as 0, they made the following comments: “theExperimental method

Two kinds of scents were used in this experiment:
lavender and banana. Four times the average value obtainedin the above detection threshold measurement experiment
was set as a reference value 1, and eight times as the averagevalue was set as a reference value 2.
Table 3 shows the respective ejection levels, and as shownin Figure 4, scent ejection with a duration of 0.3 s and an

79

International Workshop on Informatics ( IWIN 2018 )Table 8 shows the average and standard deviation (SD) of
the evaluation values. Since the values are between 2 and 3,
except for the banana reference value 1, we set the reference
value 2 as the ejection level in this experiment so that the
subject would feel it in the experiment.

Ejection levels

1.3 sec
0.3 sec

0 0.3 1.6

3.2

4.8

6.4

8.0

Figure 4: Scent ejection pattern

0
1
2
3

0
1
2
3
4

AverageSD

9.6Time (sec)

4 EXPERIMENT 2: EXAMINATION OF
THE EFFECTS OF SCENT ON VECTION

Table 4: Four-stage evaluation for continuity
I did not feel the scent midway through the exposure
periodI felt it fragmentally
I felt it every breath
I felt it continuously

In experiment 1, we confirmed the olfactory stimuli
presentation method, which enabled the subjects to continue
to perceive scent without adaption for 70 s. Using thismethod, we investigated the influence of scent on vection
perception in experiment 2. The subjects wore an HMD andwere presented with a moving image that induced vection
under scent or sound presentation or movie only conditionand evaluated its strength.

Table 5: Five-stage evaluation for the strength
I did not feel the scent midway through the exposure
periodI felt that the concentration gradually decreased
I felt that the concentration got increased or decreased
I felt that the concentration gradually increased
I felt no change in the concentration and felt the scentuniformly

4.1

Continuity
Strength

2.10

2.00

2.30

2.75

SD.

0.76

0.52

1.31

0.46

Min.

0

0

0

1

Ave.

2.55

2.06

2.20

2.80

SD.

1.16

1.31

1.85

1.04

Min.

1

0

0

1

ejection amount in reference value 2 was so large that mynose became paralyzed.” Thus, we asked three subjects(between 20 and 25 years of age, male) to rate the scentintensity with both reference values, using the six-level odor
intensity indication method [7] (see Table 7) consisting ofodorless (0) to intense odor (5). This technique was devisedin the field of odor control in Japan.
0
1
2
3
4
5

Experimental Method

The vection stimuli presentation time was set to 40 s. The
scent and sound stimuli started 30 s before the presentationof the vection stimuli and continued to be presented for 70 s.
The flow of one trial is shown in Figure 5. During the period
in which moving stimuli were presented, the subjects wereasked to report the duration of time for which they wereexperiencing vection (Report 1). This was reported bypressing the left button of the mouse. At this point, the timerequired for the first button press was recorded as thelatency of vection. The total time during which the button
was pressed in the 40 s period was recorded as the durationof vection. After the stimuli presentation was completed, thesubjects were asked to report the strength of the vection by
rating 0 when they did not feel vection and 100 when theyfelt vection very strongly (Report 2). These three variables
have been repeatedly used in previous vection experiments(e.g., Seno et al., 2013) [8]. Furthermore, in cases where an
olfactory stimulus was presented, subjects were also asked
to report the subjective intensity value of the scent byselecting from the six-level odor intensity indicator that isdisplayed in Table 8 after the stimuli presentation wascompleted (Report 2). Under each condition, the flow of one
trial, as shown in Figure 5, was repeated four times with a 1min break between each trial. The experiment consisted ofeight conditions, and each condition was carried out as one
block with four consecutive trials.
The order of eight block experiments was randomized foreach subject, with a 5-min break between the blocks.

Table 6: Evaluation values for the continuity and thestrength of the scent stimuli
Reference value 1
Reference value 2
Lavender Banana Lavender Banana
Ave.

Table 8: Scent intensity
Reference value 1
Reference value 2
LavenderBananaLavenderBanana
2.42
1.92
2..83
2.58
0.79
1.44
0.93
1.08

Table 7: Six-level odor intensity indication
Odorless
A faint smell that you can hardly perceive
Weak smell you can recognize
Easily perceptible smell
Strong smell
Intense smell

4.2

Experimental Results

The results under expansion and contraction stimuli
conditions are shown in Figure 6 and 7, respectively. The
80

International Workshop on Informatics ( IWIN 2018 )

None

5

Sound

We examined how olfactory stimuli impact the perceptionof vection by presenting subjects with visual stimuli while
also presenting scents using an olfactory display. No
significant effects of scent presentation were observed in thethree variables that represented vection intensity. However,there was a difference in the trends between two vection
stimuli. In the expansion stimulus, although significant
difference was observed at latency (P < 0.01), it was notpossible to specify which condition was different. The
condition that perceived the strongest vection was whensound stimulus was presented. It was lavender at the next
point. In the contraction stimulus, there was a tendency
towards a difference in the latency and magnitude (both P <0.10). In each variable the condition that perceived thestrongest vection was when lavender was presented. In themagnitude, the next point was a sound stimulus, but at thelatency, the sound stimulus was the lowest among the four
conditions. From the above, it turned out that there is atendency that there is a difference in the stimulus affected
depending on the vection direction. Furthermore, it appears
that lavender may promote the perception of vection. There
was no effect of scent on vection perception possibly
because olfactory stimuli were not sensed as consciously as
vision and auditory stimuli. We believe that the visual
stimuli were too strong for the scents to be consciously
sensed. Alternatively, it was also found that vection stimuli
reduced the scent intensity of lavender. There was adifference in the trends between two vection stimuli
possibly because the difference in the direction of theobtained self-motion sensation. The expansion stimulus
induces a feeling of forward movement which is frequently
experienced in everyday life, but the movement in the
backward direction obtained from the contraction stimulus
less experiences. Therefore, it is thought that under thecontraction stimulus, it became more cautious and it becamea different perception method than under the expansion
stimulus. It is necessary to conduct a more detailed
investigation on the possibility that lavender may promote
the perception of vection.

LavenderBanana×
Expansion
Contraction
030Time (sec)70

Report 1

Report 2

Figure 5: Flow of presenting stimuli
subjective vection strength in each condition was compared.
It is known that when vection is strong, the latency is short,the vection duration is long, and the magnitude is high.
In the expansion stimulus condition, the order of theduration of latency was as follows, from shortest to longest:sound, lavender, none, and banana. The order of the durationof vection was as follows, from longest to shortest: sound,none, lavender, and banana. The order of the magnitude ofvection, from the highest to the lowest, was as follows:lavender, sound, banana, and none. Therefore, it can beinferred that the sound condition had the greatest effect on
the time of vection perceived by subjects, and the lavendercondition had the greatest effect on the magnitude of vection.
A nonparametric test, using the Friedman method, showed asignificant difference only in the latency (P < 0.01).
Therefore, when multiple comparisons were performed
using the Bonferroni correction, the results showed nosignificant statistical difference (P > 0.05).
In the contraction stimulus condition, the order of theduration of latency was as follows, from shortest to longest:lavender, none, banana, and sound. The order of the durationof vection was as follows, from longest to shortest: lavender,
none, banana, and sound. The order of the magnitude ofvection, from the highest to the lowest, was as follows:lavender, sound, banana, and none. Therefore, it can beinferred that vection was perceived most strongly by
subjects in the lavender condition. In addition, the soundstimulation that tended to promote vection perception in the
expansion stimulus was the weakest vection promotor in thecontraction stimulus condition. Similar to the case of the
expansion stimulus, we examined a nonparametric test using
the Friedman method, and the results showed that there wasno significant difference in the responses to any of the threevariables (P > 0.05). However, when the significance level
was set to 10%, there was a tendency towards a difference inlatency and magnitude (both P < 0.10). Therefore, while
there was no statistical difference, the results indicate thatthe lavender stimulus tended to promote the perception ofvection.
Thus, we could say that there was an interaction between
the direction of vection and the types of scent.

6

CONSIDERATIONCONCLUSION AND FUTURE WORK

We investigate the effect of scent on vection perception bypresenting subjects with visual stimuli while also presenting
them with scents using an olfactory display. The findings
were as follows:(1) Using a scent ejection time of 0.3 s and an ejection
interval of 1.3 s, it is possible to present a subject
with the scent at a level of about eight times thedetection threshold for 70 s without adaptation.
(2) There was no significant difference between theperceived vection in the presence or absence of scentpresentation in the three variables representing
vection intensity. However, in the contraction
stimulus, there was a tendency towards a difference.
It seems that there is a difference in the stimulus
affected depending on the vection direction.

81

International Workshop on Informatics ( IWIN 2018 )Figure 6: Expansion stimulus results

Figure 7: Contraction stimulus results[4] Y. Sakamoto, S. Ebihara, T. Ebihara, N. Tomita, K.
Toba, S. Freeman, H. Arai, & M. Khzuki, Fall
prevention using olfactory stimulation with lavender
odor in elderly nursing home residents: A
Randomized Controlled Trial, J. Am. Geriatr. Soc.,
Vol.60, pp1005-11 (2012).
[5] JAOE: Japan Association on Odor Environment,
http://orea.or.jp/about/kyukakukensa.html, (2018).
[6] K. Ohtsu, A. Kadowaki, J. Sato, Y. Bannai, & K.
Okada, Scent Presentation Method of Pulse Ejection
Synchronizedwith theUser’sBreathing,
Information Processing Society of Japan Research
Report Groupware and Network Services, 2008
(7(2008-GN-066)), pp.77-84 (2008) (in Japanese).
[7] S. Saito, J. Inouchi, S. Ayabe, F. Yoshii, & S.
Nakano, Introduction to Olfactory-Foundations of
Evaluation of Odorous Environment, Japan
Association on Odor Environment, (2014) (inJapanese).
[8] T., Seno, K. Abe, & S. Kiyokawa, Wearing heavy
iron clogs can inhibit vection, Multisensory
Research, Vol.26, pp. 569-580 (2013).

(3) Many indicators resulted in the strongest perceptionof the vection when presenting lavender. Lavenderscent may promote vection perception.
In the future, the effect of the scent stimuli on vection
perception should be further examined. We plan to conduct
these experiments using weaker vection stimuli or stronger
scent stimuli than was used in the current study.
Furthermore, we will investigate the effects of vection
stimulation on scent perception, and we would like to clarify
the causal relation between the vection and scent perception.
We are also interested in exploring the transition of
consciousness between visual, auditory, and olfactory senses
and the mutual influences between these senses.

REFERENCES[1] S. Tachi, M. Sto, M. Hirose, Virtual Reality Studies,
The Virtual Reality Society of Japan, (2011) (inJapanese)[2] S. Sakamoto, Y. Osada, Y. Suzuki, & J. Gyoba, The
effects of linearly moving sound images on selfmotion perception, Acoust. Sci. & tech, Vol.25,
No.1, pp.100-102 (2004).
[3] K. Murata, T. Seno, Y. Ozawa, & S. Ichihara. SelfMotion Perception Induced by Cutaneous Sensation
Caused by Constant Wind, Psychology, Vol.5,
pp.1777-1782 (2014).

82

International Workshop on Informatics ( IWIN 2018 )A Distributed Multi-Viewpoint Internet Live Broadcasting System with Video Effects
Satoru Matsumoto* Tomoki Yoshihisa* Tomoya Kawakami** Yuuichi Teranishi ******Cybermediacenter, Osaka University, Japan
Graduate School of Information Science, Nara Institute of Science and Technology, Japan***
National Institute of Information and Communications Technology, Japan{smatsumoto, yoshihisa}@cmc.osaka-u.ac.jp

Abstract - Due to the recent popularization ofomnidirectional cameras, multi-viewpoint live videos areoften broadcast through the Internet. In multi-viewpoint
Internet live broadcasting services, viewers can arbitrarilychange their viewpoints. To reduce computational loads forvideo processing, many researchers have been developed
some distributed Internet live broadcasting systems. Thesesystems are designed for single-viewpoint live videos andscreen images (images to be watched by viewers) are thesame for all viewers. However, in the multi-viewpoint
Internet live broadcasting services, screen images differ
among viewers since they can change their viewpoints. Here,one of the main research challenges for multi-viewpoint
Internet live broadcasting is how to reduce computationalloads for adding effects under different screen images.
In this paper, focusing on this challenge, we propose anddevelop a distributed multi-viewpoint Internet live
broadcasting system. It is difficult to determine which is
better to add effects on the server side or player side. To
determine this to reduce computational loads effectively, we
use hierarchical rules.

broadcasting systems is how to reduce computational loads
for adding effects under different screen images.
In this paper, focusing on this challenge, we propose anddevelop a distributed multi-viewpoint Internet live
broadcasting system. In our proposed system, video effects
that can be shared among viewers are added by some
distributed processing servers (on the server side). Videoeffects that cannot be shared among viewers are added byvideo players (on the player side). In such systems, it isdifficult to determine which is better to add effects on theserver side or player side. To determine this to effectively
reduce computational loads, we use hierarchical rules. Also,
we develop a distributed multi-viewpoint Internet live
broadcasting system extending our previously developed
system.
The paper is organized as follows. In Section 2, we
introduce related work. We design and explain our proposedsystem in Section 3. We show evaluation results in Section 4
and discuss the results in Section 5. Finally, we conclude thepaper in Section 6．

2
Keywords: Streaming Delivery; Internet Live Broadcasting;Multi-viewpoint Camera.

1

RELATED WORK

Some systems for distributing video processing loads havebeen proposed. Most of them fix load distribution
procedures in advance. However, starting Internet live
broadcasting is easy in recent years, and it is difficult tograsp which machines start Internet live broadcastings.
Therefore, conventional systems establish load distributionsat server sides.
MediaPaaS encodes, re-encodes, and deliver videos using
a server machine provided by cloud computing services [2].
Different from MediaPaas, our proposed system establishes
load distributions using PIAX [9], a P2P agent platform. In[1], we have confirmed that video processing time such as
encoding, distributing videos can be reduced by distributing
the processing load to many different world broadcasting
servers.
An Internet live broadcasting system that allows viewing
recently recorded videos (playback) was developed in [3].
Several methods have been proposed for reducing the delay
time to distribute videos of live Internet broadcasting. In live
broadcasting using SmoothCache 2.0 [4], by caching video
data from other peers and distributing them among cachedpeers using a P2P network, the communication load anddelay times are reduced. Dai, et. al. proposed a distributed
video broadcasting system using P2P networks to reduce
delay times in [5]. In the HD method proposed in [6],communication traffic is reduced by simultaneouslytransmitting image data to many viewers by using one-to-

INTRODUCTION

Due to the recent popularization of omnidirectional
cameras, multi-viewpoint live videos are often broadcast
through the Internet. In multi-viewpoint Internet live
broadcasting services, viewers can arbitrarily change their
viewpoints. For example, major live broadcasting services
such as YouTube Live or Facebook provide 360 degrees
videos, in which we can change our viewpoints. Also inrecent Internet live broadcasting services, viewers or
broadcasters often add the video or audio effects tobroadcast videos. To reduce computational loads for adding
effects, many researchers have been developed some
distributed Internet live broadcasting systems. ([1] [2]).
These systems are designed for single-viewpoint live
videos and screen images (images to be watched by viewers)
are the same for all viewers. Therefore, screen images canbe shared among processing servers and computationalloads can be reduced by exploiting distributed computing
systems. However, in the multi-viewpoint Internet live
broadcasting services, screen images differ among viewers
since they can change their viewpoints. Thus, screen images
cannot be shared among processing servers. Here, one of themain research challenges for multi-viewpoint Internet live

83

International Workshop on Informatics ( IWIN 2018 )In this section, first, we explain our previously developed
cloud-based live broadcasting system using ECA rules.
After that, we explain the multi-viewpoint Internet live
broadcasting system proposed in this paper.

3.1

Different World Broadcasting System

3.1.1. Summary of Different World Broadcasting
SystemIn our previous research [1], we constructed a differentworld broadcasting system using virtual machines provided
by cloud service. These machines work as the different
world broadcasting servers that add video effects. In general,many virtual machines can be easily used in a cloud service.
Therefore, the use of multiple virtual machines as differentworld broadcasting servers should enable high-speed effectaddition, while distributing the load among different-world
broadcasting servers. Therefore, we implemented a
distributed live Internet broadcasting system using the cloudservice and evaluated its performance. In our developed
system, video effect addition is executed on the virtual
machines provided by the cloud service shown in Figure 1.
The clients select a server considering the load distribution.
In conventional systems, load distribution is established byconnecting processing servers via a load balancing
mechanism such as a load balancer. In this method, when
the load distribution mechanism needs to switch to anotherserver while the video is being transmitted, the connection isinterrupted. For this reason, it is difficult to smoothly switch
servers while continuing the video distribution. Therefore, in
our system, the load balancing mechanism selects a different
world broadcasting server based on the requests.

Figure 1: Our designed distributed multi-viewpoint
Internet live broadcasting system.

Figure 2: The system architecture of the
Different World Broadcasting System

3.1.2. System Architecture
The system architecture of the different world
broadcasting system is shown in Figure 2. There are three
types of machines. One is the clients which have cameras
and record live videos. Another one is the different world
broadcasting servers which execute processes for videos
such as encoding, decoding, or video effect additions. The
final one is the viewers which play live videos. Each client
selects a different world broadcasting server that executes
the desired video effect and transmits the video effect libraryand the recorded video to the different world broadcasting
server. The different world broadcasting server is a virtual
machine of the cloud service executes video processing on
the video transmitted from the clients according to theirrequests. The video processed by the different world
broadcasting server is delivered to the viewers via the video
distribution service. In the system, the viewers receive theprocessed video after selecting the server or the channel ofthe video distribution service.

Figure 3: A load distribution mechanism by PIAX
many broadcasting with one-to-one communication. Even in
our proposed system, these delay reduction methods can beapplied when delivering videos, but our research considers
video or audio effect additions.
Gibbon, et. al. proposed a system that performs video
processing by transferring video data captured by a camera
to a computer with high processing capability in [7]. Ting, et.
al. proposed a system that directly stores images captured bycomputers with low processing power in external storage
devices such as cloud storage in [8]. However, these systems
target stored video data and can not be applied for live
Internet broadcasting.

3.1.3.

3 DISTRIBUTED INTERNET LIVEBROADCASTING SYSTEM

Load Distribution Mechanism

Figure 3 shows the load distribution mechanism of ourdeveloped system. Clients software and the client side PIAX

84

International Workshop on Informatics ( IWIN 2018 )Figure 4: Server software and Client softwareｙFigure 6: Examples of ECA rules

Ｐ'
Ｐ（ｘ,ｙ,ｚ）

Q
Frame Get

Initializationw

ConvertPlane

Fisheye
Image

Image

Recognize

Ｏ
Ｒ

（ｘ,ｙ）

x
Not Humanｚ

Plane
convert

Human

Known

Not Known

Rectangle

Blur

Not Human

Human

Drawing
Person A

Other person

Red
Rectangle

Figure 7: Examples of hierarchical ECA rules(1)

viewpoint video. These panoramic images were obtained
from cameras using fisheye lenses. It is necessary to convert
these images into a planar image. There are many methods
that obtain wide images from car-mounted fisheye lenses
and correct the distortion [11]. Figure 5 shows how to obtaina wide image from a panoramic image in our proposedsystem. As shown in this figure, the wide images are
obtained by assuming an imaginary hemispherical border forthe panoramic images. The converted wide image is shownin the upper left part of Figure 4. The conversion transforms
virtual hemispherical polar coordinates into rectangularcoordinates using equation (1). In our proposed system, thedistributed processing of polar/rectangular coordinates.

Figure 5: An Image of coordinate conversion
system are installed in the client. The different world
broadcasting server software and the server side PIAX
system are installed on the different world broadcasting
servers. PIAX [9] is a Java-based platform middleware thatenables efficient server resource search using resource
search function of the overlay network. PIAX is provided as
open source software. The PIAX system of the clients andthe different world broadcasting servers connect with eachother via the overlay network.

3.2 Extension of Different World Broadcasting
System

3.2.2. Design of ECA Rules for Multi-Viewpoint
VideosIn this section, first, we explain our proposed design of
ECA rules for multi-viewpoint videos. After that, we
explain the examples of hierarchical ECA rules.

Video effects have procedures, and for example, the facedetection process is generally performed before the mosaic
effect of the detected face. The "Timer" or "Message"function of ECA rules in the proposed system can definesuch a procedure. If the procedure is defined in the ECA
rules and the ECA rules are order dependency, the systemneeds to execute the rules according to the sequence.
Otherwise, in cases that the ECA rules do not depend on theprocessing request, the system can execute the rules
concurrently, and the processing time can be shortenedcompared with the ECA rules having the order dependent
relation. In the current system, it is impossible to process

3.2.1. Image Conversion of Multi-Viewpoint
VideosWith omnidirectional cameras, it is not necessary to takedozens of omnidirectional images from a certain viewpoint
and synthesize them on a computer to create a panoramic
image. Instead, we can create multi-viewpoint videos frompanoramic images. The lower left part of Figure 4 shows
two panoramic images (front and back) for a multi85

International Workshop on Informatics ( IWIN 2018 )ECA rules in parallel. Parallel processing for cloudcomputing services is a future task. Lists of events,
conditions, and actions are described in the previous
research [1]. Figure 6 shows an example of ECA rules. In
this example, the servers with IP addresses of 192.168.0.5
and 6 are assigned as initial values to video processing
requests from the video recording terminals named"Num_Find_Object" and "Spherical_coordinates_Convert".
In cases that processes of ECA rule have sequences, thesystem should execute the processes in the order of thesequence. Otherwise, the system can execute them inparallel. For example, Figure 7 shows an example ofhierarchical ECA rules.
• 1. Is it a fisheye lens image? <- Perform full sphericalcoordinate transformation <- Human detection.

realized by different world broadcasting. In the evaluation,
we measured the turnaround time from the time to generate
original image data to the time to obtain processed image
data. We will confirm the reduction of the turnaround times.
In order to confirm the efficiency of the proposed system,
the video processing time including the processing time of
the ECA rule and the turnaround time was measured as
evaluation indexes.
We experimented with the evaluation considering thefollowing two points.
(1) The turnaround time was compared and evaluatedwhen the video effect processing requests ware concentrated
on a video effect processing server without using the ECA
rule.
(2) The image effect processing requests do notconcentrate, and the ECA rule was adopted. For evaluation,
we assigned multiple computers with the same performanceparameters.
To select an available different world broadcasting servers,
we used the PIAX overlay network. When a different world
broadcasting server becomes overloaded, the server sends a
notification to the PIAX process on the server side and waits
for decreasing the loads. The turnaround time of theevaluation was measured in two cases. One is a concentratedcase where three clients request video processing to one ofthree different world broadcasting servers. The other case is
a completely distributed case where each of the three client
requests services to three different servers.
We perform live broadcasting in this evaluation. For theevaluation, as the video effect described in the ECA rule, aprocess in which face detection video processing has been
applied after the above coordinate conversion is executed.
The time required for sending and receiving frame data wasdefined as turnaround time．

• 2. Are Humans in image <- Who? <- Match with a
specific person <- Blur is applied.
• 3. Are Humans in image <- a Known person
registered in DB? <- If you are an unregistered person,
blur.
ECA rules are classified into hierarchies of detection,
conversion, inquiry, and pixel processing.

3.2.3. Implementation of Proposed System
We developed a distributed live Internet broadcasting
system using Microsoft Azure as a cloud service. The
different world broadcasting servers run on the virtual
machine provided by the Azure service. Each virtual
machines is logically connected through a virtual network(VNet) which is one of the services provided by Microsoft
Azure. Figure 4 shows a screenshot of server software andthe client software. When starting the video effect adding
process, it provides an interface of the different world
broadcasting server software. Figure 4 shows a screenshot ofclient software for distributing video to a client. We canvisually check the result of applying the selected.
In the client software holds the IP address of a different
world broadcasting server to request video processing. If the"Apply distributed processing" checkbox in the client
software dialog box is checked, the client software requests
the different world broadcasting server to execute the video
processing specified by the pull-down menu the initial IP
address.

4

4.2 Evaluation Environment
In this evaluation, a different world broadcasting server
runs on a virtual machine provided by the Microsoft Azure
service. Table 1 shows the specifications of the virtual
machine and OS. We used five different virtual machines
for different world broadcasting servers. Open CV
parallelized by Intel 's Parallel Computing Library TBB [10]was used as a library for executing video processing on adifferent world broadcasting server. The clients are a PC
installed at Osaka University. Table 2 shows the
specifications of the client PC. We attached a full
omnidirectional camera to only one. These PCs
communicate with different world broadcasting servers via
different home optical networks and avoid the congestion of
the network.

EXPERIMENTAL EVALUATION

We evaluated the performances of our implemented
system built on the virtual machine provided by theMicrosoft Azure service. The results follow.

Table 1: Specifications of Microsoft Azure Virtual Machines
OS
Microsoft Windows Server 2016
Microsoft
Azure Standalone Server Microsoft Corporation
Plan
Virtual Machine x64-based PC
CPUIntel E5-2697 v3 Equivalent 2.4GHz
Main memory
3.584MB

4.1 Evaluation System
In the experiment, Theta S made by RICHO Co., Ltd. wasused as an omnidirectional camera. Each video frame is
encoded in jpeg format, transmitted and received, as USB
virtual camera. Image conversions and rule processing are86International Workshop on Informatics ( IWIN 2018 )1 Server

Table 2: Specifications of Client PC
OS
Client PC
Microsoft Windows 10
Pro Version 1709,1511
1～３
CPU
Client PC1Intel
i7-7660U
Equivalent 2.5GHz
Client PC2Intel
i5-6300U
Equivalent 2.4GHz
Client PC3Intel
i3-4020Y
Equivalent 1.5GHz
Main memory
Client PC1
8.00 MB
Client PC2
8.00 MB
Client PC3
4.00 MB

Turn around Time(msec)

8000
7000
6000

5000
4000
3000
2000

1000
0
0

100

200

300

400

500

Frame count
PC1

PC2

PC3

Figure 8: Turnaround times under one cloud server
3 Server4.3 Evaluation Results
Turn around time (msec)

8000

Processes were assigned based on ECA rules among
different world broadcasting servers.
Figure [8, 9] show the evaluation results of the turnaroundtime under the evaluation environment described in Section
4.2. The horizontal axis shows the recorded frame thenumber and the vertical axis shows the turnaround time. In
Figure 8 where the load is concentrated on a single different
world broadcasting server, turnaround times increase
gradually. In Figure 9, the image processing requests are
distributed to three different world broadcasting servers. In
this case, the turnaround time is under 1500 msec, and theprocessing delay is around 7500 msec during processing.
The different world broadcasting server that PC 2 requests
image effect processing is a VM in the East Japan region in
the real environment of Microsoft Azure. Therefore, there
were variations in the route and the turnaround time changes.
We also measured the turnaround time required to contact
the recommended different world broadcasting server by
PIAX. The average time to process a query for getting
recommended different world broadcasting server be 16.28[msec].
As a result, we confirmed that the processing request isallocated to the different world broadcasting server based on
the ECA rule, and the load is distributed. Moreover, we
confirmed that the turnaround time might fluctuate even if
the hardware performance of the virtual machine isequivalent due to the effect of communication delay, etc.

5

7000

6000
5000
4000
3000
2000

1000
0
0

100

200

300

400

500

600

Frame count
PC1

PC2

PC3

Figure 9: Turnaround times under three cloud servers

Figure 10: An image of the hierarchical rules

DISCUSSION

In past paper [1], the video processing was detecting theface of people in the video with the specified effectdescribed in the ECA rule. Turnaround time fluctuation was
confirmed among cloud computing service virtual machines.
This is caused by factors such as actual server performancefluctuations due to differences in the cloud environment of
the network distance. Therefore, when the user configures
the system, it is recommended to consider these problems.
In this paper, we have proposed hierarchical rules of threestages. The stages follow: Prepare rules for selecting thelocation of three types of processing, processing by a local
client, processing by edge computing, processing by cloudcomputing. An image of the hierarchical rules is shown in
Figure [10, 11] shows an example of rules proposed in thisFigure 11: An example of our proposed rules

87International Workshop on Informatics ( IWIN 2018 )live streaming,” Pro-ceedings of the IEEE International
Conference on Commu-nications (ICC 2015), pp.
6959-6964 (2015).
[6] T. Yoshihisa and S. Nishio, “A division-based
broadcasting method considering channel bandwidths
for NVoD services,” IEEE Transactions on
Broadcasting, vol. 59, no. 1, pp. 62-71 (2013).
[7] D. Gibbon and L. Begaja, “Distributed processing for
big data video analytics,” IEEE ComSoc MMTC ELetter, vol. 9, no. 3, pp. 29-31 (2014).
[8] W.-C. Ting, K.-H. Lu, C.-W. Lo, S.-H. Chang, and P.C.
Liu, “Smart video hosting and processing platform forInternet-of-Things,” Proceedings of the 2014 IEEE
International Conference on Internet of Things(iThings 2014), pp. 169-176 (2014).
[9] M. Yoshida，T. Okuda，Y. Teranishi，K. Harumoto，
S. Shimojyo, “PIAX: A P2P Platform for IntegrationofMulti-overlay
andDistributed
Agent
Mechanisms, ”
Transactions of Information
Processing Society of Japan / Information Processing
Society of Japan， Vol. 49, No. 1, pp. 402-413, (2008).
[10] Thread
Building
Blocks,https://www.threadingbuildingblocks.org/,(referred October 1, 2017).
[11] J. Jeong, H. Kim, B. Kim, S. Cho, “Wide rear vehicle
recognition using a fisheye lens camera image” 2016
IEEE Asia Pacific Conference on Circuits and Systems(APCCAS), pp. 691-693 (2016)

paper. In this rule, the different world broadcasting serverthat adds the video effects changes when the performance ofthe current server decreases.

6

CONCLUSION

In this research, we implemented and evaluated a multiviewpoint distributed live Internet (different world)
broadcasting system that adds various image processing
using the computer provided by cloud service. By describing
processing assignment with ECA rule, we can flexibly
assign video processing to appropriate virtual machines. In
the developed system, the PIAX platform was used to searchand communicate among virtual machines. Even if thenumber of virtual machines changed, video processing could
be continuously allocated during live Internet broadcast.
System evaluation confirmed that the turnaround time ofvideo processing could be shortened by using the proposedsystem.
In the future, we plan to exploit edge computing
environments in which computers on the edge of the Internet
can execute video processes. The processing time can be
reduced since the turnaround times of edge computers areshort.

ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant
Numbers JP15H02702 and JP18K11316, and by Kayamori
Foundation of Informational Science Advancement.
Also this work includes the result of NICT Osaka University
joint research “research and development of advanced
network platform technology for large scale distributed
computing.”

REFERENCES[1] Satoru Matsumoto, Yoshimasa Ishi, Tomoki Yoshihisa,
Tomoya Kawakami, and Yuuichi Teranishi, “Different
Worlds Broadcasting: A Distributed Internet Live
Broadcasting System with Video and Audio Effects,”
in Proc. of IEEE International Conference on
Advanced Information Networking and Applications(AINA 2017), pp. 71-78 (2017).
[2] Satoru Matsumoto, Yoshimasa Ishi, Tomoki Yoshihisa,
Tomoya Kawakami, and Yuuichi Teranishi, “A Design
and Implementation of Distributed Internet Live
Broadcasting Systems Enhanced by Cloud Computing
Services,” in Proc. of the International Workshop on
Informatics (IWIN 2017), pp. 111-118 (2017).
[3] Y. Gotoh, T. Yoshihisa, H. Taniguchi, and M.
Kanazawa, “Brossom: a P2P streaming system forwebcast,” Journal of Networking Technology, Vol. 2,
No. 4, pp. 169-181 (2011).
[4] R. Roverso, R. Reale, S. El-Ansary, and S. Haridi,“Smooth-Cache 2.0: CDN-quality adaptive HTTP live
streaming on peer-to-peer overlays,” Proceedings of
the 6th ACM Multi-media Systems Conference(MMSys 2015), pp. 61-72 (2015).
[5] J. Dai, Z. Chang, and G.S.H. Chan, “Delay
optimization for multi-source multi-channel overlay
88Session 4:
Network and Security( Chair: Kozo Okano )International Workshop on Informatics ( IWIN 2018 )Detection of Malware-infected Hosts with Data Compression Algorithm
Yasushi Okano, Kazunori Kamiya, Atsutoshi Kumagai, Taishi Nishiyama,
Bo Hu, Masaki Tanikawa, Kazuhiko Ohkubo
NTT Secure Platform Laboratories, Japan{okano.yasushi, kamiya.kazunori, kumagai.atsutoshi, nishiyama.taishi,
hu.bo, tanikawa.masaki, ohkubo.kazuhiko}@lab.ntt.co.jp
Abstract

to change communication patterns easily, heuristic rulecreation adds cost to operations and has diﬃculty catchMachine learning is being actively used to detect malware- ing up with malware modiﬁcation. As a result, machine
infested hosts and their malicious communications. Whenlearning is gaining attention for automatically detecting
applying machine learning, designing the right feature isevolving malware and for helping operations.
the key for accurate detection. BoW (Bag of Words)In applying machine learning for network log analybased feature extraction is widely applied in natural lansis, ﬁrst, machine learning calculates feature values fromguage processing and also utilized for malicious communetwork logs. For instance, feature values range fromnication detection. However, BoW-based feature extracthe length of a string, frequency of terms in device logs,
tion does not always scale for handling network logs thatand so on. Second, machine learning will classify theoften have new data sequences. By focusing on the fact
data into legitimate or malicious on the basis of featurethat new data sequences in network logs are in many
values. In this process, infected hosts and malicious comcases mostly similar but partly diﬀerent, we propose amunications are detected.
new detection method based on a data compression alThere are many detection algorithms from LR (Logisgorithm. Since the compression algorithm has a chartic Regression), SVM (Support Vector Machine), Ranacteristic that data size after compressing is related to
domForest, and DNN (Deep Neural Network). However,similarity of data, a compression algorithm based feathe most critical factor for accurate detection is designture can be utilized for classiﬁcation. According to ouring the right feature for the problem.
evaluation results with real-ﬁeld proxy logs in an enterBoW (Bag of Words)-based features are widely apprise network, the proposed method has better at detecplied in natural language processing and also utilized totion than a BoW-based detection method. In particular,
detect malicious communications. However, BoW-based
its true positive rate (TPR) in a low false positive ratefeature extraction does not always scale for handling net(FPR) area (0.5%) is over 30% higher than that for thework logs, which often have new data sequences.
BoW-based method. In addition, the results show that
By focusing on the fact that new data sequences in
the proposed method eﬀectively detects an infected host
network logs are in many cases mostly similar but partly
communicating with malicious URL that includes pardiﬀerent, we propose a compression algorithm based featially modiﬁed string from original malicious logs.
ture and apply it to supervised learning for detecting
malicious communications and infected hosts.
Simply put, a compression algorithm based feature is
1 Introduction
one form of the compression rate of data, which means
Malware is becoming more sophisticated and has so
how small the data becomes after being compressed. Whenmany variants that anti-virus software does not detectthe data sequence is similar to that in existing malicious
all of them. In fact, it is reported that over 127.5 million
data, the compression rate should be small because thispieces of malware were registered in 2016 [1]. To comdata sequence is eﬀectively compressed. On the otherpliment the fact that detection at the endpoint is nothand, if another data sequence is totally diﬀerent fromalways successful, network log analysis is one solutionthat in existing malicious data, the compression ratethat monitors logs taken from network devices such as
should be large. In this sense, the compression rate canproxy and ﬁrewall and ﬁnds malicious communications
be useful for ﬁnding if one data sequence is similar toderived from infected hosts.
that in malicious data. Consequently, the compression
rate can contribute to detecting malicious communicaIn current log analysis, many monitoring rules havetions.
been deployed. Examples include network scans being
We evaluated the proposed method with real proxydetected once the number of diﬀerent destination IP adlogs taken from an enterprise network. The results show
dresses from one source IP address exceeds a predeﬁnedthat the proposed method is better at detection thanthreshold and a speciﬁc malware infection being deterthe BoW-based method. In particular, the results show
mined if one host accesses a blacklisted URL. Many monthat the proposed method eﬀectively detects an infected
itoring rules are based on operators’ elaborations on crehost communicating with a malicious URL that includes
ating rules, deciding thresholds, and maintaining blacka partially modiﬁed string from original malicious logs.
lists. However, as malware has evolved to become able
91

International Workshop on Informatics ( IWIN 2018 )Overall, our research makes three contributions.
1. We ﬁrst apply the compression algorithm feature
to supervised machine learning to detect malicious
communications and infected hosts.
2. We evaluate the proposed method with real enterprise proxy logs and demonstrate that the proposed
method performs better than a BoW-based classiﬁer.
3. We analyze true positive and false negative use
cases and clarify that the proposed method eﬀectively detects partially modiﬁed strings from original malicious logs.

2
2.1

Related WorkClassiﬁcation based on Compression
Algorithm

Benedetto et al. [2] proposed relative entropy. Although patterns of the same consecutive code or similar
repeated code are eﬀectively compressed, patterns of different code are not. Relative information volume of data
sequence x against data A is linked to how well data iscompressed. Based on this observation, Benedetto et al.
deﬁne relative entropy as how well new data x will becompressed with existing data A. Consequently, this isformulated as follows.
CA (x) = Z(A cat x) − Z(A)(1)

where Z is the function to output the data size after
compression, and cat is the function to concatenate theﬁrst and second data sequences. Sometimes, normalized
relative entropy is also used, which is deﬁned to divide
relative entropy by the size of data x.
Relative entropy has been applied to classiﬁcation problems in several research areas [3–7]. To classify data x
into group A and B, data x is normally classiﬁed into
the more similar group. Relative entropy can be used as
one index of expressing similarity; when relative entropy
with group X is small, data x is similar to group X.
Bratko et al. [5] applied relative entropy to classify
spam e-mails. They reported that it was more accurate
than BoW based classiﬁcation.
Nishida et al. [6] introduced a smoothing parameterand set the score in accordance with the following equation to classify malicious tweets from twitter logs.
Score =CA (x) + γ
CB (x) + γ(2)where γ is a smoothing parameter that should be set
large to alleviate the impact of minor letters appearing
a few times in a data string. Data x is classiﬁed as A
if the score is small and B otherwise. This scoring technique enables us to apply a compression algorithm for aclassiﬁcation problem of comparably long data. Nishida

et al. [6] also demonstrated that classiﬁcation of twitter logs with this scoring mechanism has better accuracy than feature extraction with morphological analysis
and classiﬁcation with a CW(Conﬁdence-Weighted linear classiﬁcation) method [8].
Diﬀerent compression algorithms are used dependingon their purposes. It is reported that LZSS (gzip), LZW(compress), PMP (rar) are applicable for text data [3–6].
Adachi et al. [7] reported that bzip is applicable for musicpieces.

2.2

Method of extracting feature from
URL string

The BoW method is widely used to extract featuresfrom strings. BoW decomposes string text into words by
separation of letters or morphological analysis and thengenerates each word as a one-dimensional feature. Since
a URL is deemed as a one text string, BoW features canbe extracted. Kumagai et al. [9] proposed BoW-based
feature generation to apply LR supervised learning with
L1 regularization and demonstrated that their method
has better area under curve (AUC ) than blacklist based
detection.
Nelms et al. [10] proposed describing a URL attributewith a regular expression and applying unsupervised learning to generate a malware-speciﬁc URL access template.
By comparing a target URL and the above template,the method successfully detects malware communicationeven when malware slightly modiﬁes its access pattern.
In the security context, on the basis of knowledge on
malware analysis, many kinds of statistical features havebeen proposed [11] such as the length of a URL and ratioof vowels in a URL.

3

Proposal

We propose applying a compression algorithm based
feature to apply supervised learning to detect malicious
URLs and infected hosts. Since a large part of malware
uses HTTP as a communication protocol with C2 servers,
it can be mixed with normal Web access and is hard for
operators to distinguish. Thus, in our research, we focus
on analyzing HTTP proxy logs and detecting malicious
URLs to ﬁnd infected hosts.
An important observation on malware communicationin HTTP is that they tend to access C2 servers witha slightly modiﬁed URL string in order to slip throughblacklist-based detection with minimum engineering effort. In this case, simple blacklist matching does notcatch up with malicious URLs since malware may havevarious URL access patterns even if its modiﬁcations are
small. In this sense, we expect the compression algorithm based feature to correctly describe the similarity
between a slightly modiﬁed malicious URL and a known
malicious URL.
To the best of our knowledge, our proposal is the ﬁrst
to apply a compression algorithm to detect malicious
communication URLs and infected hosts. In addition,

92

International Workshop on Informatics ( IWIN 2018 )Malicious URLs
Legitimate URLs
Training
Phase

Malicious URLs
Legitimate URLs

3. Feature
Generation

4. TrainingPrediction
Model

CompressModel

2. Compression

1. PreprocessingRaw Logs

Prediction
Phase

Test URLs 

Result5. Feature
Generation

6. Detection

Figure 1: Overview of proposed method
Table 1: Dataset
Number of hosts

Number of logs

Collection Period

Log Type

Malicious Logs

71,3107,152,479

Feb. 2015 - Jul. 2015

Sandbox logs

Legitimate Logs

1,940

36,581,398

Feb. 2014 - Mar. 2014

Proxy logs in enterprise

our method is diﬀerent from existing compression algorithm based methods in that we use a compression
algorithm-based score as a feature in supervised learningand the feature can be combined with other features.
Furthermore, our research considers a URL structurethat has many kinds of attributes such as FQDN, PATH,
and QueryString to generate a multi-vector compression
algorithm feature for each attribute.
Figure 1 shows an overview of the proposed method.
The ﬂow of our proposal is as follows.
1. Input raw logs and execute preprocessing to obtainmalicious URLs, legitimate URLs and test URLs
2. Compress malicious URLs and legitimate URLs togenerate compress model
3. Input malicious URLs and legitimate URLs withapplication of compress model to generate compression algorithm features, namely Zpos (Malicious
Compression Rate) and Zneg (Legitimate Compression Rate). Zpos and Zneg are deﬁned in equation(3) and (4) respectively. Features are calculated
for each attribute of a URL.
4. Train classiﬁer with compression algorithm featuresand generate prediction model

process. First, we delete duplicate URLs in legitimate
and malicious logs. This is because hosts may access thesame URLs repeatedly. To correctly estimate a classiﬁer,
we leave ﬁrst-to-appear logs in a dataset and eliminate
duplicate logs.
Second, we eliminate URLs included in both maliciousand legitimate logs, since having the same logs in bothdatasets may degrade the classiﬁer’s performance. In
fact, there are many cases in which the same URLs areincluded in both logs. For instance, some service URLs
are automatically accessed from speciﬁc applications installed in many environments. Search engine URLs arealso often accessed from infected hosts for connectivity
checks and included in malicious logs.
As for the compression algorithm features, we deﬁne
Zpos and Zneg as follows.
Zpos (x) =(3)

Cneg (x) + γ(4)L(x) + γ
where Cpos and Cneg are relative entropy between data x
and malicious log(pos) or legitimate log(neg), L is data
size of x, and γ is a smoothing parameter.
Zneg (x) =4

Evaluation Method

5. Generate compression algorithm features from test
URLs with application of compress model

4.1

6. Detect malicious URLs and infected hosts with application of prediction model1.

As with preprocessing, suitable data must be selected
in machine learning for correctly estimating a classiﬁer’s
performance. We execute two-phase cleansing in thisCpos (x) + γ
L(x) + γ

DatasetThe dataset used for all evaluations is shown in table
Malicious logs are taken from an in-house sandbox [12]where we run over 70K malware downloaded on a daily
basis from a malware-sharing site and collect pcaps toextract URL information. Legitimate logs are taken fromreal-environment proxy in an enterprise network.

93

International Workshop on Informatics ( IWIN 2018 )4.2

Evaluation Indices

Evaluations are executed on the basis of a holdout test
that uses previous data in time series as the training
dataset and evaluates with later data in time. Evaluation
indices are AUC , partial AUC (pAUC ) [13], and true
positive rate (TPR)0.5% [14].
AUC is the area under the curve drawn on a 2D surface
of a false positive rate (FPR) and TPR by changing thescore threshold. pAUC is the area under the curve of alimited range of a FPR [p1, p2]. Considering the TPR asa function having a FPR as a variable, AUC and pAUC
are deﬁned as follows.
AUC =pAUC = 1
0

TPR dFPR p2
p1

TPR dFPR(5)(6)

Through our evaluation, we set [p1, p2] = [0, 0.1].
TPR 0.5% is the TPR value for a low FPR, speciﬁcally
FPR = 0.5%. In security operations, a low FPR is crucial since the ﬁnal judgment is done by operators. pAUC
and TPR 0.5% are important indices to estimate detection
capability with a low FPR.

4.3

Table 2: Evaluation with conventional BoW-based classiﬁcation method

5.1

Evaluation Result
Comparative Evaluation

Table 2 shows the evaluation results for the proposed
and BoW-based classiﬁcation methods. From these results, the proposed method has better TPR 0.5% and pAUC
than the conventional BoW-based classiﬁcation method.
Figure 2 shows the TPR 0.5% deterioration over time
where the vertical axis is TPR 0.5% and the horizontal
axis is time in weeks. This ﬁgure shows that TPR 0.5%
gradually decreases over time. However, the proposed
method always achieves a higher TPR 0.5% than the BoW
method until 14 weeks have past.
Table 3 and 4 show comparison with BoW method on
CPU time and memory usage, respectively. Proposed
method consumes most of CPU time for compression
process and its time is longer than any other process ofpAUC

TPR0.5%

Proposed
BoW

0.93060.90300.08250.065765.30%
32.00%

Table 3: CPU Time Comparison with Conventional
Method (seconds)
MethodProposed
BoW

Compress Generate Train
Feature13,809
-

3,752
7,145

22
950

Detect438
408

Table 4: Memory Usage Comparison with Conventional
Method (MB)MethodProposed
BoW

6
5

AUC

BoW method. Still, once compression is completed, feature generation, training and detection are ﬁnished withless CPU time than BoW method. As for memory usage, proposed method consumes small memory for compression process and less memory for feature generation,
training and detection compared with BoW method. Although BoW method generates one-hot vector for every single word appeared in URL so that memory usage
tends to increase, proposed method generates compression algorithm feature vector in several dimensions sothat memory usage does not steeply increase.

Comparative Evaluation

We evaluate the proposed method in comparison withthe conventional BoW-based detection method. First,
we compared detection capabilities of the proposed andconventional methods. We also measured detection accuracy over time to ﬁnd out how fast trained models
deteriorate. Computational eﬀorts are another important factor for practical use, so we measure CPU timeand memory usage of the proposed and BoW method.
BoW of a URL is extracted by setting /, ?, =, & as aseparator and splitting the URL. In this evaluation we
set the classiﬁer as SVM, the compression algorithm as
LZT [15], and TPR/FPR as a per host-basis calculation.

MethodCompress Generate Train
Feature2,401.0-

15,999.8
56,988.2

4.7
1,241.2

Detect424.4
529.4

Consideration

We consider the reason the compression algorithm feature contributes to better classifying malicious and legitimate logs. Figure 3 shows the histogram of Zpos of URL
attributes for both malicious and legitimate logs, where
the red and blue zone are histograms of malicious andlegitimate logs, respectively. The histogram of malicious
logs contains three peaks: A) the compression rate isvery small, B) the compression rate is as high as thatfor legitimate logs, and C) the compression rate is veryhigh.
A sample URL that belongs to pattern A is shownin table 5. For security reason, FQDN is masked with’www.example.com’ and QueryString values are masked
with meta words. The ﬁrst row shows the original URL
string and its length, the second row shows the LZT compressed state and relative entropy with malicious logs,
and the third row shows that with legitimate logs, where

94International Workshop on Informatics ( IWIN 2018 )Table 5: Eﬀectively classiﬁed URL and compression state
Uncompressed URL: 1352bit:
http://www.example.com//oﬀers/DynamicOﬀerScreen?
oﬀerid=foo&distid=bar&leadp=baz&countryid=qux&
sysbit=quux&dfb=corge&hb=grault&isagg=garply&
version=waldo&external=fred&external=plugh&
Relative Entropy for Legitimate Logs: 900bit|http://www.example.com/|/of|fers|/D|yna|mic|Oﬀ|erS|
cre|en|?o|ﬀe|rid=foo|&dis|tid=b|ar|&le|adp|=ba|z&c|ou
ntry|id=qu|x&s|ys|bit|=quux&|dfb|=corge&|hb=|grault&is|agg|=garply&ver|sion=|waldo|&ex|ternal|=fred&e|x
ter|nal|=plugh&|Relative Entropy for Malicious Logs: 220bit|http://www.example.com//oﬀers/DynamicOﬀerScreen?oﬀerid=foo|&distid=b|ar|&leadp=baz&|countryid=qux&sysbit=quux&df|b=corge&hb=grault|&isagg=garply&
versio|n=wal|do&e|xternal=fred&e|xternal=plugh&|

Figure 2: TPR deterioration over time

Malicious Logs

Legitimate Logs

A

Table 6: Poorly classiﬁed URL and compression state
Uncompressed URL: 4688 bits
http://www.example.com/api/vp/1?clk=gLg PHWA9a
SyioXkt-F4b3J9cI1ybf-t-x7VxWH5dmAWXwln-z...(omit)

B

Relative Entropy for Legitimate Logs: 4420bit|http://www.example.com/api/|vp|/1|?cl|k=|gLg| PH|WA9|aS|yio|Xkt|-F4|b3J|9cI|1y|bf-|t-|x7|VxW|H5d|
mAW|Xwl|n-|zʜ(omit)C

Relative Entropy for Malicious Logs: 4980bit
A|http://www.example.com/api/v|p/1|?cl|k=|gL|g |PH|WA|9a|Sy|io|Xk|t-|F4|b3|J9|cI|1yb|f-|t-x|7V|xWH|5d|
mA|WX|wl|n-|zʜ(omit)
Figure 3: Histogram of Zpos .
’|’ shows that data sequences between ’|’ marks are expressed in 1 code. In compressing with malicious logs,
the table shows that a 1,352-bit-long URL is compressed
to 220 bits and many data sequences are expressed as
1 code. Especially in QueryString of URL, almost one
key (e.g. ”dstid=1”) or one combination of a key andvalue (e.g. ”countryid=...”) is compressed as 1 code.
This observation suggests that a QueryString key andvalue combination that exists in the training dataset is
automatically recognized and compressed as 1 code. In
contrast, a key and value combination that does not exist in the training dataset is automatically split. This is
one use case that QueryString key exists but its value is
modiﬁed in malware communication.
Other examples of pattern A for the FQDN attribute
are FQDNs having sequential numbers in host names
such as host1.example.com and host2.example.com. TheseFQDNs are recognized as totally diﬀerent strings by exact matching, but in the compression algorithm that has
the characteristic of longest matching, two FQDNs arerecognized as similar strings. In fact, host2.example.com
is compressed as |host|2.|example.com| after training datahost1.example.com. This is another use case that FQDN
is partially modiﬁed to similar FQDN.
URLs belongings to pattern B tend to have same FQDNs
existing both in malicious URLs and legitimate URLs.
Since Path of these URLs are diﬀerent, compression rate
does not get so small against both malicious logs andlegitimate logs and makes detection diﬃcult.
A typical URL belongings to pattern C is shown intable 6. This URL has an encoded or encrypted string.
The second row shows compression results of the URL. It
shows that the compression rate becomes large for both
malicious and legitimate logs and makes classiﬁcationdiﬃcult.

95

International Workshop on Informatics ( IWIN 2018 )7

Conclusion

We proposed a novel method for detecting malicious
communication of infected hosts by generating a compression algorithm feature of URL attributes and classifying with supervised learning. Through evaluation,
we demonstrated that the proposed method has higher
detection capability than the conventional BoW-based
detection method. In particular, its TPR in a low FPR
area (0.5%) is over 30% higher than that of the BoWbased method. In addition, we clariﬁed how the compression algorithm works in classiﬁcation and demonstrated
a real use case in which the proposed method detected
malicious URL strings that are similar to but slightly
diﬀerent from existing malicious URLs.

REFERENCES[1] https://www.av-test.org/fileadmin/pdf/
security_report/AV-TEST_Security_Report_
2016-2017.pdf, ”SECURITY REPORT 2016/17”,
AV-TEST, 2017.
[2] Dario Benedetto, Emanuele Caglioti, and Vittorio
Loreto, “Language trees and zipping”, Phys. Rev.
Lett., vol.88, 2002.
[3] Eamonn Keogh, Stefano Lonardi, and Chotirat
Ann Ratanamahatana, ”Towards Parameter-Free
DataMining”, KDD, pp. 206-215, 2004.
[4] Yuval Marton, Ning Wu, and Lisa Hellerstein, “Oncompression-based text classiﬁcation”, European
Conference on Information Retrieval, pp. 300-314,
2005.
[5] Andrej Bratko, Gordon V. Cormack, Bogdan Filipič, Thomas R. Lynam, and Blaž Zupan, “Spamﬁltering using statistical data compression”, Journalof Machine Learning Research, vol.7, pp.2673-2698,
2006.
[6] Kyosuke Nishida, Ryohei Banno, Ko Fujimura, andTakashide Hoshide, ”Tweet-Topic Classiﬁcation using Data Compression”, DBSJ Journal, Vol.10,
No.1, pp.1-6, 2011.
[7] Hanae Adachi, Masayuki Okabe, and Kyoji
Umemura, ”Recognition of Music Composerwith Compression-based Dissimilarity Measure”,
DEIM2013.
[8] Mark Dredze, Koby Crammer, and Fernando
Pereira,ʠConﬁdence-weighted linear classiﬁcationʡ,Proceedings of 25th International Conference on
Machine Learning, pp.264 r 271, 2008.
[9] Atsutoshi Kumagai, Yasushi Okano, Kazunori
Kamiya, and Masaki Tanikawa, ”Supervised Classiﬁcation for Detecting Malware Infected Host in
HTTP Traﬃc and Long-time Evaluation for Detection Performance using Mixed Data”, IEICE-ICSS,
Vol.116, No.522, pp.43-48, 2016.
[10] Terry Nelms, Roberto Perdisci, and Mustaque
Ahamad, ʠExecScent: mining for new C&C domains in live networks with adaptive control proto-

col templatesʡ, 22nd USENIX Conf., pp.589 r 604,
Aug. 2013.
[11] Karel Bartos, Michal Sofka, and Vojtech Franc,
ʠOptimized invariant representation of network
traﬃc for detecting unseen malware variants”, in
USENIX Security Symposium, pp. 807 r 822, 2016.
[12] Kazufumi Aoki, Takeshi Yagi, Makoto Iwamura,and Mitsutaka Itoh, ʠControlling malware httpcommunications in dynamic analysis system using
search engineʡ, Cyberspace Safety and Security(CSS), 2011 Third International Workshop onIEEE,
pp.1 r 6 2011.
[13] Yasushi Okano, Atsutoshi Kumagai, Masaki
Tanikawa, Yoshito Oshima, Kenji Aiko, Kazumitsu
Umehashi, and Junichi Murakami, ”Proposal of selection of training data using misdetected goodwarefor preventing misdetection of a static detector of
malware”, IEICE-PRMU, Vol.115, No.224, pp.163170, 2015.
[14] L. E. Dodd and M. S. Pepe. ʠPartial AUC estimation and regressionʡ, Biometrics, 59(3), pp.614 r
623, 2003.
[15] Peter Tischer, ʠA modiﬁed Lempel-Ziv-Welch datacompression schemeʡ, Aust. Comp. Sci. Commun.
9, 1, pp.262-272, 1987.

96

International Workshop on Informatics ( IWIN 2018 )Detection of Malware Infection using Traffic Models based on the Similarity
between Malware Samples
Masatsugu Ichino† , Yuuki Mori† , Mitsuhiro Hatada‡ , and Hiroshi Yoshiura††

Graduate School of Informatics and Engineering, The University of Electro-Communications, Japan‡NTT Communications, Japanichino@inf.uec.ac.jp, y-mori@uec.ac.jp, m.hatada@ntt.com, yoshiura@uec.ac.jp

Abstract - New types of malware are appearing every day,and malware attacks have become an urgent problem. Current methods of detecting malware use malware signatures,
which need to be identified and registered in advance. However, the daily appearance of new types of malware makes
such identification and registration impractical. A more practical approach is to identify malware on the basis of traffic
behavior since each malware type displays a unique behavior. We have developed a method for detecting malware infection using traffic models based on the similarity between
traffic of malware samples. Malware-infected traffic is divided into clusters on the basis of traffic behavior, and a model
representing each cluster is created. These models are used
to identify target traffic samples as infected or normal. Thismethod should enable the detection of infection caused by a
new type of malware if the malware’s traffic behavior is similar to that represented by one of the models. Simulation evaluation demonstrated that the proposed method can effectively
identify malware-infect traffic with high accuracy.

This paper is organized as follows. Section 2 introduces related work. Section 3 describes the proposed method. Section
4 describes the evaluation method. Section 5 presents and discusses the key results. Section 6 concludes the paper with abrief summary of the key points and a mention of future work.

2 Related work
There have been various studies of malware detection using
traffic data. Some used the definitions provided by security
vendors for detecting malware infection, and some did not.
Studies in the first group (e.g., [1][2][3]) classified malware
traffic samples into groups on the basis of the definitions andcreated models of infected traffic for each group. However,security vendor definitions are not always based on the characteristics of infected traffic. It is thus better to create models
on the basis of the characteristics of infected traffic.
Some studies in the second group ([4][5]) created models
of infected traffic on the basis of the characteristics of infectedtraffic but did not consider the time series of the traffic data
and the similarities between malware samples. Other studies([6][7]) created models of infected traffic on the basis of thecharacteristics of infected traffic considering similarities between malware samples but did not consider the time series of
the traffic data. Still other studies ([8][9]) created models of
infected traffic on the basis of the characteristics of infectedtraffic considering the time series of the traffic data but did
not consider the similarities between malware samples.
Traffic data is a stream of network information, and previous studies have demonstrated the effectiveness of considering the time series of the traffic data. Consideration of thesimilarities between malware samples is also necessary for
representing common characteristics of infected traffic.
Therefore, in our study, we create feature values by considering the time series of each malware traffic sample. Next, we
divided the malware samples into clusters on the basis of theirsimilarities. Then, we created models representing the common characteristics of the infected traffic for each cluster.

Keywords: security, malware, malware detection, traffic,
clustering

1 Introduction
New types of malware are appearing every day, and malware attacks have become an urgent problem. Current methods of detecting malware use malware signatures, which need
to be identified and registered in advance. However, the daily
appearance of new types of malware make such identification
and registration impractical. A more practical approach is to
identify malware on the basis of traffic behavior. We have developed a method for detecting malware infection using traffic models based on the similarity between traffic of malware
samples.
Since each malware type displays a unique behavior, we
propose malware infection detection using the traffic models based on the similarity between malware samples. In themethod we developed, there are three basic steps: create feature values by considering time series of the traffic data, cluster malware samples by considering similarities between them,and create a representative model for each malware cluster.
Malware-infected traffic is divided into clusters on the basisof traffic behavior. A model of each cluster is created, andthe models are used to identify target traffic as infected or
normal. With this method, it should be possible to detect infection caused by a new type of malware if the resulting traffic
behavior is similar to that represented by one of the models.

3 Proposed method
3.1 Create feature values by considering timeseries of traffic data
To create a feature vector representing the time series of
the traffic data, the time series is divided into 1-s time slots,
as shown in Figure 1. The time slots are then grouped into
intervals lasting a defined number of seconds for analysis.
97

International Workshop on Informatics ( IWIN 2018 )0s

Features1,2,3,41s

2s

Features1,2,3,43s

4sFeatures Features
1,2,3,4
1,2,3,4

5s

6s

…

7s…

Time

The number of time intervals is thus adjusted by applying dynamic programming matching (DP matching) to the digital
sequences of the two samples. The correlation coefficient iscalculated using the adjusted digital sequences. DP matching
adjusts the time lengths of the two samples by considering
the time-series information and stretching the parts that are
similar between the samples.

Traﬃc data

Figure 1: Division of time series of traffic data into 1-s time
slots

0s

Dividing time-varying traffic into time slots with a fixed duration and monitoring that traffic in units of time slots enables
normal and infected traffic to be distinguished by focusing on
the overall temporal variation in that traffic. In this work, we
set the time-slot width to 1 s and determined the features forevery time slot. The feature values are extracted from eachtime slot, and a vector concatenating the feature values is created for each time slot. In this study, we used minimum packet
size per time slot, number of SYN packets per time slot, ratioof SYN packets to TCP packets per time slot, and number of
ACK packets per time slot as the feature values.

CodedTransition
pattern

1s

Code
c

2s

d→ c

Code
c

c→ c

3s

Code
d

c→ d

4s

Code
d

d→ d

5s

Code
d

d→ d

6s

Code
a

d→ a

7s…

Time

…

Traﬃc data…

Figure 2: Example transition pattern
Malware A
0s~40s40s~80s…
0s~40s40s~80s…

a→a a→b a→c a→d b→a
0%0% 0% 0% 0%
0%
0% 0% 5% 0%…
…
…

a→a a→b a→c a→d b→a
82% 0% 0% 7% 0%
67% 0% 0% 5% 0%

Malware B

3.2 Cluster malware samples by considering
similarities between malware samples

…
…
…
…

d→c d→d
5% 82%
0% 60%

Calculate… correlationd→c d→d
5% 0%
0% 3%

coeﬃcientFigure 3: Example occurrence frequency ratios
To represent the traffic data as a code sequence, extracted
features are clustered (in this study, we used the LBG + splitting vector quantization algorithm [10] to do this), and codeis created for each cluster. The distances between the feature
vector of target time slot and the code for each cluster is calculated, and a search is made for the nearest code. The timeslot is then shifted, and a search is again made for the nearest code. This series of nearest codes is called a “transition
pattern.” An example transition pattern is shown in Figure 2.
The number of occurrences of each transition pattern pertime interval (for example 40 s) is counted, and the ratio of
each target transition pattern to all types of transition patterns
is calculated, as shown in Figure 3. The time interval is thenshifted, and the number of occurrences of each transition pattern per time interval is again counted, and the ratio of eachtarget transition pattern to all types of transition patterns is
calculated.
To evaluate the similarities between two malware trafficsamples, their correlation coefficient is calculated using theoccurrence frequency ratios, like those shown in Figure 3.
The correlation coefficient represents the correlation between
each sample’s digital sequence of the number of transition
patterns × the number of time intervals. The coefficient iscalculated using

3.3 Create representative model for eachmalware cluster
A representative model is created for each malware clusterby extracting a representative malware sample.
The malware samples are classified using hierarchical clustering based on correlation coefficients. Since the optimal
number of clusters is unknown in advance, hierarchical clustering is used as it does not require advance setting of thenumber of clusters. We extracted the malware sample that had
the most malware’s traffic samples with a correlation coefficient greater than the threshold(upper threshold) as the initial
representative malware sample for the cluster. By the same
token, the malware samples for which the correlation coefficient between the two malware traffic samples is less than thethreshold (lower threshold) are deselected in each cluster toremove the samples for which the correlation is weak. This
clustering is repeated until all training samples are divided
into clusters.
The extracted malware traffic sample for a cluster is used
as a representative model for that cluster in order to model
sequential traffic data that actually occurred.

∑ni=1 (xi − x̄)(yi − ȳ)√∑n22i=1 (xi − x̄)
i=1 (yi − ȳ) ,

√∑n

(1)

4 Evaluation

where x and y are the probability variables, x̄ is the meanvalue of x, ȳ is the mean value of y, n is number of transition
patterns × the number of time intervals.
Calculation of the correlation coefficient requires that the nof x equals the n of y. However, each malware traffic sample
has a variable number of time intervals because each malware sample has a variable number of infected time intervals.

4.1 Experimental datasets
We used the anti-Malware engineering WorkShop (MWS)
Datasets [11] for our evaluation. In particular, we used the
CCC DATAset and the D3M Dataset for training. As malwareinfected traffic data, we used 317 malware samples (151 from
CCC DATAset 2010, 156 from CCC DATAset 2011, and 10
98

International Workshop on Informatics ( IWIN 2018 )from D3M 2012) for the training data such that the data used
for training were older than the data used for testing. The normal traffic data used for training were captured between 2011
and 2015.
We also used the CCC DATAset and D3M Dataset for testing. As malware-infected traffic data, we used 200 malware
samples (177 from CCC DATAset 2011, 15 from D3M 2013,
5 from 2014, and 3 from D3M 2015) for the testing data suchthat the data used for testing were newer the data used fortraining.
The CCC 2010 and CCC 2011 attack data include communications prior to malware infection. Thus, given the purposeof our evaluation, we extracted from this attack traffic only
the traffic following malware infection using the method described by our group et al. [6].

Step 3-1 We calculated the distances between the feature vector of time slot and each code. And we assigned codeof minimum distance to the time slot. The time slot isthen shifted, and a search is again made for the nearestcode.
Step 3-2 We assigned a code to all time slots.
Step 3-3 We counted the frequency of each transition pattern in each time interval and represented the ratio of
the frequencies as time-series information. There were
16 transition patterns (a→a, a→b, · · ·, d→c, d→d) because we used four codes. We set the time interval to10, 20, 30, 40, or 60 s. For example, when we set thetime interval to 10 s, we calculated the frequency ofeach transition pattern in each 10-s interval (comprising
ten time slots) and calculated the ratio of the frequencies of each transition pattern. We then shifted the timeinterval and calculated the frequency of each transition
pattern per interval and calculated the ratio of each target transition pattern to all types of transition patterns.

4.2 Experimental methods
To evaluate the effectiveness of the proposed method, we
compared its detection performance with that of three reference methods.

(Step 4) Calculate similarity (correlation coefficient) between
each pair of samples in training data

4.2.1 Detection using proposed method

We calculated the correlation coefficient between eachpair of malware samples. A total of 50,086 (=317 C2 )correlation coefficients were calculated for each interval. We adjusted the time length (number of transition
pattern × number of time interval) of each pair of malware samples by using DP matching.

We performed six basic steps .
(Step 1) Extract features of training data
Step 1-1 We divided the traffic data into 1-s time slots. We
used the packet header information because the payload
information was often encrypted.

(Step 5) Create representative model for each malware cluster for training data
We performed hierarchical clustering using the correlation coefficients calculated in step 4. A coefficient
greater than 0.7 (upper threshold) generally means thatthe correlation is high. A coefficient less than 0.2 (lower
threshold) generally means that the correlation is low.
Given these criteria, we selected the malware sample
that had the most traffic samples with a correlation coefficient greater than 0.7 as the initial representative
malware sample for the cluster. To keep a somewhathigh correlation between each pair of malware trafficsamples in the cluster, we deselected the malware samples that did not correspond to more than 70% of samples in the cluster; that is, the correlation coefficient
was more than 0.2.

Step 1-2 From each time slot, we extracted four features that
we had determined to be effective for infection detection: minimum packet size per time slot, number of
SYN packets per time slot, ratio of SYN packets to TCP
packets per time slot, and number of ACK packets pertime slot. The four features are evaluated as effective
features for infection detection in [6].
Step 1-3 We normalized the extracted feature values by using
the min-max method.
Step 1-4 We represented the time-slot information as a fourdimension feature vector by concatenating the normalized values.

(Step 6) Calculate similarity between two samples in testingdata(Step 2) Create codebook for training data

Step 6-1 We created the time series features of the testingdata using steps 1 and 3.

Step 2-1 We applied the LBG+splitting vector quantization
algorithm to the vectors with the cluster number set tofour.

Step 6-2 We created a model of normal traffic using steps 1
to 5.

Step 2-2 We calculated a representative vector (cluster center) for each malware cluster and collected the vectors
into a codebook representing the characteristics of eachcluster.

Step 6-3 We calculated the cumulative minimum distance between each representative cluster model and the target
malware traffic sample and calculated the cumulative
minimum distance between the model of normal traffic
and the target malware traffic sample.

(Step 3) Create time-series representation of training data99

International Workshop on Informatics ( IWIN 2018 )Step 6-4 We compared the two distances for each sample. If
the distance between the representative cluster modeland the sample was greater than that between the modelof normal traffic and the sample, the sample was identified as normal. Otherwise it was identified as infected.

Table 1: Identification rate of proposed method
Time
No. of patterns
Identificationinterval (s) of infected traffic
rate (%)
10
17
100
20
15
100
30
12
100
40
12
100
60
11
99.0

4.2.2 Detection using one representative model
For detection using one representative model, we used thetime-series information. We did not use the similarity between pairs of malware samples. The average malware trafficsample of the training data was treated as the representativemodel of malware-infected traffic.
We created the time-series information for the target malware traffic samples using steps 1 to 3 above. We calculated
the mean ratio of the frequencies of each transition pattern for
all malware traffic samples and selected the sample that wasclosest to the mean as the representative model of malwareinfected traffic.
For testing, we created time-series information for the malware traffic samples using steps 1 to 3 above. We calculated
the cumulative minimum distance between the target sample and the model of infected traffic. We also calculated thecumulative minimum distance between the sample and themodel of normal traffic and identified the sample as normal
or malware-infected on the basis of the two distances.

correctly identified malware-infected traffic samples divided
by the total number of such samples in the testing data.
The identification rate was 100% for time intervals of 10,
20, 30, and 40 s, meaning that it is robust against the timeinterval.

5.2 Identification rate of other methods
To evaluate the effectiveness of proposed method, we compared its identification rate with those of the three reference
methods. The identification rate of the one-representativemodel method (section 4.2.2) is shown in Table 2. The identification rate of the security-vendor-definition-based method(section 4.2.3) is shown in Table 3. The number of patternsof infected traffic is five because the data used included fivemalware families.
The identification rate of proposed method is better thanthose of other methods.

4.2.3 Detection using models based on security vendor’s
definitions

Table 2:
methodFor detection using models based on a security vendor’s definitions, we used the time-series information and clusters forclassification. We did not use the similarity between pairs of
malware samples.
We created time-series information for the target malware
traffic sample using steps 1 to 3 above. Next, we divided
the training malware traffic samples into clusters defined by
the security vendor: BKDR, PE, Mal, TROJ, andWORM. We
calculated the mean ratio of the frequencies of the transition
patterns of the malware samples in each cluster. We selected
the sample in each cluster with the frequency closest to themean as the representative model of malware-infected traffic.
We calculated the cumulative minimum distance between
the model of malware-infected traffic and target traffic sample
and calculated the cumulative minimum distance between themodel of normal traffic and target sample. We identified thesample as normal or infected on the basis of the distances.

Identification rate of one-representative-modelTimeinterval (s)
10
20
30
40
60

No. of patternsof infected traffic
1
1
1
1
1

Identification
rate (%)
12.5
14.5
25.5
32.5
52.5

Table 3: Identification rate of security-vendor-definitionbased method
Time
No. of patterns
Identificationinterval (s) of infected traffic
rate (%)
10
5
47.5
20
5
78.0
30
5
92.0
40
5
87.5
60
5
98.5

5 Results
5.1 Identification rate of proposed method

6 Conclusion

The identification rate of the proposed method by changing the time interval is summarized in Table 1. The time interval is the duration during which the code transitions werecounted, as described in section 3.2. The number of patternsof infected traffic is the number of hierarchical clusters, as described in section 3.3. The identification rate is the number of

Our method for detecting malware-infected traffic samples
is based on the similarity between the pair of malware samples in this paper. Simulation evaluation demonstrated thatthe proposed method can effectively identify malware-infecttraffic with high accuracy.
100

International Workshop on Informatics ( IWIN 2018 )Future work includes conducting a large-scale experiment
to better evaluate the effectiveness of the proposed method.

Acknowledgment
This work was supported by JSPS KAKENHI Grant Number 15H01684.

REFERENCES[1] Y. Otsuki, M. Ichino, S. Kimura, M. Hatada, H.
Yoshiura, Evaluating payload features for malware infection detection, Journal of Information Processing(JIP), Vol. 22, No. 2, pp. 376-387 (2014).
[2] K. Kuwabara, H. Kikuchi, M. Terada, M. Fujiwara,
Heuristics for Detecting Types of Infections from Captured Packets, Computer Security Symposium, pp.1-6(2009)(in Japanese).
[3] A. Mohaisen, A. G. West, A. Mankin, O. Alrawi, Chatter, Classifying Malware Families Using System Event
Ordering, IEEE Conference on Communications andNetwork Security, pp.283-291 (2014).
[4] D. Chiba, T. Yagi, M. Akiyama, K. Aoki, T. Hariu,
Design and Evaluation of a Profiling Method to Detect Post-infection Communications, Computer Security
Symposium, pp. 960-967 (2014)(in Japanese).
[5] Gautam Thatte，Urbashi Mitra and John Heidemann,
Parametric Methods for Anomaly Detection in Aggregate Traffic，IEEE/ACM Transactions on Networking,vol. 19, issue 2, pp.512-525 (2011).
[6] M. Ichino, K. Kawamoto, T. Iwano, M. Hatada, H.
Yoshiura, Evaluating header information features formalware infection detection, Journal of Information
Processing, vol.23, no. 5, pp. 603-612 (2015).
[7] T.-F. Yen and M. K. Reiter, Traffic Aggregation for Malware Detection, 5th International Conference, Detectionof Intrusions and Malware, and Vulnerability Assessment, pp.207-227 (2008).
[8] T. Ichida, M. Ichino, M. Hatada, N. Komatsu, H.
Yoshiura, A study on malware detection method using
feature’s state transition, Symposium on Cryptography
and Information Security, 1E1-2 (2012)(in Japanese).
[9] G. Nychis, V. Sekar, D. G. Andersen, H. Kim, H.
Zhang, An Empirical Evaluation of Entropy-based Traffic Anomaly Detection, IMC ’08 Proceedings of the 8th
ACM SIGCOMM conference on Internet measurement,
pp.151-156 (2008)．
[10] Y Linde, A. Buzo, R. Gray, An Algorithm for Vector
Quantizer Design, IEEE Trans. on Commun., vol. 28 no.
1, pp.84-95 (1980).
[11] M. Hatada, M. Akiyama, T. Matsuki, T. Kasama, Empowering Anti-malware Research in Japan by Sharing
the MWS Datasets, Journal of Information Processing,
pp.579-588 (2015)101

International Workshop on Informatics ( IWIN 2018 )A Method of Lightweight Secure Communication
Considering Reliability in IPv6 Wireless Sensor Network
Shunya Koyama*, Yoshitaka Nakamura ** , and Hiroshi Inamura ***Graduate School of Systems Information Science, Future University Hakodate, Japan**
School of Systems Information Science, Future University Hakodate, Japan{g2117020, y-nakamr, inamura}@fun.ac.jp

Abstract - Recently, IPv6 wireless sensor networks arewidely spread in various fields including IoT environments.
However, on these sensor networks, it is difficult to use
secure communication technologies that can become large
overhead, due to power saving of the wireless nodes isimportant. As one approach to deal with this problem, amethod of focusing on Nonce which is one element of
security and separating it from secure communication is
proposed. However, it remains a problem that not suit inenvironments such as wireless sensor networks wherereliability of communication is not ensured. In this paper,
we propose a Nonce truncation method that can deal withsuch environments. Our method transfer information of
about several bits that can estimate the Nonce associated theciphertext as the truncated Nonce value. We evaluated theeffectiveness of our method by comparing the lifetime of thenodes between the method and previous methods.

Personal Area Networks) which compresses IPv6 header or
UDP header, RPL (IPv6 Routing Protocol for Low Powerand Lossy Networks) which is a routing protocol to support
the above-mentioned unstable network path reliability, andso on.
While it is expected that such communication scheme
targeting LLNs based on IEEE 802.15.4 will become
widespread, security problems aimed at these constrained
networks are also becoming apparent [4]. Also, there areproposals for the lightweight secure communication
methods for sensor networks not using IP which was amajor before the spread of IoT service, but it has abackground which is very different from recent sensor
network. For this reason, it is difficult to apply conventional
security technology for the LLNs environment. Especially,
the problems that be not able to support IEEE802.15.4 small
frame size, and unstable communication quality are left.
In this paper, we discuss the unstable communication
quality and the resource constraints of sensor devices which
are the features of LLNs. Then, we design the securecommunication method that can deal with these features. To
this end, we focus on Nonce which is one of the security
elements and address a method to truncate this. Also, in thismethod, we design a lightweight secure communication
scheme that can operate without applying excessive
overhead to sensor devices at any frame loss rate.

Keywords: IoT, Reliability of Communication, Secure
Communication, Nonce

1

INTRODUCTION

IPv6 wireless sensor networks are widely spread in various
fields including IoT environments lately, because of thedevelopment of low-power sensor devices and wirelesscommunication technologies. The penetration rate of thesedevice has been increased, and about 50 billion devices will
be interconnected in 2020 [1]. It is also expected to be
utilized in various fields.
On the other hand, there are constrained networks that
impose strict restrictions on the computing power and thecommunication quality of the sensor devices. They arecalled LLNs (Low power and Lossy Networks) [2], arecomposed of communication devices with limited
computing resources. Also, the reliability of communication
is not guaranteed due to high packet loss rate and so on.
These strictly constrained networks are needed to meet thedemand of IoT services in various fields.
As one of the proposals for dealing with such constrained
networks, IETF (Internet Engineering Task Force) has
established a policy to expand part of it based oncommunication standards used in conventional wirelesssensor networks representing Zigbee [3]. As a typicalexample of this, there is a method of providing an adaptation
layer for using IPv6 (Internet Protocol v6) technology onIEEE 802.15.4 which is a data link layer technology for
power saving of communication equipment. Specifically,
there are 6LoWPAN (IPv6 over Low-Power Wireless

2

RELATED WORKAs previous method, a lightweight secure communication
system has been proposed, which is focusing on Nonce(Number used once) that is a part of security elements. In
the following, we describe the mechanism of the previous
method and its applicability to LLNs, based on the basic
secure communication technology.

2.1 Overview of basic secure communication
In this section, we describe the general securecommunication establishment method, and the secure design
when applying it to the LLNs based on IEEE 802.15.4, inconsideration of the data frame structure.

2.1.1 Establishing general securecommunication
Strictly speaking, the establishment method of securecommunication differs depending on the Block Cipher
Modes of Operation selected. As famous example of the103International Workshop on Informatics ( IWIN 2018 )Modes of Operation, there is CCM (Counter with CBCMAC) mode combines confidentiality and authenticity in an
efficient way as authenticated encryption mode. The CCM
mode coincides exactly with the design concept of thecommunication scheme for LLNs from the viewpoint of its
versatility, resource constraints, frame size limitation.
Therefore, we focus on the CCM mode and explain theoperation outline.
CCM mode provides security using secret key and Nonce,and MAC (Message Authentication Code). Here, Nonce is asecurity element to make it possible to use the same Key
multiple times without security risk, and MAC is anarbitrary security element to provide integrity or authenticity.
At this time, in particular with respect to the calculation
method of Nonce, the value corresponding to each encrypted
frame must be unique from the viewpoint of security risk. In
the NIST (National Institute of Standards and Technology),
they have listed several recommended specifications andcalculation methods of Nonce, and the size should be 8
bytes or more [5]. Further, as one of the calculation methods,
a method using a counter value starting from an arbitraryvalue is recommended. The value is incremented and sharedevery time different ciphertexts are generated. The methods
are described later are based on this calculation method.

completely eliminated, on average about 16% and about
12% of the payload can be expanded.
In order to properly operate the Block Cipher Modes ofOperation, there is no strict restriction that each security
element must secure a certain size or more. However, if you
select the smallest value among the simply selectable sizes,
there is also the possibility of impairing the safety of securecommunication. From that point of view, NIST recommends
size of Nonce is 8 bytes or more. Thus, a method of
reducing the size without losing the safety of securecommunication is ideal.

2.2 SNEP (Secure Encryption Network
Protocol)Following the previous section, a method of separating
Nonce from communication and reducing its size to zero
without reducing the safety of secure communication called
SNEP (Secure Network Encryption Protocol) [6] has been
proposed. Specifically, this method shares only the initialvalue of Nonce using communication, and thereafter
incrementing Nonce value stored in the sensor devices
according to the number of received encrypted frames. If an
encrypted frame is lost in the middle due to interruption ofcommunication, resynchronization process is performed totransmit the entire value of Nonce. It is shown that in anenvironment with the stable communication quality, thecommunication overhead by the secure communication is
reduced. On the other hand, in the environment such as
LLNs which the communication quality is unstable, theresynchronization process frequently occurs. Therefore, thismeans secure communication overhead is actually increase,and network congestion problem may occur.

2.1.2 Secure communication design in LLNs
Fig.1 shows an example of a simple data frame structures
when the above described secure communication is applied
to LLNs.
802.15.4 Frame Structure (127 [bytes])(a) 802.15.4 + IPv6 + UDP
MAC Header

Nonce

IPv6 Header

UDP
HeaderData Payload

3 ~ 23 [bytes]

8 [bytes]

40 [bytes]

8 [bytes]

30 ~ 66 [bytes]

MAC

3

FCS

3.1 Research tasks(b) 802.15.4 + 6LoWPAN + Compressed UDP + RPL
MAC Header

Nonce

6LoWPAN
HeaderUDP
Header*Data Payload

3 ~ 23 [bytes]

8 [bytes]

3 ~ 36 [bytes]

1 ~ 6 [bytes]

36 ~ 110 [bytes]

MAC

In the previous method, if an encrypted frame is lost in the
middle due to interruption of communication or the like, it isnecessary to repeat the resynchronization process of Noncefor recovery secure communication. Therefore, it is notsupport to environment where the frame loss rate can behigh. Also, according to a general secure communication
method, the ratio of encrypted frames occupied by Nonce islarge, and there is a possibility that a heavy load is applied
to the sensor devices and the network itself due to inefficient
fragment processing. For this reason, any method is difficult
to adopt to LLNs where communication quality is unstableand frame size is limited, and a method capable of dealing
with these problems is required.
In this paper, we propose a method to deal with the above
problem by estimating Nonce only by sensor devices itself
from the truncated value that the size changes according to
the frame loss rate.

FCS

0 ~ 16 [bytes] 2 [bytes](c) 802.15.4 + 6LoWPAN + Compressed UDP + RPL + SNEP
MAC Header

6LoWPAN
HeaderUDP
Header*Data Payload

3 ~ 23 [bytes]

3 ~ 36 [bytes]

1 ~ 6 [bytes]

44 ~ 118 [bytes]

PROPOSAL METHOD

0 ~ 16 [bytes] 2 [bytes]

MAC

FCS

0 ~ 16 [bytes] 2 [bytes]* Compressed

Figure 1: Structure pattern of encrypted frames in LLNs
In Fig.1, (a)(b)(c) commonly indicate the frame structure
when IP technology is introduced on IEEE 802.15.4 andencrypted using CCM mode. In addition, for each frame
structure, (a) is introduced UDP into IPv6, (b) is introduced
6LoWPAN and RPL over (a), and (c) is introduced SNEP
described later in section 2.2 of previous method over (b).
As can be seen from the figure, the security elements
communicated can be large overhead and suppress MAC
payload in the environment with limited frame size as LLNs.
Therefore, there is a possibility of increasing the processing
load of sensor devices through fragment processing, it isdesirable to make the size as small as possible. In particular,
in each frame structure excluding (c), the ratio of Nonce to
MAC payload occupies so large that if Nonce can be

3.2 Basic operation
In this section, we describe the basic mechanism fortruncating a Nonce. As the block cipher mode of operation
104

International Workshop on Informatics ( IWIN 2018 )for establishing secure communication, CCM mode are used.
Also, as a calculation method of Nonce corresponding toeach encrypted frame, a counter value that increment thevalue according to the frame is used. First step, the initialvalue of Nonce is shared between Sender and Receiver, andthe whole value is stored in the sensor device as in theprevious method. Thereafter, in the sharing of Nonce, only
the N of least significant bits (N LSBs) are assigned oncommunication. Hereinafter, this N bit is called a truncatedNonce length. Fig.2 shows the operation flow in the casewhere the truncated Nonce length is 1 as the specificexample.
In Fig.2, (a)(b) commonly begin already synchronizedentire Nonce between Sender and Receiver devices andestimate the entire Nonce value from the truncated value
while the devices communicate several encrypted frames.
First, (a) shows that the receiver succeeds in receiving thethird encrypted frame after losing only the second encrypted
frame. At this time, since there is a difference in the entire
value of Nonce internally stored between the two sensor
devices, receiver fails in decryption the third encrypted
frame. At this stage, move on to step 2 of (a). Since thevalue of the received truncated Nonce is 1, receiver can
decrypt the third encrypted frame by estimating entire valueof Nonce that is greater than internal Nonce value and 1
LSB equals 1. On the other hand, in the case of (b), receiverreceives the fourth encrypted frame after losing the second
and third encrypted frames, hence fails in decryption even if
estimates entire value of Nonce like (a). In the case (b)shown in this figure, although correct Nonce is "…0010
0110b", in fact it is estimated "…0010 0100b" by mistake.
In such a case, recovery secure communication by
performing resynchronization process sharing the entire
value of Nonce. Generally, such resynchronization process
occurs only when the truncated Nonce length is x bits andFigure 2: Operation flow in the case where the truncatedNonce length is 1x

the frame is lost consecutively for 2 times or more. For
example, in the case of (b), this process occurs because the

3.3 Optimization of resynchronization process
occurrence count1

frame has been lost 2 times that is twice consecutively.
Therefore, depending on the selection of the truncatedNonce length, the same problem as SNEP may still occur.
For this reason, it is necessary to select the truncated Nonce
length flexibly so as to minimize the number of resynchronous process according to the frame loss rate of thecommunication environment. Table 1 shows the occurrence
probability of resynchronization process according to x bitsof truncated Nonce length and frame loss rate.

Considering the characteristics of LLNs, it is necessary tominimize the occurrence probability of resynchronization
process as much as possible so that the same problem as
SNEP does not occur. For that purpose, it is ideal to flexibly
select the truncated Nonce length as short as possible
according to the frame loss rate.
In order to deal with this problem, we use ETX (ExpectedTransmission Count) adopted in routing protocols used inmany wireless sensor networks. ETX is a metric index using
link quality, and its value is defined as the reciprocal of theframe arrival rate. Therefore, the sensor devices having
information corresponding to Table 1, can select thetruncated Nonce length dynamically to minimize theoccurrence probability of the resynchronization process toany value or less by using this value.

Table 1: Probability of resynchronization process
occurrence according to the truncated Nonce length andframe loss rate
Frame Loss Rate

80%

60%

40%

20%

Ept

1

64%

36%

16%

4%

Ept2

2

40%

13%

2.5%

0%

Ept2

4

2.8%

0%

0%

0%

Ept2

Truncated
Nonce Length

x

80%

2x

60%

2x

40%

2x

20%

1

2

4

4

2x

EVALUATIONS

About the proposed method and previous methods in thisresearch, we performed evaluation experiments after

x

Ept2

105

International Workshop on Informatics ( IWIN 2018 )implementing these on the network simulator. Hereinafter,
we describe the experimental environment, evaluation
method, experiment method and the detail of these.

4.3.1 Experiment for each combination offrame loss rate and truncated Nonce
length

4.1 Experiment environment

In this experiment, we evaluate whether the length of eachNonce can correspond to any communication quality
assuming LLNs environment as proposed method andprevious method. First, about the lifetime ratio of eachsensor devices, we calculated from the power consumption
until the Receiver took 1,000 KB of data from the Sender 10times and decrypted all the data. At this time, to evaluate theperformance for each length of Nonce in accordance withthe frame loss rate, the experiment was proposed at intervalsof 10% to 20% in the frame loss rate in Table 2.

We implemented the proposed method and (b)(c) in Fig.1
as the previous methods on devices and operated on the
network simulator. Specifically, we created a simple smallscale model that established secure communication between
two sensor devices such as Sender and Receiver, operated
each method in this model. At this time, we emulated all
sensor devices as Zolertia Z1 hardware [7]. For simplicity,
unidirectional communication is performed from the Sender
to the Receiver, and encrypted frames are transmitted andreceived in this scenario.
Detailed simulation parameters in the experimental
environment are shown in Table 2. The communication
standard conforms to (b) in Fig.1 as general standard. In
addition, a frame loss rate is used as an index representing
communication quality in LLNs. Also, only the length ofNonce is selected from among 0 to 8 bits or 8 bytes different
according to the frame loss rate. Moreover, considering
resynchronization process due to frame loss, experiments
were performed until all data arrives at Receiver andcompletely
decrypted
after
establishingsecurecommunication.

4.3.2 Experiment for continuing to select the
optimal truncated Nonce length
In this experiment, we evaluate whether the effectivenesscan be shown compared with the previous method when
dynamically selecting optimum Nonce length using the
proposed method. The basic simulation parameters were asshown in Table 2, but the frame loss rate was changed
randomly between 20% and 80%, and the occurrence
probability of resynchronization process was always 5% orless using ETX. Also, it was assumed that 1000 KB of data
was transmitted ten times a day. In such an environment, we
calculated the average lifetime ratio of sensor devices ineach method.

Table 2: Simulation parameters
Parameter

Value

Data Link Protocol

IEEE802.15.4

Network Protocol

6LoWPAN + RPLTransport Protocol

Compressed UDP

Frame Size

127[bytes]

Transfer Data

1000[Kbytes] * 10

Cipher Mode

AES-CCM*Key Length128[bits]

Block Length128[bits]

MAC Length8[bytes]

Frame Loss Rate

0%～90%

Nonce Length0, 1, 2, 4, 8[bits], 8[bytes]

5

RESULTS AND DISCUSSION

5.1 Results
The result obtained in each experimental method areshown in the following section.

5.1.1 Experimental results for eachcombination of frame loss rate andtruncated Nonce length
The result obtained by the experiment according to the
combination of frame loss rate and truncated Nonce lengthis shown below.

4.2 Evaluation method

1.2

1.1

LifeTime（Ratio）

In each experimental method, we measured the lifetime ofthe sensor device from the power consumption of the Senderemulated as Zolertia Z1 hardware. We evaluate theeffectiveness by calculating and comparing the lifetime ratioof each method where general method (b) in Fig.1 as 1 value.

4.3 Experimental method1
x=0[bit]0.9

x=1[bits]
0.8

x=2[bits]
x=4[bits]

0.7

x=8[bits]
0.6
0.5
100% 90%

One experiment for each combination of frame loss rate
and truncated Nonce length and the other experiment in thecase of continuing to select the optimal truncated Nonce
length to minimize the number of synchronization process.
We describe the details of each experiment method below.

x=8[bytes]

80%

70%

60%

50%

40%

30%

20%

10%

0%

-10%

Frame Loss Rate

Figure 3: Lifetime ratio according to truncated Nonce
length and frame loss rate by simulation

106International Workshop on Informatics ( IWIN 2018 )Fig.3 shows Sender's lifetime ratio measured for eachframe loss rate and truncated Nonce length (hereinafterreferred to as x) in the simulation parameters shown in Table
2. In the case where the frame loss rate was 20% or less, all
the proposed method and the previous method had improved
the lifetime compared with the general method oftransmitting 8 bytes of Nonce. On the other hand, when theframe loss rate exceeded 20%, the lifetime sharply
decreased according to the length of Nonce. In particular,
the rate of decreased was remarkable when the truncatedNonce length was 4 bits, but in the case of 8 bits, any frame
loss rate was improved. Also, it cloud be seen that thetruncated Nonce length at which the lifetime improves mostwas different depending on the frame loss rate except 0%.

by network congestion problem. It is also necessary toconsider approaches to deal with these problems.

6

In this paper, we discussed the unstable communication
quality and the resource constraints of sensor devices which
are the features of LLNs. Then, we designed the securecommunication method that could deal with these features.
To this purpose, we focused on Nonce which is one of the
security elements and proposed the method to truncate this.
As a result, we confirmed the proposed method improved
the lifetime as lightweight secure communication methodthat deal with unstable communication quality which was
the problem of the previous method. As the evaluation
method, we implemented the proposed method andconventional method on sensor terminal which emulated,measured its lifetime ratio and compared it.
As future prospects, there are we should address examine
experiments and evaluation methods considering various
more real environments. And, it is also necessary to deal
with the response to burst loss and the delay problem of theencrypted frames in connectionless network. Furthermore,in order to improve the proposed method, we will adjust the
number of times to estimate the entire Nonce value
according to the truncated Nonce length by measuring andcomparing the processing load in the decryption process andresynchronization process.

5.1.2 Experimental results for continuing toselect the optimal truncated Nonce
lengthThe result obtained by the experiment for continuing toselect the optimal truncated Nonce length so that theoccurrence probability of the re-synchronous process within
5% is shown in following Table 3.
Table 3: Lifetime ratio obtained for each method bysimulation
MethodLifetime Ratio

General( Nonce Length: 8[bytes] )1

Previous( Nonce Length: 0[byte] )0.625

Proposal

1.058

CONCLUSIONREFERENCES[1] D.Evans (Cisco Internet Bussiness Solutions Group),“The Internet of Things - How the Next Evolution of the Internet Is Changing Everything, ” <https://www.cisco.com/c/dam/en_us/about/ac79/docs/innov/IoT_IBSG_
0411FINAL.pdf > [Accessed May 20, 2018].

The effectiveness of the proposed method is clear because
the proposed method was improved the lifetime by about
6%, while SNEP that previous method dropped the lifetime
about 37% when compared with the lifetime of general
method that transmitted 8 bytes of Nonce.

[2] C. Bormann, M. Ersue, and A. Keranen, “Terminology
for Constrained-Node Networks,” Internet Engineering
Task Force RFC7228, 2014.

5.2 Discussion[3] ZigBee Alliance., “Zigbee specification. Technical Rep
ort Document 053474r20,” Zigbee Alliance, 2014.

From the results shown in Fig.3 and Table 3, the lifetimeof the proposed method is better than previous methods.
Also, if the truncated Nonce length is about 4 to 8 bits, thelifetime is roughly improved in any frame loss rate, except
when the loss rate is extremely high. This means thattruncated Nonce length is enough size to operate in the
LLNs environment. On the other hand, depending on theselect of the truncated Nonce length, it is also clear that thepossibility of greatly decreasing the lifetime also remains.
So, it is effective to select continually the optimum
truncated Nonce length.
However, in the experimental environment, sinceevaluation is limited to a simple secure communication
model between two devices, in the future it is necessary toverify the effectiveness from many aspects according to thereal environment. Particularly, there are many problems
such as dealing with frame delay, handling burst loss caused[4] D.Airehrour, J.Gutierrez, and S.K.Ray, “Secure Routin
g for Internet of Things: A survey, ” Journal of Networ
k and Computer Applications, Vol.66, pp.404-412,201
6．
[5] National Institute of Standards and Technology, “FIPS
PUB 140-2 Security Requirements for Cryptographic
Modules, ” 2002.
[6] A.Perrig, R.Szewczyk, J.D.Tygar, V.Web, and D.E.Cul
ler.,“SPINS: security protocols for sensor networks,”
Wireless Networks Journal, Vol.8, pp.521-534, 2002.
[7] Zolertia, “Zolertia Z1 Datasheet,” <http://zolertia.sourc
eforge.net/wiki/images/e/e8/Z1_RevC_Datasheet.pdf>[Accessed May 20, 2018].

107

International Workshop on Informatics ( IWIN 2018 )Effective Countermeasure Against Biometric Spooﬁng Attacks using
Unsupervised One-class Learning
Vishu Gupta† , Masakatsu Nishigaki† , Tetsushi Ohki††

Faculty of Informatics, Shizuoka University, Japanvishu@sec.inf.shizuoka.ac.jp, {nisigaki, ohki}@inf.shizuoka.ac.jp
Abstract - With the advent of new technologies, the methods of presentation attacks as well as the security measurestaken against it is diversifying with each passing day and arecompeting with each other. Yet the imposter is able to makean access to a system illegally by deceiving the security throughthe use of material containing artiﬁcial biometrics traits likeprinted photo, display etc. Therefore, we propose a novel presentation attack detection algorithm which can ensure security against unknown presentation attacks without any priorknowledge of fake samples. Moreover, our proposed algorithm can detect presentation attack with single static image
only. The key tasks are divided into two parts, creating asmooth manifold of live samples and determining whether
the query image is included in the manifold. In this paper,
we utilize one class system such as SVM(Support Vector Machine) and DCGAN(Deep Convolutional Generative Adversarial Network) to learn the manifold of live samples. For
DCGAN we propose a liveness scoring scheme based on the
AnoGAN(Anomaly Generative Adversarial Network) framework. Based on these, we utilize the proposed method to facepresentation attack detection. Through our experiment wewere able to produce decent results by using palm live/fakeimage dataset.
Keywords: biometrics, spooﬁng, presentation attack detection, anomaly detection, generative adversarial networks

1

Introduction

In a society where not only simple tasks but decision making of people to computers is coming it will become an important requirement to guarantee that the outsourcing was performed by the user’s own will, and also to correctly detect itwhen counterfeiting acts are forged or improperly tampered.
It is essential to guarantee the authenticity of the terminal in
addition to the authenticity of the terminal itself to satisfy
these requirements. Biometric authentication system is drawing attention, which can guarantee the authenticity of the terminal user.
Biometric authentication system (BAS) registers preliminary collected biometric information as a template and veriﬁes whether it belongs to a legitimate user by calculating thesimilarity with the biometric information acquired at the timeof authentication. BAS uses a biometric feature of the person without fear of forgetting, losing, or theft compared to anauthentication method using a password or a token.
On the other hand, biometric information such as faces,sounds, ﬁngerprints, handwriting is difﬁcult to keep secret indaily life. Biometric presentation attack is becoming a large

threat since fake biometric information becomes more sophisticated along with the rapid development of sensors, printers,
and manufacturing machines.
To develop a BAS that is secure against presentation attacks, demand for designing a robust presentation attack detection(PAD) algorithms which classify an input sample aslive or fake is increasing.
Many previous approaches discussed the PAD features which
can guarantee security against a speciﬁc impersonation attack
such as frequency spectrum for printed photo [1], three dimensionalities of live face [2] and so on. However, the methods of presentation attacks are diversifying day by day. It isdifﬁcult to learn in advance PAD features that can detect allthese attacks.
Regarding the problem, PAD algorithms have made it possible to detect various presentation attacks by combining multiple classiﬁers that solve binary classiﬁcations problem between live and fake samples such as [3, 4]. However, thesemethods still have some problems. At ﬁrst, it is necessary
to obtain not only biometric samples but also a large numberof fake samples for each type of presentation attack. Second, the PAD algorithm does not guarantee whether an unknown sample is classiﬁed as a presentation attack. Here wedeﬁne unknown sample as a sample that is not included in
the samples for training. Note that unknown sample includes
not only samples intended to resemble live sample but also
any synthetic samples since it is sufﬁciently effective in theregistration process.
The subject of this paper is to investigate the security against
the presentation attack using an unknown sample. This solves
the fundamental problem of making PAD difﬁcult due to thediversiﬁcation of attacks. The main contributions of this work
are as follows:
1. We propose a novel Presentation Attack Detection (PAD)
algorithm which can be learned only with live samples
and guarantee security against unknown sample by utilizing GAN based anomaly detection algorithm.
2. Proposed PAD algorithm is evaluated with public database(Replay-Attack Database [5]) and achieved 3% of HTER
(Half Total Error Rate) by using a model trained onlywith live samples.

2

Related WorkAll the prior research that has been conducted on Anomaly
detection is performed by having to train the system by using
both live samples along with fake samples which are used for109International Workshop on Informatics ( IWIN 2018 )presentation attack. For all these conducted researches, thecore difference lies in the method used to model the real andfake attempts. Prior methodologies based on the employedcues are being classiﬁed in a recent study [6] where they are
divided into three major categories.
The ﬁrst category is a method to detect face liveness which
relies on image quality/distortion measures. Work in [7] which
consists of detecting print attacks using the difference in the
2D Fourier spectra is an example of the method in this category. The work stated in [8] utilizes the Lambertian model
which comprises of variational Retinex based approach andGaussian ﬁlters difference as its two methods.
The second category uses methods which are based on detecting different signs of vitality which make use of characteristics corresponding to live faces. For example, presentation
attack detection in [9] uses blinking which is used with others cues in other work. Such as, [10] recommend the use of
all the dynamic information content of the video represented
using dynamic mode decomposition method.
The last category consists of methods based on the difference in motion patterns between real and presentation attacks. It is assumed that presentation attack has rigid motion
whereas real-access attempts has both rigid and non-rigid motion. These approach depends on the fact that real accesses
correlate with 3D structures whereas presentation attack media are often at 2D planes. Eulerian motion magniﬁcation using two sets of features composed of LBPs(Local Binary Patterns) [11] to enhance facial expressions is a typical case of
the method in this category. New liveness detection method is
proposed in [12] which utilizes the difference in optical ﬂow
ﬁelds generated by the movements of 2D planes and 3D objects.
While most of the existing methods uses real access data to
try and learn a general classiﬁer to outline presentation attack
attempts, work in [13] uses both texture and motion cues, theauthors built two presentation attack detection methods, one
being a generative approach while the other being a discriminative method to study the client-speciﬁc information embedded in the feature space and its effects on the performance ofthe system.
The current work in terms of detection mechanism sharesome similarities to the existing approach which utilizes image content representation, is distinct in the way we formulate the existing the detection problem. The common approach used to detect anomaly in an image uses two-class
formulation where they separate the negative from the positive samples, our proposition uses one-class pattern classiﬁcation methods, testing it in a modiﬁed as well as an existing
method which yields good results to identify presentation attack attempts. Moreover, the evaluations are performed byusing a self-made database which better reﬂects the difﬁculties of detection in realistic scenarios.

3

Presentation Attack Detection using
Anomaly GAN

3.1

Generative Adversarial Networks

Goodfellow et al. introduced a concept of Generative Adversarial Network(GAN) [14] which learns a generator expression indistinguishable by a discriminator by training agenerator model and discriminator model simultaneously. The
aim of the generator is to fool the discriminator by learning
the probability distribution of the input samples. Let x be an
input sample whose true probability distribution is p(x). G
is a generator that takes a latent vector z randomly selectedfrom the latent space Z and outputs a new sample G(z). The
discriminator D then outputs the probability that the given
input is either the true input from p(x) or the G(z) from thegenerator. These two models are simultaneously trained using the min-max game of the formula:min max V (D, G) = Ex∼p(x) [log D(x)] +DGEz∼pz (z) [log(1 − D(G(z)))](1)

Radford et al. [15] introduced deep convolutional generative adversarial networks (DCGAN) for unsupervised learning of features by utilizing convolutional neural networks as
the generator and discriminator network. More speciﬁcally,
they replaced the pooling layer with stride convolution layer
so that the network can learn its own spatial upsampling. Additionally, they removed the full connection layer at the topof the convolution feature to improve the model stability. Finally, batch normalization was utilized to suppress training
problems caused by poor initialization and helps propagationof gradients in deep models by normalizing each unit to havezero mean and unit variance.

3.2 Proposed Anomaly GAN for PAD
To detect presentation attack using single image, we propose a unsupervised learning to identify anomalies in imaging data as candidates for fake sample. Figure 1 shows an
overview of our proposal. Our proposed scheme is based on
unsupervised anomaly detection scheme proposed in [16](hereafter, AnoGAN). AnoGAN uses DCGAN to learn a manifoldof live sample variability, acompanying a anomaly scoring
scheme based on the mapping from an image space to a latent space.

3.2.1 Palm Imaging Model
We learn the palm image manifold X on the image space with
unsupervised learning using only the live palm images. When
query image is not included in the learned manifold X , it canbe detected as an unknown input.In DCGAN [15], generator
uses latent vector z chosen from latent space Z uniformly
at random to obtain a smooth mapping G(z) to palm image
manifold X .

110

International Workshop on Informatics ( IWIN 2018 )LD (z ) using at the α update iteration of the mapping procedure to the latent space, respectively. All score functions
output a large score for an anomaly image. In our experiments, we use λ = 0.9 in equations (2) and (5) which was
found empirically due to preceding experiments on our palm
dataset.

4

Figure 1: Overview of our proposal
3.2.2

Deriving latent vector

We can detect the Presentation Attack by checking whether
query image xq is included in the palm image manifold X
learned in the clause 3.2.1. Since DCGAN calculates G(z)using the randomly chosen latent vector z, G(z) corresponds
to a random point on the palm image manifold X . Consequently, the distance between G(z) and the query image xq
does not necessarily become small even if it is a live sample.
Therefore, to detect an unknown sample, we should conﬁrm
the existence of a latent vector z that has sufﬁciently small
distance between the query image xq and G(z ) on the manifold X .
For ﬁnding the z from randomly chosen latent vector z, we
use the backpropagation approach proposed in [16]. The loss
function L(z γ ) for backpropagation is deﬁned as follows:L(z γ ) = (1 − λ) · LR (z γ ) + λ · LD (z γ )(2)where z γ is an updated latent vector to fool discriminator
D, LR (z γ ) is the generator loss, LD (z γ ) is the discriminatorloss and λ is a ﬁxed parameter for convex combination. The
residual loss and the discriminator loss can be obtained asfollows:LR (z γ ) =|xq − G(z γ )|(3)LD (z γ ) =|f (xq ) − f (G(z γ ))|(4)
where f (·) is an output of the discriminator function. Only
the coefﬁcients of z are adapted via backpropagation. The
trained parameters of the generator model and discriminator is obtained by apmodel are kept ﬁxed. In our proposal, zplying backpropagation process α times with query image xq
 is used in classiﬁand randomly selected z. The obtained z
cation process.
3.2.3

Experiment

In this section, ﬁrst a description of the custom made database
and the evaluation protocols used in this experiment is provided, following by experimental results obtained from thedatabase used. All the experiments were carried out using
Python with the tensorﬂow and pytorch library on a machinewith conﬁguration (Intel i7-5930K, 64GB RAM, 12x Intel(R)Core(TM), Ubuntu 64bit) environment.

4.1

Database

In our experiment, we constructed a custom made databaseto make sure that the system is being able to make clear distinction between live and fake samples even when the system encounters unexpected inputs which has no direct relation with the hand as though many previous works have usedpublic live/fake dataset such as Replay-Attack Database [5], it
contains only a speciﬁc type of fake photo and video samples
making it inadequate in terms of unknown samples.
The custom made database used in the experiment consists of 8748 live samples and 6648 fake samples of palm
with an image resolution of 160x120 pixels taken directlyfrom approximately 2000 people with the camera of resolution 1280×720pixels. For training, total of 8000 samples
were selected to train the model. The test set comprised of
748 live palm samples and 6648 fake palm samples from cases
not include in the training set. The training that we are performing in this experiment is totally completely unsupervised.
Example of true samples and different variety of fake samples
that were used while training the system is given as below in
Figure 2. In order to include as many variety of unexpectedfake samples as possible to check the accuracy of the system,
we included photos such as (b)printed photo, (c)hand wearing
synthetic glove, (d) hand wearing cotton glove etc.

4.2

Evaluation Protocol

The manifold of live images was solely learned on image
data of 8000 live cases from the database with the aim tomodel the variety of live appearance. For performance evaluation in anomaly detection we ran various different protocols
exploited by researchers.

Classiﬁcation

In classiﬁcation process, we investigated the three types ofscore function, anomaly score A(x), residual score R(x), anddiscriminator score D(x), respectively. The relationship between each score is deﬁned as follows
A(xq ) = (1 − λ) · R(xq ) + λ · D(xq )(5)where the residucal score R(xq ) and discrimination score D(xq )are deﬁned by the residual loss LR (z ) and discriminator loss

4.2.1

Proposed Protocol

All the training and test conducted for the anomaly detectionin this work are based on the one class system where only thelive samples are used to develop the model. In particular, thefollowing systems are used for the development and evaluation:

111• AnoGAN+RAW: The AnoGAN which uses one classsystem trained using the original image

International Workshop on Informatics ( IWIN 2018 )(a) live(b) printed photo(c) synthetic glove(d) cotton glove

Figure 2: Example samples used for training the model for
1 class system. (a) is an example of true sample used fortraining the model and (b) to (d) are the different variety of
fake samples that were used for testing the model

Figure 3: The above ﬁgure represents the ROC graph of theAnoGAN model trained for 25 (orange) and 50 (blue) epochs
respectively by using live samples as an input image for training.

• SVM1+RAW: The one-class SVM with a Gaussian kernel trained using the original image• SVM1+LBP: The one-class SVM with a Gaussian kernel trained using the LBP feature
For each of these protocols, the model was trained using
8000 live samples and the test was conducted by using 100
fake samples along with 100 live samples which were not included in 8000 live samples that were used for training themodel in order to check the accuracy of the model in order
to distinguish the fake samples from the live samples. Residual score R(xq ) is taken into account in order to differentiate between live samples and fake samples. The purposeof this unsupervised one class training is to ﬁnd the epochs
whose training accuracy as well as prediction accuracy aregood and which does not cause overﬁtting. For this dataset
the epochs which showed the best result is 50. As we try toincrease the epochs the accuracy of the model decreased. In
this experiment the unsupervised learning was conducted bychanging the epochs as 20, 25, 50,· · · ,100. In the result section of this experiment, we used the result of 25 epochs as
the representative example and the result of 50 epochs as it
shows the best result. The residual score can vary each time
the test is conducted even if the image used for testing is thesame because the residual loss measures the visual dissimilarity between query image xq and generated image G(z ) inthe image space by ﬁnding a point z in the latent space that
corresponds to an image G(z ) that is visually most similar toquery image xq and that is located on the manifold X . We
ran 100 back-propagation steps (α = 100) for the mapping of
new images to the latent space Z.
4.2.2

Evaluation Metric

For evaluating the result obtained, we consider the Area Under Curve (AUC) obtained from Receiver Operating Characteristic (ROC) curves. The ROC curve was made using residual loss as the parameter which yields good results as shown

SystemAUC(%)

AnoGAN+RAW (25 Epoch)
AnoGAN+RAW (50 Epoch)
SVM1+RAW
SVM1+LBP

87.0
96.8
34.3
83.5

Table 1: Area under the ROC (AUC) (%) for different systems
obtained by using custom databasein [16]. Vertical axis and horizontal axis of ROC curves usually present True Positive and False Positive Rate respectively.
It indicates that the plots top left corner is the optimal point.
Preferable TPR for the ROC curve is equal to one which makes
the preferable AUC’s values approaching to one.

4.3

Evaluation Results

The one-class systems introduced earlier are evaluated on
the custom made database which used 8000 live samples todevelop the model. In order to make sure that there is no bias
in the result obtained after testing each of the model we tookout a total of 200 samples randomly from the palm database,
100 samples each from live samples and fake samples. For
the fake samples, even though the 100 images taken out wereselected at random, it was made sure that it contained all thevariety of samples that were taken into account while creating
the fake samples. By doing so, we can see to what extent thetrained model produces the desired result even if it encounters
unexpected input which is fake but has no direct relation withhand.
Figure 3 represents the ROC graph of the results obtained
from different models where the Y-axis shows True Positive
Rate and the X-axis shows False Positive Rate. Additionally, Table 1 and 2 show the AUC and HTER (Half Total Error Rate) respectively. Note that HTER can be calculated bymin(T N + F P )/2.
Table 1 shows that the best performing one-class system interms of average performance is Ano-Gan+RAW with an av-

112

International Workshop on Informatics ( IWIN 2018 )SystemHTER(%)

AnoGAN+RAW (25 Epoch)
AnoGAN+RAW (50 Epoch)
SVM1+RAW
SVM1+LBP

13
3
34
17

tionally, it should be noted that our method can detect presentation attack by using single static image. Therefore, thismethod can also be directly applied to deal with video presentation attack or be integrated with a video-based palm liveness detection method for better performance. It is left to investigate about loss function suitable for presentation attack
and reduce the number of backpropagation α to improve ourmethod more secure and convenient.

Table 2: Half Total Error Rate(HTER)(%) for different systems obtained by using custom database

Acknowledgement

erage AUC of 96.8%. The result obtained in terms of AUC byusing one class SVM system [17] as a model for training andtesting the dataset as used for training and testing AnoGAN
is also shown in Table 1 and 2. It is clearly visible that the
proposed AnoGAN system is far more better than the conventional one class SVM system. As far as the one class SVM
system are concerned, SVM1+LBP performed better as compared to SVM1+RAW. It is because as stated in [18] by using
LBP feature they were able to perform their experiment ina robust way which was computationally fast and didn’t required any user-cooperation. Moreover, the extensive experimental analysis done by them on a publicly available database
showed excellent results compared to existing works which
proves clariﬁes that SVM1+LBP will show better results ascompared to SVM1+RAW.

5

This work was partially supported by JSPS KAKENHI Grant
Number JP18K11294. We thank Eizaburo Iwata and Hiroki
Kamanaka for useful discussions. The Palm Presentaton Attack Dataset is provided by Normee co, ltd.

References[1] A Pacut and A Czajka. Aliveness Detection for IRIS
Biometrics. In Carnahan Conferences Security Technology, Proceedings 2006 40th Annual IEEE International,
pages 122–129. IEEE, 2006.
[2] Maria De Marsico, Michele Nappi, Daniel Riccio, andJean-Luc Dugelay. Moving face spooﬁng detection via
3d projective invariants. In Biometrics (ICB), 2012 5th
IAPR International Conference on, pages 73–78. IEEE,
2012.

Discussion

While we believe that an unsupervised model such as AnoGAN
could have many beneﬁts, we also see some research problems. First, we have to ﬁnd a way to determine the numberof epochs for which you have to train the system by relating it with the number of images used for training. Second,
while testing the one-class SVM method the SVM1 + LBP
produced better results as compared to SVM1 + RAW. So if
the code of AnoGAN is designed in such a way that it calculates the loss function while taking LBP (histogram, cosine
similarity) into consideration from the point of training thenthere is a chance that it might produces better result. Third,
we have only examined anomaly detection systems based on 1class SVM and AnoGAN, it would be better if we study otheranomaly detection approach also. So, it can be concluded thateven if you train the system by using the true samples only,
it does not perform well enough and more modiﬁcation andresearch should be conducted to improve the performance of
this type of system. In future experiments we would like totake the cost included in the making of the data set as wellas the time it takes to detect the fake samples from a group of
samples into consideration which would serve as a checkpointfrom where we can strive for further improvement.

6[4] Ricardo N Rodrigues, Niranjan Kamat, and Venu
Govindaraju. Evaluation of biometric spooﬁng in a multimodal system. In 2010 Fourth IEEE International
Conference on Biometrics: Theory, Applications andSystems (BTAS), pages 1–5, 2010.
[5] André Anjos and Sébastien Marcel. Counter-measures
to photo attacks in face recognition - A public databaseand a baseline. In Biometrics (IJCB), 2011 International
Joint Conference on, pages 1–7. IEEE, 2011.
[6] Tiago de Freitas Pereira, Jukka Komulainen, André Anjos, José Mario De Martino, Abdenour Hadid, Matti
Pietikäinen, and Sébastien Marcel. Face liveness detection using dynamic texture. EURASIP Journal on Imageand Video Processing, 2014(1):2, 2014.
[7] Jiangwei Li, Yunhong Wang, Tieniu Tan, and Anil K
Jain. Live face detection based on the analysis of fourier
spectra. In Biometric Technology for Human Identiﬁcation, volume 5404, pages 296–304. International Society for Optics and Photonics, 2004.

Conclusion

In this study, we investigated a face presentation attack detection method based on an anomaly detection using Generative Adversarial Network. Our remarkable result is thatthe proposed PAD scheme achieved 96.8% AUC and 3% of
HTER by using a model trained only with real samples. It
is clearly visible that our proposal can achieve far more better result than conventional one-class SVM systems. Addi-[3] Heeseung Choi, Raechoong Kang, Kyungtaek Choi, andJaihie Kim. Aliveness Detection of Fingerprints using
Multiple Static Features. In Proc. of World Academy of
Science, Engineering and Technology, pages 201–205,
2007.

113

[8] Xiaoyang Tan, Yi Li, Jun Liu, and Lin Jiang. Face liveness detection from a single image with sparse low rankbilinear discriminative model. In European Conference
on Computer Vision, pages 504–517. Springer, 2010.

International Workshop on Informatics ( IWIN 2018 )[9] Gang Pan, Lin Sun, Zhaohui Wu, and Shihong Lao.
Eyeblink-based anti-spooﬁng in face recognition froma generic webcamera. In Computer Vision, 2007. ICCV2007. IEEE 11th International Conference on, pages 1–
8. IEEE, 2007.
[10] Santosh Tirunagari, Norman Poh, David Windridge,
Aamo Iorliam, Nik Suki, and Anthony TS Ho. Detectionof face spooﬁng using visual dynamics. IEEE transactions on information forensics and security, 10(4):762–
777, 2015.
[11] D.Harwood T.Ojala, M.Pietikainen. Performance evaluation of texture measures with classiﬁcation based on
Kullback discrimination of distributions. In Image Processing. Proceedings of the 12th IAPR International
Conference on, pages vol. 1, pp. 582–585. IEEE, 1994.
[12] Wei Bao, Hong Li, Nan Li, and Wei Jiang. A livenessdetection method for face recognition based on opticalﬂow ﬁeld. In Image Analysis and Signal Processing,2009. IASP 2009. International Conference on, pages
233–236. IEEE, 2009.
[13] Ivana Chingovska and André Rabello Dos Anjos. On
the use of client identity information for face antispoofing. IEEE Transactions on Information Forensics andSecurity, 10(4):787–796, 2015.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,
Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio. Generative adversarial
nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
[15] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
arXiv:1511.06434, 2015.
[16] Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Proceedingsof the 25th International Conference of the Information
Processing in Medical Imaging IPMI 2017, Boone, NC,
USA, pages 146–157, June 2017.
[17] Alex Smola John Shawe-Taylor John Platt Bernhard Schlkopf, Robert Williamson. Support vector
method for novelty detection. In Proceedings of the
12th International Conference on Neural Information
Processing Systems, pages 582–588, 1999.
[18] Jukka Määttä, Abdenour Hadid, and Matti Pietikäinen.
Face spooﬁng detection from single images using
micro-texture analysis. In Biometrics (IJCB), 2011 international joint conference on, pages 1–7. IEEE, 2011.

114

Session 5:
Data Models( Chair: Yoshia Saito )International Workshop on Informatics ( IWIN 2018 )A Proposal of Deep Learning Training Data Generator for Image
Recognition
Tsukasa Kudo† , and Ren Takimoto††

Faculty of Informatics, Shizuoka Institute of Science and Technology, Japan
kudo.tsukasa@sist.ac.jp

Abstract - At present, image recognition technologies
utilizing the deep learning are developing rapidly, andsome cases have been reported in which their recognition
accuracy has exceeded the human vision. On the other
hand, in order to conduct deep learning, it is necessary
to accumulate a large amount of training data. And, thisoften becomes the obstacles to applying the deep learning to actual business systems. So, in this study, we propose a method to generate the images of training data
automatically by using computer graphics (CG). For example, in inventory management of parts, target objects
are composed only of inventory shelves and parts. Thatis, although it is diﬃcult to accumulate a large numberof their actual images, it is expected that these images
can be easily generated by using CG. However, while it
becomes easy to prepare a large amount of the training
data by this method, there may be a deviation between
the actual images and generated CG images. Therefore,
firstly, we evaluate the influence of this deviation on the
deep learning by using plural kinds of CG images. Next,
we conduct experiments using training data in which apart of the CG images are replaced with the actual images, and we evaluate the improvement of the accuracy
corresponding to the ratio of actual images. As a result, we show that by mixing a certain ratio of actual
images, eﬀective training data for the deep learning canbe generated with CG images.

satisfaction by image recognition utilizing deep learning,and constructed and evaluated its prototype. Here, thenumber of image data used for this evaluation is 1,600 per
part. However, in the actual factory, the parts are delivered to the assembly field from parts shelves collectively:by each product lot, namely product manufacturing unit,
or by each order composed of several products. So, thenumber of times of variation of the shelves is comparatively small. For example, in the case where its variation
occurs once a day, about 250 images are obtained a year.
That is, to accumulate 1,600 images, it takes more than
6 years. In addition, parts are stored in each inventory
shelf, and their kinds extend to thousands. And, sincemost parts are heavy, it takes large man-hours to deliberately change the situation of the inventory shelvesby hand so many times. For these reasons, to prepare
the training data eﬃciently has become the problem ofapplying the deep learning to the actual inventory satisfaction discrimination by the image recognition.
Here, each image of an inventory shelf is composedof only two types of objects: the inventory shelf itself
and the parts. And, the parts are placed in the shelfaccording to a certain rule. For example, in the caseof storing relatively small parts in a bulk container, thestate of the inventory shelf can be composed by pilingup the parts randomly from the bottom of the container.
This suggests that a large amount of various image dataof each inventory shelf can be generated by the computer
graphics (CG) automatically.

Keywords: Deep learning, Computer Grapahics, Inventory management system, Stocktaking, Convolutional
neural network

1

The motivation of this study is to show there are fields
as follows: it is diﬃcult to accumulate a large amount oftraining data composed of photographs of the actual objects (hereinafter, “actual images”) for the deep learning;but, it is easy to accumulate the training data by utilizing CG. In addition, it includes showing the method to
obtain higher image recognition accuracy in the case ofutilizing the CG images. Therefore, the final goal of thisstudy is to develop a system for image recognition of actual object by using deep learning which training datais generated by utilizing CG tool eﬃciently.In addition,
it is expected to be eﬀective in the case of the abovementioned stocktaking and so on.

INTRODUCTION

Currently, the accuracy of image recognition utilizing
the deep learning is rapidly improving, and the case ofachieving accuracy exceeding human vision has also beenreported [4], [6], [14]. As a result, its applications are
rapidly spreading to various fields [3], [5]. On the other
hand, since a large amount of training data is necessary
in the deep learning, a large amount of image data has tobe accumulated for image recognition, too. Therefore, in
several application fields, there is a problem that it takes
a long time to accumulate such an image data.
For example, in the manufacturing factory of mechanical products which our laboratory supports its production management system improvement, the stocktakingof parts inventory is a heavy workload. So, we proposeda method to automatically discriminate the inventory

In this paper, we perform a feasibility study to develop
such a system, namely the evaluation of the image recognition accuracy in the case of using CG data for the deeplearning. Concretely, presupposing the inventory shelf,
we classify the number of stored parts from 5 to 80 every
5 and evaluate the accuracy of their estimated quantities
117International Workshop on Informatics ( IWIN 2018 )Production
management
systemOrder
pointsInventory
display
featureInventory
shelf
imagesParts
order form
Inventory administrator

Figure 1: Inventory shelves of bulk container

Figure 2: Inventory management utilizing imagesby using the supervised learning.
On the other hand, the creation of high-precision CG
image similar to the actual images also requires a large
number of man-hours. Conversely, the actual images canbe obtained even in the case of the above-mentioned inventory shelves in the factory, if the number of the images
is small.
So, we conduct this evaluation in the following twostages. Firstly, we extract the factors in the CG image
creation and evaluate the influence of each factor on theaccuracy of the image recognition. Next, we evaluate thechange of this accuracy in the case where the CG images
are replaced with the actual image by each ratio. As aresult, we show that by replacing just a part of the CG
images with the actual images, it is possible to achieve
the same accuracy in the case of using the training data
constructed of only the actual images.
The remainder of this paper is organized as follows.
Section 2 shows the related works and the problem tocreate the training data, and we propose the training
data generation method with CG for the stocktaking in
Section 3. Section 4 shows the implementation of thismethod, and Section 5 shows the experiments and evaluations with the training data generated by this method.
We discuss the evaluation results in Section 6, and conclude this paper in Section 7.

On the other hand, with the progress of the Internet ofThings (IoT), various sensors such as surveillance cameras are controlled remotely, and their data is accumulated and analyzed in the server. As a result, since so various and enormous data has been stored in the database,it has become diﬃcult to deal with such a data withconventional relational databases. So, various NoSQL
databases have been put to practical use to manipulate
such a data eﬃciently [11]. For example, MongoDB isa kind of document-oriented NoSQL databases and provides the GridFS interface to manipulate such a dataeﬃciently in the distributed environment [1]. That is,
at present, an environment has been developed, in whichlarge capacity of image and video data can be easily handled.
Due to such a technical background, we proposed aninventory satisfaction discrimination method using theinventory shelf images shown in Fig. 2 [9]. Here, the ordering point quantity is determined for each part in theinventory shelves by the production management system,
and inventory is replenished when its inventory quantity
falls below this ordering point [12]. Therefore, to discriminate visually that the inventory of each part satisfies this ordering point quantity is more eﬃcient thanperforming the stocktaking of the inventory. For example, in the case of the inventory shelf on the upper rightof Fig. 1, while it is diﬃcult to grasp its exact quantity,
it is relatively easy to discriminate that there are ten or
more parts.
And, in the case where the discrimination is diﬃcult,by replenishing the part from the viewpoint of safety, it
becomes not necessary to count the inventory quantity.
As a result, we showed that the eﬃciency of the inventory management could be achieved by showing both the
production plan data and current images of the inventory
shelves to the inventory administrator as shown in Fig.
2. However, even by this method, some problems about
the workload of the inventory manager remain: he must
check many inventory shelves one by one; especially, inthe case where a large number of parts are stored in thebulk container, it takes time for the discrimination.
On the other hand, currently, the accuracy of image

2

RELATED WORKS AND
PROBLEMIn this section, we explain the background and related
works of our idea described in this paper. Our laboratory
supports the factory to introduce and operate its production management system, which manufactures mechanical products. Since thousands of parts in various
shapes are stored in each inventory shelf in the factory,
the workload required for stock-taking of inventory is aserious problem. In particular, in the case where parts
are stored in the bulk container as shown in Fig. 1, theycannot be counted from the outside. So, it is necessary
to take out the parts in the container and count themup. So, it is a major factor increasing man-hours.
118

International Workshop on Informatics ( IWIN 2018 )timation in this deep CNN by using the test data ofimages prepared separately. Firstly, we conducted thecomparative evaluations between the deep CNN and human vision by using marbles. As for human vision, we
showed the images of the training data of each class to
let the humans learn its quantity. Then, the humans
were shown the test image for 10 seconds to answer its
quantity. As a result, we found that although the human vision was able to accurately estimate the quantity
in the case where it was small, the deep CNN’s accuracy was higher in the case where the quantity is equal
or more than 20. This experiment corresponded to theinventory shelf of the bulk container shown in Fig. 2.
And, as shown the image of 60 in Fig. 3, it was diﬃcult
for humans to estimate the quantity in a short time inthe case of many objects.
Furthermore, we evaluated the distribution of the estimated quantity. And, we confirmed that there was thetendency that they are gathered in the vicinity of theactual quantity, although some of them may disperserough. As a result, we concluded that it could be applied to actual inventory management systems by taking
the following countermeasures: the safety stock should
be increase to permit the error; the estimated quantity
should be compared with the logical inventory quantity
calculated by the production management system to detect the error.
Here, in this experiment, as we treated small marbles
and nuts shown in Fig. 3, it was easy to obtain a large
amount of photo to create the training data by shuﬄing
these objects in the bowl at every time. However, in
the actual factory, the bulky and heavy parts shown in
Fig. 1 are treated. In addition, since it is in operation,
we cannot hinder the workers. That is, there is a problem that it is diﬃcult to accumulate a large amount oftraining data for applying the deep learning.
Generally, to apply the deep learning, accumulating alarge amount of training data is an important factor to
improve its performance, and training data is collectedin various ways in each application field. On the other
hand, there are application fields where collecting thetraining data is diﬃcult like this case. Therefore, it
is considered eﬀective to develop an eﬃcient preparing
method of the training data in such a field.

(1) Picture of marbles (5, 25, 60)(2) Picture of nuts (5, 25, 60)
Figure 3: Picture example of experimental objects
recognition is rapidly improving by utilizing deep learning. For example, in the ImageNet Large Scale Visual
Recognition Challenge (ILSVRC), algorithms compete
for object detection and image classification of largescale. And, after the deep learning was applied in 2012,
the recognition rate has been greatly improved. Moreover, it exceeded 5.1% of human recognition rate in 2015[14]. Along with the improvement in recognition rate,
the application of deep learning to image recognition is
spreading in various fields such as face authentication,
medical image diagnosis, plant disease detection, and soon [3], [5], [10].
So, we conceived to apply the deep learning to the inventory satisfaction discrimination utilizing image recognition. And, as a feasibility study, we conducted theexperiment to recognize the classified quantity of the objects by the supervised deep learning. Then, we constructed the deep convolution neural network (deep CNN)
and performed the deep learning; we evaluated its accuracy of quantity estimation by using the test data of images prepared separately[7], [8]. For the target objects,
we used the marbles and nuts, which is easy to create
training data. And, we classified each of them from 5 to
80 every 5 with the label of their quantity.
Figures 3 shows image examples of training data in thecase of 5, 25 and 60. We prepared 100 image data foreach class, that is, the total is 1,600 for each of marble
and nut. Then, we padded 400 images by up/down andright/left inversion, and 90 and 270 degree rotation. As
a result, we created 500 image data for each class, that is,
the total was 8,000 for each of marble and nut. Next, we
separated them into 6,000 training data and 2,000 test
data, then we separated 600 data from the training dataas the verification data. Ultimately, with 5,400 training
data, we conducted supervised deep learning with theabove-mentioned labels. Fig. 4 shows the compositionof deep CNN for this deep learning.
After that, we evaluated the accuracy of quantity es-

3

PROPOSAL OF TRAINING DATAGENERATION METHODUTILIZING CG

As a solution to the problem of the fields where theaccumulation of a large amount of training data is diﬃcult, we propose a training data generation method byutilizing CG. Considering the characteristics of the inventory shelf, usually only one kind of parts is stored in
one shelf. That is, from the viewpoint of CG, it is possible to construct the state of the inventory shelf with
only two objects, one inventory shelf and one part. Furthermore, it is not necessary to consider the deformation
119International Workshop on Informatics ( IWIN 2018 )2XWSXW)XOO\FRQQHFWHG'URSRXW)XOO\FRQQHFWHG'URSRXW)XOO\FRQQHFWHG)ODWWHQ

3RROLQJ&RQYROXWLRQDO&RQYROXWLRQDO

3RROLQJ&RQYROXWLRQDO&RQYROXWLRQDO

3RROLQJ&RQYROXWLRQDO,QSXWFigure 4: Construction of deep convolutional neural networkof each object. That is, by piling up the part objects
randomly in the inventory shelf object, the state of theinventory shelf can be constructed virtually.
Basically, to construct such an inventory shelf by CG,
the four factors should be considered: the shape of theobjects, the material of the objects, the illumination as
the environment, and the placement of the part objects.
Firstly, the shapes of objects can be realistically createdby using CG modeling tools, due to the measuring dataof the actual inventory shelf and part. Secondly, the material expresses the textures of these objects, and it canbe also added by the CG modeling tools. However, it isnecessary to adjust the material by not only human vision, but also checking the influence to the deep learning.
Thirdly, the illumination needs to be determined basedon the actual factory illumination environment, and it
can be added in the same way as the material. These
three factors are static with respect to each inventory
shelf. So, it can be used repeatedly after once created.
Fourthly, the placement of the part objects is the mostimportant factor to create a large amount of training
data, and the following three requirements should beconsidered. The first is a physical requirement, for example, it is necessary that the placement in CG does notcollapse even if it is actually placed. The second is therealization of random placement, that is, in order to accumulate a large number of training data, it is necessary
to construct diﬀerent placement states even in the caseof the same number of the same parts. The third is therule of placement. For example, the parts in the second
container from the left of the uppermost shelf of figure 1
are the one like plates, and they piled for every 4.
Regarding such a placement, various kinds of CG tools
are provided. And, many of them can be controlled
by the programming language automatically and provide the feature of physical simulation such as free-fallby gravity. Therefore, it is possible to create the various
placement state similar to the real world by executing
the physics simulation after randomly placing parts byusing programming languages.
For example, we show the case to generate the marble
training data shown in Fig. 3 by CG tool. Firstly, as
shown in (a) of Fig. 5, we create the bowl and marble
objects, then place the marble objects randomly above
the bowl. Next, the marble objects are dropped in the
direction of the arrow shown in (a) of Fig. 5 by using thephysical simulation of free-fall by gravity. After falling in
the bowl, it converges to the natural state shown in (b)(a) Placement of marbles(b) After physical simulation

Figure 5: Image example of experimental objects!EOHQGHUEDFNJURXQGS\WKRQ0DUEOHBEDVLFS\

Figure 6: Batch file to control Blender by Python
by the control of the physical simulation. And, by saving the rendering image of this result, the image shown in
Fig. 3 can be obtained. By repeating these processes automatically by using the programming language, various
large amount of training data can be created.
As described above, it is expected that a large amountof training data can be generated eﬃciently by using CG
tools in the following case: the target images are composed of a small number of objects; and, a large amountof diﬀerent image data can be created according to acertain procedure. As a result, it is considered that thetraining data can be generated with CG images as substitute for actual images in a certain field, where it isdiﬃcult to accumulate the training data by actual images.

4

IMPLEMENTATION OF
TRAINING DATA GENERATORIn this study, we used 3DCD creation software Blender
2.79 [2] to generate the training data. Blender can becontrolled by Python script, and Python 3.5.3 is shippedwith the above-mentioned Blender. So, after we place
the necessary objects and lights in Blender, the arbitrary number of training data can be generated automatically by using Python program according to the processes shown in Section 3. We created Blender objects
and Python programs separately and executed them withthe batch file shown in Figure 6.
(a) of Fig. 7 shows the placement of objects in Blender,120

International Workshop on Informatics ( IWIN 2018 )Table 1: Experiment cases on marbles(a) Initial placement of objects in Blender(b) Camera view

No

Case

Position

Illumination

Material

(1)
(2)
(3)
(4)
(5)

Rough
Position
Illumination
Matrrial
All

⃝
⃝
⃝
⃝

(0) Real(1) Rough(2) Position(3) Illumination(4) Material(5) All⃝⃝⃝⃝Figure 7: Initial placement of objects in Blenderwhich is composed of a bowl, four marbles, and a square
tray. In this experiment, same as the previous study, weused four types of marbles. So, we placed each one under
the tray. And, the tray is used to detect the spillage of
marble objects from the bowl object. That is since there
is a possibility that the marble objects spill out in thephysical simulation in the case where the number of themis large, such an image must be excluded. (b) of Fig. 7
shows the view from the camera for the rendering. The
white solid quadrangle is the outline of the tray, and thelower right is the lowest position. So, the spilled marble
objects gather here. And, the spillage can be detectedby comparing the image before and after the simulation
with trimming the bowl object. Incidentally, as for theillumination, we placed Hemi lamp, which lights up thewhole area equally, and a directional lamp on the side of
the camera; the range between white dashed lines in (b)of Fig. 7 was the rendering area.

Figure 8: Images of marbles by photography and CG

5

EXPERIMENTS ANDEVALUATIONSIn this experiment, firstly we evaluate each factor of
creating CG images from the viewpoint of the influenceon the accuracy of image recognition. Next, we evaluate the change of this accuracy according to the ratio of
replacing a part of the CG images with the actual images. Firstly, images with spilled marble objects were
excluded, then the outside of the bowl object in the remained images was trimmed to create images similar to
Fig. 3.

The procedure to generate the images for the training data by using the batch file in Fig. 6 was as follows. Firstly, the Blender file was opened and the marble objects in (a) of Fig. 7 were randomly selected, then
their copies were placed hierarchically as shown in (a) ofFig. 5. Here, to shorten the falling time, each hierarchy was divided into four quadrants, and marble objects
were placed randomly within the range of each quadrant.
Next, the state of (b) in Fig. 5 was generated by physicalsimulation. Then, rendering was performed by the camera placed right above the bowl object as shown in (b) ofFig. 7, and its image was saved in a file. After that, theBlender file was reopened to return to the initial state,then the same procedure was repeated.

5.1

Evaluations of CG images formarbles

Table 1 shows the factors of creating the CG image,which is evaluated in these experiments: (2) Position, (3)Illumination and (4) Material. We evaluated the influence of the improvement of each factor, and the case of
combining them. Incidentally, in Table 1, (1) Rough isthe state before improvement, and (5) All is the combination case. Figure 8 shows the examples of the imagesof the training data of 30 marbles: (0) Real shows theactual image and the others show the CG images createdwith each case of Table 1. Since marble objects were randomly placed in Blender in each case, the placements ofthe marbles were not the same.
In (1) Rough of Fig. 8, since bowl modeling accuracy was low, the placement of marbles expanded andAfter all the image data generated, they are converted
to the training data by another Python program by thefollowing procedures. Firstly, images with spilled marble objects were excluded, then the outside of the bowl
object in the remained images was trimmed to create images similar to Fig. 3. Then, the designated number ofimages were saved as the training data. After that, by
the same procedure as the previous experiment as shownin Section 2, the images were padded 4 times to create
the final training data.
121

International Workshop on Informatics ( IWIN 2018 )Remarks) Pre.: Prediction Quantity;
Err.: Error from correct quantity
Pre.

Standard deviation of prediction error

Case number

Err.
(0) Real

Figure 9: Change in standard deviation of errors
Pre.

there were more gaps between marbles than the one in(0) Real. So, in (2) Position, we improved the shape ofthe bowl, then the placement of marbles became closer
to (0). Next, since the actual images were taken in aroom where multiple fluorescent lamps were installed on
the ceiling, we placed multiple lamps in Blender to makethe CG images close to the actual environment in (3)Illumination. In (4) Material, we improved only the material of the marbles from (2). In Blender, since it wasnecessary to change the rendering engine from “Blenderrendering” to “Cycles rendering” in order to produce themarble material, the brightness of the whole image waschanged in (4). In (5) All, we added the lamps of (3) tothe CG image of (4).
Next, we trained the deep CNN shown in Fig. 4 bythese training data. And, we prepared the test data composed of 50 actual images for each case of 5, 20, 40, 60
and 80 marbles. Then, we obtained the estimated quantity of each test data by the deep CNN. Figure 9 shows
the standard deviation of the estimate error of each caseof Table 1. This error is the diﬀerence between the actual quantity in each image and the estimated quantity
using the deep CNN. In this case, since we classified thenumber of marbles from 5 to 80 every 5 and trained by
the supervised learning, the error also changed in unitsof 5.
Here, “Case number” corresponds to “No.” in the Table 1. Although the standard deviation of (0) Real was
5.3, the one of (1) Rough worsened to 73.7. And, it was
bettered to 20.1 by improving the position as shown in(2). On the other hand, each improvement of (3) Illumination and (4) Material worsened rather in the caseof executing separately. However, in the case of applying both shown in (5) All, it bettered to 13.8, that is, it
was necessary to improve the illumination and material
together.
Also, Fig. 10 shows the distribution of errors of estimated quantities with respect to each actual number ofmarbles in the case of (0) Real and (5) All. In (0), the error is distributed in the range of ±10 around the correct“0”; while in (5), many peaks of distribution appearedErr.
(5) AllFigure 10: Distribution of errors of estimated quantities
before and after “0”. However, even in the latter case,though the error “over” namely 15 or more was occurred
about 20% in the case of 40 marbles, the error was within
the range of roughly ±10 in the case of other numbers.

5.2

Evaluations of CG and actual mixed
imagesNext, we evaluated the change of the accuracy in thecase where the CG images are replaced with the actual
images by each ratio. We used CG images of (5) All in
Table 1 in this experiment, and the designated numberof images from the end were replaced with the actual images. For example, in the case of “5%”, we prepared 500
images using 425 CG images and 75 actual images. In
addition, the images for evaluation are the actual images
same as Section 5.1.
Figure. 11 shows the change of the standard deviationof errors according to the mixed ratio of the actual datafrom 5% to 20% for every 5%. The standard deviation
improved to about 2/3 when 5% of the CG images were
replaced with the actual images, which was about 55%
improvement as for the diﬀerence between (5) and (1).
Similarly, as of 10%, it improved to about half, which
was about 80% as for the diﬀerence between (5) and (1).
Furthermore, as of 20%, the standard deviation became
122International Workshop on Informatics ( IWIN 2018 )Standard deviation of prediction error

As a result, we found that even the fields where itis diﬃcult to accumulate a large number of actual images for the deep learning, there is the case which target
is composed of the simple objects from the viewpointof CG. In such a case, we consider that to use the CG
images generated automatically for the training data iseﬀective.
However, in this study, we have verified the eﬀectiveness of the proposed method by using only the marbles
at just the laboratory. So, the verification in the actual factory remains as the future study: the first is theverification with the actual parts and in the factory environment; the second is the development of the method
to collect the actual images eﬃciently in the factory.

Mixing ratio of actual data
Figure 11: Standard deviation of errors on mixing ratio

7

almost the same as of the actual images.
That is, in the case of replacing the CG images withthe actual images, the improvement of the accuracy was
relatively larger than the replacement ratio.

Currently, the accuracy of image recognition utilizing
the deep learning is rapidly improving, and such an image recognition is applied to various fields. On the other
hand, in order to improve the accuracy by the deep learning, it is necessary to accumulate a large amount of training data. And, the preparation of the training data often
becomes the obstacle to apply the deep learning.
For this problem, we proposed a method to generate
the training data automatically with CG in this study.
And, we confirmed that the training data can be generated by using the CG tool automatically; a certain
degree of the accuracy could be achieved by using sucha training data. Furthermore, we showed that the same
recognition accuracy could be obtained in both cases: in
one case, the training data was created by a large numberof the actual images; in another case, the training data
was created by a large number of the CG images and lessof the actual images. As a result, we found it is eﬀective
to generate the training data by using the CG images,
in some fields where it is diﬃcult to accumulate a large
number of the actual images for the deep learning.
Future studies will focus on the confirmation that thismethod can be used generally such as various materials
and environments.

6

DISCUSSION

In this study, we evaluated the accuracy in the caseof using the CG images instead of the actual images inthe training data of the deep learning. As a result, asshown in Fig. 10, we found that a certain degree of accuracy could be achieved, while the errors spread larger
compared to the case of the actual images. On the other
hand, as shown in the relation between the CG images ofFig. 8 and the standard deviations of the error in Fig. 9,
the accuracy was greatly aﬀected by the factors to create
the CG images.
In other words, it is expected that the accuracy can be
improved by refining the CG image shown in (5) of Fig.
8 to make it closer to the actual images shown in (0).
However, the more we refine the CG images to look likethe actual images, the more the workload becomes big.
That is, it is necessary to repeat the several works accompanied by trial and error: the first is the adjustmentsof the CG images such as material and illumination; thesecond is the deep learning of the deep CNN. And sinceeach takes a long time, the ratio of improvement and cost
is expected to decline.
Therefore, we showed that the accuracy can be improved eﬃciently by using the training data, in which apart of the CG images were replaced with the actual images, as shown in Fig. 11. That is, we found that in thecase where the training data was generated with the CG
images are used, it is eﬀective to prepare the actual images in possible and to use them as a part of the training
data. In addition, at the factory like the target of thisstudy, the operation to estimate the inventory quantities
by using the deep learning as follows is possible: initially,
the training data is composed of the CG images and a
few actual images; while increasing the actual images,
the training of the deep CNN is repeated by using theseimages and the accuracy can be gradually improved.

CONCLUSION

Acknowledgments
In this study, we utilized the results of graduation research of Mr. Tenma Kawanaka for the actual data ofmarbles and the experimental environment of the deeplearning.

REFERENCES[1] K. Banker, “MongoDB in Action,” Manning Pubns
Co. (2011).
[2] Blender
Foundation,“Blender,”https://www.blender.org/ (reﬀered, May 2018).
[3] T. H. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, and Y.
Ma, “PCANet: A simple deep learning baseline forimage classification?,” IEEE Transactions on Image
Processing, Vol. 24, No. 12, pp. 5017–5032 (2015).
123

International Workshop on Informatics ( IWIN 2018 )[4] D. Ciregan, U. Meier, and J. Schmidhuber, “Multicolumn deep neural networks for image classification,” 2012 IEEE conference on computer vision andpattern recognition (CVPR), pp. 3642–3649 (2012).
[5] H. Greenspan, B. van Ginneken, and R. M. Summers, “Guest editorial deep learning in medical
imaging: Overview and future promise of an exciting new technique,” IEEE Transactions on Medical
Imaging, Vol. 35, No. 5, pp. 1153–1159 (2016).
[6] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, “Deep learning,” MIT press (2016).
[7] T. Kawanaka, and T. Kudo, “Proposal of Required
Amount Satisfaction Determination Method Using
Deep Learning,” Proc. 80th national convention of
IPSJ, pp. 4-543–4-544 (2018) (in Japanese).
[8] T. Kawanaka, and T. Kudo, “Inventory Satisfaction
Discrimination Method Utilizing Images and Deep
Learning,” Procedia Computer Science (2018) (inpress).
[9] T. Kudo, Y. Ito, and Y. Serizawa, “An Application of MongoDB to Enterprise System Manipulating Enormous Data,” Int. J. Informatics Society,Vol. 9, No. 3, pp. 97–108 (2017).
[10] S. P. Mohanty, D. P. Hughes, and M. Salathé, “Using deep learning for image-based plant disease detection,” Frontiers in plant science, Vol. 7, Art. 1419
(2016).
[11] E. Redmond, and J.R. Wilson, “Seven Databases inSeven Weeks: A guide to Modern Databases and theNoSQL Movement,” Pragmatic Bookshelf (2012).
[12] S. P. Singh, “Production and Operation Management,” Vikas Publishing House Pvt Ltd (2014).
[13] E. A. Silver, D. F. Pyke, and R. Peterson, “Inventory management and production planning andscheduling,” John Wiley & Sons (1998).
[14] M. Verhelst, and B. Moons, “Embedded Deep Neural Network Processing: Algorithmic and Processor
Techniques Bring Deep Learning to IoT and Edge
Devices,” IEEE Solid-State Circuits Magazine, Vol.
9, No. 4, pp. 55–65 (2017).

124

International Workshop on Informatics ( IWIN 2018 )                     † †    ‡  

‡ †   †             



      4        6  -        8       9  7   

Abstract                                              
  ! "                     "
          #     "     #     "                              $#%#  $&        ! "  "    '      (    )  * "           !    "    #     "  +     )   '     "   
  $& #  +     !     !            

                
!  +              "        "
     "!           )   

  1:    #'    "                                                    ! "                     "          
 $ # +       ,        + -  ! "   "   "        # 
 "  )        )  )       "  +  )   &    !    )       )  ( '.*   "  /               
                 " )     .'    "                     )   + 012 032 &    -      $& ( " $   &! *042
$&       )   &  5 062
  "  $& 5        )
    '   #        7  "   3  '.

 1  7       '. $  '   #  '. '   . )  ('#*   "     f   f  -  & "   !
     n f (n) = f  (n) 
      "        7   (
 1 "  6;;;; *   !   -     $ "                 "   

   !

# " "   "! " 
  #082   



'

    

# #         " ∀f   ∀n ∈ Z|n| , " f (n) "       

 <     !       " ∀f, f    ∀n ∈ Z|n| , " f (n) =f (n) "       
 "         <     !
     #     
!    
125

International Workshop on Informatics ( IWIN 2018 )      !    
|n|∀f, f    ∀n ∈ Zt , " f (n) = f  (n) "          f   f       n ∈ Z|n|t , " Zt   "   t     )-  t #  7  Zt  2t  #    7  Z|n|
  t2t|n|  #      "  !      |n|f   f   "      bmn ∈ Zt . #"      f (n)   f  (n) )  
α(n)   α (n)        "    f (n) = f  (n) "  )   2t|n| ·max(α(n), α(n)).

 3: #    $&     

 '. " !   -  ∀n : f (n) = f (n)"     f   f   -       '.
=   n           43        !   )  # 4 "       f    f  "     -     )   !
 $#%# 092>0132
$&   5 062        )   !   

!   -     !  $& 
!                   '        !"•      $,'01C2     
$#%#  D4092 E 0C2   '.'60F2•        "         301F2    )  -     "# "- "  )    "      "       ? 1 3   4

$& ( " $   &! *042   <         "   )  " 
'          ??.
 . ( .   *       )   + 0142>0182  .   ??.     $ ??. )      ' '@@  AB ' ) ??.          (      *         7               $    "            7    ??.     " "        '     
$&  +   ! " " '      ??.  ,   -   +   !
                 $& $&   
 !  $&             '    $   3 "     $&# "   7    $&

? 








}

∗



}





    





{






1: #"   









<<  

{? 1  "           (*         (*          (* B         " "        (*             1   #                          ?  3   4      ? 3:. ) ( -*•    "  )     $& "    !           •     )  !     

 







<−        ) )  -  -   <−   




126






















   






!





{|

 




|}







"









"

  #





"

   #










 





 #















" 



International Workshop on Informatics ( IWIN 2018 )

? 

<−















 







 







 & !

 

'



 





( 

























)









"

  #"   #

 #



)


































<−          <−    

























.

%



& !









 

'





 

 









 %








$





 







"



 









/

? 

 



{{





""

}} 



 



0








 

0

    ? 4:. ) (    *

 



" 

{

0













 {



"



<−       



















 







 

 







? 













 



%

? 

 







6: ($&: -* 

'



 





( 
 

 



) 


























 

'



 

 







+







( 



,











(








}



∗

 



 


 

}


""

? 



   




"



 



'

  234 



#

6/







 

! % 7 % −7%%??.  













  










   <<  

{





}



   



 #

{

 










  







{

"



  



"





"

  









<<  



}127

5 

K: . )     "   ( *          K "  -   )  

? 

/



{ ∗$          



0

 %

         
 

."}           −−−−−−−−−−

5

• ?! 








9: &

)

0





     #          L  ' 03;2  G. H   G I J
      H "   "           -         ?   9 C   F "    '.   -? •            " !

-

0



5 062   ) '    
5   "



8: ($&:    *

& !



! 



# "$

*

? 



+








! 



( 

 

!

  $   $&   7  ??.    -   #    -    $< ($   <* 0192 $<     )   
  $,'01C2 $,'      +   ! 01K2 " "      $< $,'       $&

}}



 6   8 "    







 !#$



 

& !

)





{{  << 



}


-   " x +  10     
$&  "   - #          $&   !

   #/<−     

%

1

−−−−−−−−−−4  


F: ($&:    *0

}



! 



<−





 








!

 



( 

"C: ($&: -*    

""



∗ 



   

{International Workshop on Informatics ( IWIN 2018 )

%+  

5                 (*     ?  1; "      )  
? 
&





−−−


   −−− 0112         $#             $#     
   3;;;   , $&                
' B   =  ('= *  $<0192  )


1;: ( *−−  !−    



)

 %

 

 - % - 

 

 %

#   "   $   "         
 "            #                 "           )- )        )        
     "  )      "                    " 
-           )  A        )      7     

=   5 !   )   
"             
5 

% "&" '('   )   $# ($# )   *              
  )           
"       B    ,            B    
#         )    )
  )     ! "   = 5" $#   )       
# ( )    # *   -   
$#  ,       +         ! "  #            "   )   

#,   )           )-  &      7     #                  !  -           & )-   7
    n &      )       "   7 n   4
# "      )      - "  1;;  n # "    )  "   7 1  1;;           )   "              7    #       "        n   "     )       n       "  )-  n "       "   7 1  n " n        
 6 "       "   7  4   "      )     &     "  !       $& &          

% &D4092      #        #'A  #       
-             #  $&

% $$%'.'60F2      '.' (' .   '!*   M"      .' 
        #'A 3;1C  "         % ' E 0C2   #        "  
 E  3   3;;K    -   "  #'A 3;1C         #   $&%%) *# ! "+,-!'&   -     "( )*!

# $#01;2   #        " ' @@   $&             )    3

• #" • ?    "    )  
128  

International Workshop on Informatics ( IWIN 2018 )





 {
−>    , " 9='' {−>   8        } {
−>   }  

  

}







−>     #    "                     #

/ #       43          (  C*?  14 "        4: ,   '!
#  
4 "   )       )   7     "  " 
4 "  )               7

? 






• ,   "    )        " "       )  

}

{

3>(<<




 

∗
∗



3>(<<

  



3>(<<









,)-  

? 




}

 



 

} 

11: #" 

{





}

{



  



 




∗  {−>    8 −> 





%



∗  {
−> 8  −>      



 

}



∗

 





9:;<



∗







 

  

 




8

  {−>      8−>       .

 

 ??.: 4F; D4: 68;
 E : 383
  : 33; # $#: 31;



 

 {
−>    "" 9='' {−>   } {
−>   8        
}



}



 

/• $&: ;3 (3;1F;141*

13: ?   ? 

 

9:;<

3>(<<

{

−>      : 1F;1; : 6;K9 ,
 #   ': 3

{



∗



9=''

8



• : F;; <,

9:;<



8


 

−>  −>      • L!

? 



""

  −> • ':    C68;; ' J 1F;<57 34K<57#       43    
     (  9*
?  13 "       




• A: & " 1; 96   .   . 

}  {#        7  "#    "                  & B           

 

#    "                   #         


}





 





}

 


}





3>(<<

{

/

  



∗



9=''







""



 













 








 







}



 




  #       43        (  8*
?  11 "   

14: ,  #• 5 : ;1;4−>     

 ??.: 49;
129

International Workshop on Informatics ( IWIN 2018 ) 6:    ,  # 8: #"  

 9: ?  ? %/'  " "('0 .

&   )     '         GA  )  $#%# H  $& $,' "    !         $#%#  "    !   -% C: ,  ## 4  7  )   ,  #   8"   $#%# 

/'  " * "$0 1/&   )     '    $&  5  &    !        % $   ,  !"1  

# 6  7  )   #"  "  
$#%# 
# 8  7  )   ?  ?    7 1;;"   $#%# 
# 9  7  )   ,  #   8"   $#%# 

%# $  " "   #%A  )   )      49;;   N>/ "     )         % $    ! 0 ( -

2 " ""# 1  7  )   #"  "  
$#%# 
# 3  7  )   ?  ?    7 1;;"   #%$# &  )      )        "       &   "          

130

International Workshop on Informatics ( IWIN 2018 )# 1:   #" #%$#'  

$,'
1;9

D4
134

E 
16C

 
136

# 4:   ,  ## $##%A

#%$#'  # 3:   ?  ? #%$#'  

$,'
16C

D4
>

E 
>

 
>



D4
>

E 
>

 
>

# $#># 6:   #"  ($&   5 *# $#>. )  #' #      5      "  "              )   -1     3  "   
G  H 5"        "      "  $#%#  A
     )   " "  G    H#    # $#       )     # $# -3  "      ?  ?   " 1;;   & "   "              "  8;;;     )      )    )      "             1;;;     )      #      )                            ?      )    )         (    *   )   5    )    #"          $&          $&  "    )    5" "         "             !         #              $        $&            5    

3

$,'
#%A

   
012

$&
1;9

5
;1;6"

 '! $ ,      E D: G,  '!   )    H  
   .1K  1 C46 (3;13*032 # ?   =     #    G,   . )    # : $ ' H     8      '     " #  . )    .    1;111;(3;13*
042  L!  $ 7  5  - , 5   L =   $#: G'        "    " $   &! H   .## 3;19 (3;19*
062 $ <) ! # O  $ O     $
=: G# 5 . )  "!H     '    ' $  . )  464491 (3;18*
082   5   "     L  : G    $ # ?     '
  (3     *H $  & (3;;;*092 ?     = ,BP : GD4: $  )   #H    #$'$ 3;;F ?=' . 6K94
44C46; (3;;F*
0C2 , L: GE  33H    '$.3;16?=' .F88K C4CC66 (3;16*0F2 ' , ' ? ' "  L ? 5 L   Q  # O  $     ' #  :G'.'6H     34         '    )  ('$./11*  1C1
1CC (3;11*0K2 $ '   $ <  ,      
  : G# $#8 # H   #$'$ 3;14 ?=' . CCK8 K41;C (3;14*
01;2  '7   < O        $: G# $#: $ A  '@@ #      #  H         '    #   $      )   # ($# 3;18* 49;49F(3;18*
0112 =   $   !   = R  : G$?        $#H    1;      '    #         )     3C33F9 (3;;C*0132 $ ,   5 5 .     # &:G5 !   )  H A  (3;;K*"#     "    '  . )       !
  "      &    -      $&  "    
"! "       "   "        $  "! " "   "           -  

45*6!
       '    ! "  #  "    ! O      " -  
131

International Workshop on Informatics ( IWIN 2018 )# 8:   ?  ?  ($&   5 *. )  #'  

$&
16C

5
># 9:   ,  # ($&   5 *. )  #'  

$&#%A

5
>0142 O A!   5  # ! 7"  A    =!B : G +   '!    S#"  # L    SH    39      '    ' '      ="! 19 ('''= 3;1C* ($ 3;1C*
0162 ' , ?    ?   =!
B       : G . )  '         H    18     &!  $ . )  '    $.'/18 (3;18*0182   ?   =!B : G$  $  $   ? 7      H       
 .36 =1 FFKF (3;19*0192 $ L  & 5    ' ? ,   ? #  : G?            H ,       L .38 (6* 3C33F; (1KF1*
01C2  ,   $   !: G$,': $ $     . )  #H ?=' .91C6
366; (3;1;*
01F2 ' , $    ' #  : G# #?,  .  3;H (3;1;*
01K2 $ O  .    O    O <  :G    +   !
        )  H #  
 '$L 31 (13* 14CC14K6 (3;;3*03;2 , : GABA    " '        H    5 (1KKC*

132

International Workshop on Informatics ( IWIN 2018 )Occurring Frequency for Selecting Optimal Window Length in Motif Discovery
Makoto Imamura*, Mao Inoue*, Tadao Yagou*, Daniel Nikovski****

School of Information and Telecommunication Engineering, Tokai University, Japan**
Mitsubishi Electric Research Laboratories, USA
imamura@tsc.u-tokai.ac.jp

Abstract - Repetitive patterns in time series obtained fromsensors attached to humans or machines often show typicalbehavior, and finding such repetitive patterns is useful invarious domains such as smart factory, health care, andseismology. Motif discovery is not only a fundamental
method for finding repetitive subsequences in a longer timeseries, but is also used as a sub-routine in higher-level
analytics including classification, clustering, visualization,and rule-discovery. However, existing motif discovery
algorithms depend critically on knowledge of the correct
subsequence length. Therefore, deciding on a suitable
subsequence length is necessary before using thosealgorithms. In this work, we investigate criteria how suitablea subsequence is for a motif in order to select top K motifs
among subsequences with different window lengths. We
propose a new index ‘occurring frequency’ that counts thenumber of similar subsequence occurrences with removing
trivial matching subsequences. We also show normalized
motif occurring frequency, which we call ‘motif coverage’in this paper, can decide optimal window lengths of motifs
in simulated time series containing motifs with different
window lengths.

subsequences in the neighborhood of a motif. Furthermore,
motif discovery algorithms expect that a subsequent length
be chosen beforehand, which usually means in practice thatusers must try several possible lengths, and must confirm
that the discovered motif indeed has frequent similarsubsequences in a time series.
In this work, we introduce a new index ‘occurring
frequency’ to measure the number of subsequences within agiven range from the motif. We will further define a novel
index ‘motif coverage’ by normalizing occurring frequency
with respect to a window length so as to decide which thebest motif is among motifs with different lengths.

2 RECONSIDERRATIONDIFINITION OF MOTIF

THE

This section describes the commonly used definition of
motif and summarizes the problems for deciding optimal
length of motif.

2.1 Definition of a Motif
Motif is defined by using nearest neighbor distance in thespace consisting of subsequences in a time series.

Keywords: Time series data mining, Motif discovery

1ONDefinition: time series X
A Time Series X=[x1,…,xm] is a continuous sequence ofreal values. We denote the value of the i-th time point as
X[i] = xi.

INTRODUCTION

Time series motifs [1][2] are approximately repeating
subsequences embedded in a time series. Motifs are one of
the most important primitives in time series data mining,and motif discovery has been used as a sub-routine inhigher-level analytics, including classification, clustering,visualization and rule-discovery. Moreover, motif discovery
has been applied to domains as diverse as factory operation[3], medicine [4], and seismology [5]. The notion of a motif
is useful for a wide range of applications, because a repeated
and frequently occurring pattern implies a latent system thatoccasionally produces a repeatable output. For example, thissystem may be an over-caffeinated heart, sporadicallyintroducing a motif pattern containing an extra beat [6], orthe system may be a factory worker, producing repetitive
movement in a series of assembly operations [3].
Since the Matrix Profile [7], a fast and scalable algorithm
subsequence all-pairs-similarity-search in time series wasintroduced, it has helped to develop new innovative ideas
for time series data mining [8]. However, as a motif isdefined as a pair of subsequences the distance between
which is the smallest, it does not necessarily imply thatsimilar subsequences to the motif occurred frequently in atime series. That is, there are not necessarily very many

Definition: subsequence X[p:q]A subsequence s = [xp, xp+1,...,xq] = X[p:q] is a list which
consist of continuously occurring values in X, starting at
position p and ending at position q. We also denote a, which means a subsequencesubsequence X[p:q] as
staring at p with length w.
The length of a subsequence is w
1 and wedenote it as length(s).
Definition: support of a subsequence
The support of a subsequence is a set of time at which a
subsequence have a value. The support of X[p:q] is [p:q]= [p, p +1,..., q-1, q] and we denote it as support (s).
Definition: subsequence space
Subsequence space is the set of all the subsequences with.
lengthin a time series X. We denote it as
is a dimensional vector space.
Subsequence space
Therefore, for given subsequences and , the distance
between and , which we denote as dist , , can be133International Workshop on Informatics ( IWIN 2018 )defined the same as vector space. We use
paper.
,|X≡

this property “trivial match”. Most time series which havemotif patterns satisfy this assumption. Trivial match is1] , X[ :described formally by that dist(X[ :| ≪ . Trivial matching1 ]) is small, if |suggests that the simple definition may count subsequences
duplicately. When we count the similar subsequences, weshould take trivial matching into consideration.

distance in this1 |1Definition: disjoint subsequence
Let si and sj be subsequences. When support (si) andsupport (sj) are disjoint, that is support (s1) ∩ support (s2)∅, we call that si and sj are disjoint.

3 MOTIF OCCURING FREQUENCY AND
COVERAGE
3.1 Our Approaches

Definition: Motif subsequence (1-NN)
Let w be a window length and X be a time series. A
subsequence s with length w in X which satisfies the below
condition is called a motif.
There is a subsequence s with length , such that
| , ∈
and
dist(s, s ) = min dist ,This subsection describes our approaches for each ofproblems described in the precious section.
(1) Error dependency
We will define a neighborhood of a subsequence infor a given time series with two parameters, a window
length and a distance error .

,support∩ support∅

The above definition is based on one nearest neighbor (1NN) distance. We can extend this definition for k nearest
neighbor distance by replacing minimum with k-th
minimum in the above condition.

(2) Window length dependency
If an error is equal, the longer subsequence seems to be
more appropriate than the shorter subsequence as a motif.
Therefore, we introduce an error per subsequence length fora longer length motif to be advantageous. Furthermore, wewill define ‘motif coverage’ as an index for deciding thebest motif among subsequences with different window
lengths.

2.2 Problems in Defining Motif Criteria
The intuitive meaning of a motif is a subsequence which
has many similar subsequences in a time series X. The firstidea for defining motif criteria, which are indexes tomeasure how suitable a subsequence is for a motif, is to giveindexes to measure ‘similar’ and ‘many’ in the above
intuition. A similar sequence can be measured by a distance
between subsequences. With regard to the definition of“many”, it seems to be possible to count the number ofsubsequences which are similar to a given motif. We callthis number “occurring frequency”. There are several
problems for defining occurring frequency and motif criteria.
Those problems can be summarized in the following threepoints.

(3) Trivial match
When we define a neighborhood of a subsequence in, we remove trivial matchingsubsequence spacesubsequences of .

3.2 Neighborhood of a SubsequenceFirst, we introduce a disjoint neighborhood for treating atrivial match problem. Then, we define a maximal disjoint
neighborhood of a subsequence for defining occurring
frequency.

(1) Error dependency
When we call a sequence is similar to a subsequencemeans that dist( , ) is small. Therefore, the threshold of anerror parameteris necessary for counting similarsubsequences to . A simple definition of occurring|,
frequency of is | | ∈where |A| means the number of elements of a set A. Thisdefinition needs a window lengthand an error as
parameters.

Definition: Disjointof a subsequence
Let X, w , and s are a time series, a window length, apositive real number and a subsequence with length w
respectively. Disjoint neighborhood of s , which we, is a subset ofwhere for every
denote as∅.
, support∩ support
pair , ∈,The above definition allows little important subsets. In an
extreme case, a null set satisfies the above definition. Ourconcern is the one which has maximal number of disjoint
subsequences.

(2) Window length dependency
As the previous simple definition shows, a window length
should have been given before we define the distance
between subsequences. When we select the better motif
from motif candidates with different window lengths, weneed the criteria for comparing the appropriateness ofsubsequences with different lengths as motifs.

Definition:neighborhood of a subsequenceis aMaximal neighborhood of a subsequence,
disjoint neighborhood
which
has
the
largest,number of elements.

(3) Trivial match
Subsequences close to a subsequence are similar to , iftime series is continuous and has a small vibration. We call

There are several choices in selecting the elements of a
maximal disjoint neighborhood. However, there is no134

International Workshop on Informatics ( IWIN 2018 )We can select the best motif among subsequence withdifferent window lengths by the following procedure.
1. Give a list of window lengths W.
2. Select the subsequence which has the maximal occurring
frequency for each window length in W. The selected
subsequences are each best motifs for each window lengths
in W.
3. Select the subsequence that has the maximal motif
coverage among the motifs obtained by procedure 2. The
selected subsequence is the best motif among all thesubsequences with all the window lengths in W.

problem, because it is sufficient to know only the number of
elements of a set without knowing the contents of a set for
defining occurring frequency.
Theorem: Construction of maximal disjoint neighborhood
Let X, w , and s are a time series, window length, a positive
real number and a subsequence with length w. When we
repeatedly select the nearest disjoint subsequence from s
towards right to the end of X (we call the selected
subsequences right neighborhood) and conversely search
towards left to the beginning of X (we call the selected
subsequences left neighborhood), the union of the right andleft neighborhoods is a maximal neighborhood of . We
call this construction method ‘nearest order selection’.
Proof:
is a maximal neighborhood of , weWhen a,will show that the subsequence set constructed by the above
procedure has the same number of elements. We will provethe case of right neighborhood, because the proof of the leftcase is similar to it.
by time ordering, we get
When we sort the element of ,
…,
,
,
,…,
,
where
⋯
.
By nearest order selection, the first selected subsequence
inequalty 1 . Because
satisfies
where
which has the smallest,
p
support∩ supportp∅.
satisfies
Next, secondly selected subsequence
inequalty 2 similarly.
If we continue these procedures, i-th selected subsequence
1,2, … ,inequaltysatisfies
The above inequalities show that the disjoint neighborhood
constructed by nearest order selection
…,
,
,
,…,
,
has equal or larger number of elements to ,.
However, because ,is a maximal disjoint
neighborhood, the number of elements ofmust be,equal to that of ,. This means thatis also a,
maximal neighborhood of a subsequence .

4

ALGORITHM

We can get algorithms for calculating occurring frequency
and motif coverage by operationally interpreting thedefinitions and the theorem in the previous section.
TABLE 1 shows an algorithm that counts the occurring
frequency of a given subsequence. The inputs are a timeseries X, a window length w of the given subsequence s, astarting time t of s and an error per window length . The
outputs are the occurring frequency and the motif coverageof the given subsequence s.
Line 01 calculates each distances between the given. Line 02
subsequence s and each subsequences incounts the number of elements of a maximal neighborhood
subsequence set whose elements are in the right side of thegiven subsequence s. Line 03 counts the number of theelements of a maximal neighborhood subsequence set whose
elements are in the left side of s. Line 04 counts the total
occurrence frequency of s by adding the right side one
obtained by line 02 and the left side one obtained by line 03.
Line 05 calculates the motif coverage of s by multiplying
the window length w and the occurring frequency obtainedby line 04.

TABLE 1. CountOccurringFrequency Algorithm
Algorithm: CountOccurringFrequency (X, w, t, )[Input] X: Given time series
w: Length of a given subsequence s
t: Stating time of a given subsequence s: Error per unit length[Output] OF: Occurring frequency of s in error
MC: Motif coverage of s
01 DL = distanceListFromS(X, t, w);
02 OFR = countRightOccurence (DL, t, w, )03 OFL = countLeftOccurence (DL, t, w, )04 OF = OFR + OFL;
05 MC = OF * w;
06 return (OF, MC);

3.3 Occurring Frequency and Motif Coverage
This subsection defines occurring frequency and motif
coverage.
Definition: Occurring frequency
Let X, w , ϵ and s are a time series, a window length, apositive real number and a subsequence with length w
respectively. Occurring frequency of a subsequence s on anerror ϵ is the number of the elements of a maximal, that is,.
neighborhood of a subsequence ,,TABLE 2 shows an algorithm that counts the number of
elements of a maximal neighborhood subsequence set whose
elements are in the right of the given subsequence s. The
inputs are the distance list DL obtained by line 01 in
TABLE 1, the window length w of a given subsequence s, astarting time t of s and the error per window length . The

Definition: Motif Coverage
Let X, and are a time series and a unit error per window
length respectively. Motif coverage of a subsequence withlength w is,135

International Workshop on Informatics ( IWIN 2018 )5

output is the number of maximal neighborhood
subsequences in the right side of s.
Line 01initializes a time cursor ‘Cur’ and a normalized
error ‘Err’. Line 02-13 is a while-loop that chooses maximal
subsequences that are in the right side of the given
subsequence s to the end of the time series X. Line 03-05 isa while-loop that searches the next disjoint subsequence
whose distance from s is smaller than ‘Err’. Line 06-09
increments ‘Right’ when the line 03-05 found a new disjoint
subsequence. Line 10-12 exits while-loop 02-13 after
checking all the subsequences in the right side of s.

We show that our proposed index ‘motif coverage’ canselect the best motif among subsequences with different
window lengths.

5.1 Window Length Selection
This subsection shows that for two simulated time series
in which motif patterns are intentionally inserted, motif
coverage can select the best window length of the inserted
motif.

TABLE 2. CountRightOccurence(1) Experiment 1
First, we will show that ‘motif coverage’ can decide thebest window length 15 by selecting window length that has
the highest ‘motif coverage’ among each best motifs withwindow lengths 5,9,15 and 31 in time series shown by
Figure 1.
Figure 1 is a simulated time series that combines sine
curves with length 15 and random subsequences withvarious length.
Figure 2 shows motif coverage values on each times andeach window lengths in case that an error per window lengthis 0.01. The top graph is a motif coverage trend graph inwindow length 5. The second, third and fourth trend graphsfrom the top to the bottom are each motif coverage ones
with window length 5, 9, 15 and 31 respectively. The third
trend graph for length 15 shows that times at which motif
patterns start have high and sharp peaks. The trend graphs
for length 5 and 9 shows that times at which sub-patterns of
the best motifs start with length 15 have relatively highmotif coverage values and longer peak duration than thoseof length 15. These observations match the fact that the best
motif pattern includes motifs with smaller window lengths.
Figure 3 shows each best motifs in each window lengths.
The best motif in length 15 is an intentionally inserted one.
The best motifs for window lengths 5 and 9 are the subpatterns of the best motif with length 15. The best motif
with length 31 is a subsequence including the best motif
with length 15.
Figure 4 shows motif coverage values for each best motifs
with each window lengths. The window length which has
the highest motif coverage is 15. It shows that motif
coverage can decide the optimal window length. It also
shows that the nearer window length to 15 is, the higher
motif coverage is. This result is what we expect for motif
coverage.

Algorithm: countRightOccurence (DL, t, w, )[Input] DL: Given time series
w: Window length of a given subsequence s
t: Stating time of s
: Error per unit length[Output] Right: the number of maximal neighborhood
subsequences in the right of s.
01 Cur = t+1; Err = ∗ ;
02
03
04
05
06
07
08
09
10
11
12
13
14

EXPERIMENTAL EVALUATION

while Cur <= length(DL)while DL(Cur) > Err or Cur <= length(DL)Cur = Cur + 1;
end
if DL(Cur) <= Err
Right := Right + 1;
Cur := Cur + w – 1;

end
if Cur > length(X)break;
end
end
return Right;

TABLE 3 shows an algorithm that counts the number ofmaximal neighborhood subsequences which are in the left
side of the given subsequence s. The left case can be
reduced the right case by reversing the time series values
from right to left.
Line 01 reverses the distance list ‘DL’ from right to left.
Line 02 reverses the starting time t of s from right to left.
Line 03 gets the value of the left case by calling thealgorithm ‘CountRightOccurence’ with the reversed
arguments.

TABLE 3. CountLeftOccurence Algorithm
Algorithm: countLeftOccurence (X, w, t, )[Input] DL: Given time series
w: Window length of a given subsequence s
t: Stating time of a given subsequence s: Error per unit length
X: Given time series[Output] Left: the number of maximal neighborhood
subsequences in the left of s.
01
02
03

Figure 1. A time series with a motif of length 15 samples

DL_rev = fliplr(DL);t_rev = length(X) – t + 1;
Left = countRightOccurence (DL_rev, t_rev, w, )136

International Workshop on Informatics ( IWIN 2018 )intentionally inserted ones. The best motifs in each window
lengths 5 and 9 are sub-patterns of the best motifs withlengths 15 or 31. The best motif with window length 47 is asubsequence including the best motif in window length 31.
Figure 8 shows motif coverage for each best motifs witheach window lengths. It shows that 15 and 31 are top 2
window lengths.

Window
Length = 5

Window
Length = 9

Window
Length = 15

Window
Length = 31

Figure 2. Motif coverage of each times for eachwindow lengths

Figure 5. A time series with length 15 and 31 motifs.
Window
Length = 5
Window
Length = 9
Window
Length = 15
Window
Length = 31
Window
Length = 47

Figure 6. Motif coverage of each times for eachwindow lengths

Figure 3. Best motifs for each window lengths
300
250
200
150
100
50
0

5

9

15

31

Figure 4. Motif coverage of best motifs with eachwindow lengths(2) Experiment 2
Next, we will show that ‘motif coverage’ can decide thebest window length in a time series including two motif
patterns with different lengths.
Figure 5 is a simulated time series that combines sine
curves with length 15 and 31 and random subsequences withvarious length.
Figure 6 shows motif coverage trend graphs in eachwindow lengths in case that an error per window length is
0.01. Each trend graphs are for window lengths 5, 9, 15, 31
and 47 from the top to the bottom. As is the first experiment,
the trend graphs for 15 and 31 have high peaks of motif
coverage values at times when motif patters start. The
trend graphs for 5, 9 and 15 have relatively high motif
coverage at times when sub-patterns of the motif patterns
with length 15 or 31 start.
Figure 7 shows each best motifs in each window lengths.
The best motifs in window lengths 15 and 31 are

Figure 7. Best motifs for each window lengths
300
250
200
150
100
50
0

5

9

15

31

47

Figure 8. Motif coverage of best motifs with eachwindow lengths
137

International Workshop on Informatics ( IWIN 2018 )5.2 Error dependency of Motif Coverage

6

The previous subsection shows that ‘motif coverage’ with
an appropriately selected error per window length candecide the best window length. However, the problem how
to decide an error per window length still remains. This
subsection shows error dependency of motif coverage andthe consideration on this remaining problem.
Figure 9 shows error dependency graphs of each motif
coverages of the best motifs with each window lengths forthe time series shown in experiment 1. This graph showsthat the best window length has overwhelmingly highest
value in small error values. However, as the larger the error
value is, the smaller the difference is. This observationsuggests that an occurring frequency should be measured ina small error value, because motif is local characteristics in a
. This observation also suggests
subsequence spacethat a rising point in an error dependency graph seems to bea candidate for deciding the optimal error per window length.

CONCLUSIONSWe proposed novel indexes ‘occurring frequency’ and‘motif coverage’ for deciding optimal window length inmotif discovery. We also showed that ‘motif coverage’ candecide the best window length by selecting window lengththat has the highest ‘motif coverage’ among each best motifs
with each window lengths in simulation data.
From a theoretical point of view, a future work is how todecide an error per window length, which is a remained
parameter of motif coverage. From an experimental point ofview, we plan to apply our algorithms to more complex
simulated data as well as real data.
This work is supported by JSPS KAKENHI Grant Number
17K00161.

REFERENCES

Motif coverage[1] Pranav Patel, Eamonn J. Keogh, Jessica Lin, Stefano
Lonardi: Mining Motifs in Massive Time Series
Databases. IEEE ICDM pp. 370-377 (2002).
[2] Abdullah Mueen, Eamonn J. Keogh, Qiang Zhu,
Sydney Cash, M. Brandon Westover: Exact Discovery
of Time Series Motifs. SDM pp. 473-484 (2009).
[3] Takuya Maekawa, Daisuke Nakai, Kazuya Ohara,
Yasuo Namioka: Toward practical factory activity
recognition: unsupervised understanding of repetitive
assembly work in a factory. UbiComp pp. 1088-1099(2016).
[4] Zeeshan Syed, Collin M. Stultz, Manolis Kellis, Piotr
Indyk, John V. Guttag: Motif discovery inphysiological datasets: A methodology for inferring
predictive elements. TKDD 4(1): 2:1-2:23 (2010).
[5] Yan Zhu, Zachary Zimmerman, Nader Shakibay
Senobari, Chin-Chia Michael Yeh, Gareth Funning,
Abdullah Mueen, Philip Brisk, Eamonn J. Keogh:
Matrix Profile II: Exploiting a Novel Algorithm andGPUs to Break the One Hundred Million Barrier forTime Series Motifs and Joins. IEEE ICDM, pp. 739748 (2016).
[6] William R. Lovallo, Michael F. Wilson, Andrea S.
Vincent, Bong Hee Sung, Barbara S. McKey, Thomas
L. Whitsett: Blood Pressure Response to Caffeine
Shows Incomplete Tolerance After Short-Term
Regular Consumption, Hypertension.43.4 p.760-765(2004).
[7] Chin-Chia Michael Yeh, Yan Zhu, Liudmila Ulanova,
Nurjahan Begum, Yifei Ding, Hoang Anh Dau, Diego
Furtado Silva, Abdullah Mueen, Eamonn J. Keogh:
Matrix Profile I: All Pairs Similarity Joins for Time
Series: A Unifying View That Includes Motifs,
Discords and Shapelets. IEEE ICDM, pp. 1317-1322(2016).
[8] Yan Zhu, Makoto Imamura, Daniel Nikovski, Eamonn
New Primitive for Time Series Data Mining (BestStudent Paper Award). IEEE ICDM, pp. 695-704(2017).
[9] Eamonn J. Keogh, Jessica Lin, Wagner Truppel:
Clustering of Time Series Subsequences is

Motif coverage

Figure 9. Error dependency of each motif coverage ofeach best motifs with each window lengths (Data 1)

Figure 10. Error dependency of each motif coverageof each best motifs with different window lengths(Data 2)
Figure 10 shows error dependency for time series shown inexperiment 2. This graph shows that the best window length
15 and 31 have overwhelmingly higher values in small error
values. However, as larger the error value is, the smaller thedifference is. This observation is the same as that in data
shown in experiment 1.

138

International Workshop on Informatics ( IWIN 2018 )Meaningless: Implications for Previous and Future
Research. IEEE ICDM, pp. 115-122 (2003).
[10] Tsuyoshi Idé: Why Does Subsequence Time-Series
Clustering Produce Sine Waves? PKDD pp. 211-222(2006).

139

International Workshop on Informatics ( IWIN 2018 )Visualization of Skills and Techniques Using Big Data Analysis for Vocational Skill
Development
Masaki Endo*, Takuo Kikuchi*, Shigeyoshi Ohno *, Makoto Imamura**, and Hiroshi Ishikawa****

Division of Core Manufacturing, Polytechnic University, Japan{endou, kikuchi, ohno}@uitec.ac.jp**
School of Information and Telecommunication Engineering, Tokai University, Japan
imamura@tsc.u-tokai.ac.jp***
Graduate School of System Design, Tokyo Metropolitan University, Japan
ishikawa-hiroshi@tmu.ac.jp
Abstract–– At manufacturing sites throughout Japan, labor
force reduction and industrial structure changes areprogressing. To meet these and related challenges, manysmall and medium-sized enterprises need not only toadvance skills and techniques, but also to hand down skills
and techniques to new groups of workers. Therefore, theneed for human resource development in small and mediumsized enterprises is increasing. Although many suchenterprises acknowledge the importance of vocational
training, a shortage exists of human resources responsible
for guiding vocational ability development. Visualization ofskills using ICT and advancement of skills are urgently
needed at manufacturing sites in Japan. This study examines
the visualization of skills using big data analysis at sites ofvocational ability development. As described herein, we
explain construction of the analytical environment using theoperation history in programming practice as the first stage.

are all examples of big data recognized by the Ministry ofInternal Affairs and Communications (METI) [1]. Big data
is defined as "a large amount of data that is difficult tomanage with existing technology" and which is represented
by Volume, Variety, and Velocity: the 3 Vs (generation
speed / update frequency). Large varieties of data having
various forms and structures can be generated, collected,
accumulated, and otherwise processed using informationand communications technology (ICT). Furthermore, bigdata are fundamentally heterogeneous large-scale datasetswith non-stationarity that captures data with different
frequencies and accuracies from moment to moment.
Fundamentally, conventional data analysis processes
input data collected for some purpose with an algorithm,
yielding necessary information as an output. However,analysis of big data makes it possible to realize scientificdiscovery, prediction, and knowledge acquisition that could
not be achieved through conventional data analysis.
However, representing natural phenomena and social life
requires the use of analyzed results of data of various kinds
gathered using sensor arrays, monitors, and recorded
information transmitted on a daily basis. Furthermore, for
information processing, society is shifting frominterpretation of data and from appropriate modeling ofanalysis methods to information and knowledge acquisition.
For the reasons enumerated above, the E-Science Data
Center Science Subcommittee of the Informatics Committeeof the Science Council of Japan [2] and the Ministry of
Education, Culture, Sports, Science and Technology(MEXT) [3] recommend the necessity of developing big
data utilization technicians who have acquired new
information processing technologies such as data acquisition
technology dealing with big data, data utilization technology,
machine learning, and statistical modeling. Furthermore,they are tackling human resource development. In this way,
top-down efforts centering on MEXT, METI, and IT
industries are being developed, such as academic research
responding to technological innovation and further
improvement of technical skills of information processing
engineers.
Nevertheless, bottom-up training for practical engineers
to use big data mainly in manufacturing industries in Japan
has not been implemented as public vocational training.
Figure 1 portrays the situation in Japan and an outline of thisresearch, which was conducted to construct a curriculum

Keywords: big data; big data utilization technology;
curriculum model; job training; polytechnic science

1

INTRODUCTION

With the dramatic development of information andcommunication technologies such as the spread ofsmartphones and Internet of Things (IoT), enormous
amounts of information generated in society and many
academic research fields are accumulated as big data. Such
big data include vast knowledge and potential value.
Therefore, their effective use for analysis and processing ofdata using artificial intelligence (AI) is crucially importantfor future industrial development. Actually, big datautilization is anticipated not only to strengthen theinternational competitiveness of companies, but also tosolve social problems, to create new businesses and services,to improve the convenience of individual lives, etc. Already,intense international competition has begun. In the industrial
world, Germany has promoted Industry 4.0; the US
advocates an Industrial Internet. Moreover, a struggle forinternational standardization has developed. Comprehensive
efforts including human resource development to cope withthis trend have become urgent in Japan as AI, IoT, big data,and structural reforms for technological innovation of robots.
Social media data, website data, sensor data, log data,
multimedia data, customer data, office data, operation data

141

International Workshop on Informatics ( IWIN 2018 )model for learning Big Data utilization technologies withthe current training contents of public vocational training. At
universities and large enterprises designated by MEXT,
research and efforts are being conducted on advanced data
science. Efforts related to big data utilization are limited at
production sites of small and medium enterprises. Amongthem, the introduction of data science to production sites ofsmall and medium enterprises represents an urgent issue forJapan's manufacturing industry, where the labor force isdecreasing and the aging of skilled technicians is
progressing. Therefore, this study was undertaken to
construct a data collection infrastructure to promote theutilization of big data for engineers at the production site.
Based on data gathered in this research, we assess acurriculum through which on-site technicians acquire basic
knowledge of utilization technology using data science andby which they can collaborate with advanced data scientists.
Therefore, we consider data necessary for analyses from aperson, machine, material (3M) perspective of the model
used for problem identification and analysis at the
production site and aimed at modeling. As described in thispaper, we explain programming practice as the first
approach. As described herein, we present an approach to
standardize the data model for analysis by accumulating
data of a trainee's programming work, targeting
programming practice. Chapter 2 describes earlier research
related to this topic. Chapter 3 explains our proposed
method for estimating proficiency in education and fortraining using data from a trainee's programming work.
Chapter 4 describes experimentally obtained results for our
proposed method and a discussion of the results. Chapter 5
presents a summary of contributions and future work.

recording of human activities as body movement big dataand present approaches to visualize differences between
expert skills and novice skills using a technique called skill
grouping. They identify skill differences in sports activities
by comparison with skilled athletes. Miyadera et al. [6]present a system in which teachers monitor real-timelearning of beginner programmers in the classroom andprovide support to groups of students with common
problems. Their system has a program animation functionthat passes through the program with understanding of thestudent's historical record of operations. By analyzing thisrecord, the system accurately notifies the teacher of common
problem areas and an inferred cause of the difficulty.
Tanigawa et al. [7] reports that a teacher can not teachefficiently when a student's source code is read at the time ofquestioning in the programming exercise. Furthermore, thetrial and error process from the record of the function callingorder at the time of source code creation by the student isanalyzed. A calculation method is proposed for student
acquisition items. In their method, by deriving a patterncommon to many students as a calling pattern, a user canascertain learning items efficiently and can then pursue
effective teaching.
Techniques related to skills and techniques have been
proposed in various fields, but examination of datacollection and analytical methods for the manufacturing
field have not been systematized. In many cases at manylarge enterprises, a PDCA cycle such as kaizen isincorporated into a business, leading to productivity
improvement. However, in small and medium-sizedenterprises supported by public vocational training,
implementing such a cycle effectively and efficiently isdifficult because of labor shortages and individualization.
Therefore, we adopt a scientific approach for visualizationand sophistication against skills required of manufacturingexperts possessed by skilled technicians such as craft skills
and Olympic champion skills. In addition to engineering, weare considering the application of methodologies to fields
such as social system science, pedagogy, and human
information science [8].

3This section presents a description of the analyticalmethod used for target data collection. As the first step ofsystematizing data collection that is applicable to analysis of
the skill, which is the ultimate objective, we present a datacollection method modeling the operation history at the timeof programming practice. Our proposal is presented in Fig. 2.
This paper introduces data collection in programming
practice, but the ultimate goal of this research is to collectlarge-scale data targeting the Polytechnic Centers of 64
public training centers and Polytechnic Colleges of 25
schools nationwide. Furthermore, the data collection target
area is expected to include mechanical systems, electricalsystems, construction systems, and others.
We describe an acquisition level estimation method using
analysis with the operation history at programming practice.
Section 3.1 describes gathering of the operation history at
programming practice. Section 3.2 describes preprocessing

Figure 1: Present situation in Japan and outline of thisresearch.

2

OUR PROPOSED METHODRELATED WORK

Research investigating proficiency related to skills andtechniques is conducted in various fields. Furukawa et al. [4]specifically examine such sophisticated physical skills as
playing sports and playing instruments and introduces
research activities aimed at elucidating and verbalizing thoseskills. They introduce approaches from physical modeling,measurements and data analysis, cognitive science, andhuman interfaces. Yamagiwa et al. [5] simplify the142International Workshop on Informatics ( IWIN 2018 )for conducting analyses. Section 3.3 describes the method of
estimating the learning degree. With an analytical method
using the operation history during programming practice,
one can estimate the learning degree to programming
practice. Section 3.4 presents output of the estimation result.

execution file. Figure 3 presents an overview of acquisitionlevel estimation.

Figure 2: Summary of our proposal.

3.1 Data Collection

Figure 3: Estimating acquisition level.

This section presents a description of the Method of (1)data collection presented in Fig. 2. As described in thispaper, the file operation history at the time of programming
practice is the collection target. Items to be monitored arefile creation, editing, and saving. This data collection was
done for Arduino programming practical training using
folder monitoring free software [9] in a Windows
environment and the inotifywait [10] command in Linux.
Next, we describe the number of collected data. As theinitial stage of standardization this time, we obtained data ofseveral cooperators. To classify the programming
experience, data were classified as those of programmers
with no experience, with less than five years experience, orwith over five years of experience. We analyzed theacquisition level by applying processes using these data.

Here, the number of times the coding file was saved
might be saved for backup, but it is assumed that manysaves are done during compilation. Writing to themicrocomputer is performed when the executable file iscreated or updated. Because Create is the first write to themicrocomputer, a file update occurs. Therefore, judging thepossibility of overlapping compilation errors means that one
file is saved many times. When the corresponding execution
file is not updated at the time of saving of the coding file, it
is judged that a compilation error has occurred. Furthermore,
if the coding file is saved after the execution file is updated,
it is judged that a runtime error has occurred and that an
operation to specify has not been completed.
Therefore, using the proposed method, the shorter thework time, the higher the degree of acquisition. Additionally,
we conducted an analysis assuming that a higher skill levelis associated with fewer saves of the coding file and fewer
updates of the executable file within the working time. The
proposed method is a very general analytical method. As
programming analysis methods, advanced and highly
accurate analysis methods such as empirical software
engineering have already been examined in earlier reports.
Nevertheless, no report of the relevant literature in the fieldof public vocational training and analyzing beginnerlearning technology describes a study conducted to establish
an effective public vocational training system byconstructing a large-scale data collection infrastructure.
Therefore, this study was conducted to construct a datacollection base conforming to the actual circumstances ofpublic vocational training sites.

3.2 Preprocessing
Preprocessing stores the creation, editing, and saving ofeach user's code file and the creation, editing, and saving
history of executable files generated by compilation in the
database. The file name is targeted for programming
practice. Therefore, it is specified beforehand. The code fileand the executable file are associated. In the database, theoperation history of the user's coding file and executable file
are stored together with time information.

3.3 Estimating acquisition level
Analysis of the degree of acquisition uses data
accumulated through preprocessing. The time from creationof the coding file by each user to the final time at which theexecution file was created by compilation is set as theworking time. Analysis of the acquisition level was done
during that time according to the relation between thestorage time of the coding file and the update time of the

3.4 Output
This section presents a description of (4) the output
method portrayed in Fig. 2. Output considers visualization
143

International Workshop on Informatics ( IWIN 2018 )using the acquisition degree estimated by processing, as
explained in the preceding section. We assess the use as an
auxiliary system for programming learning by showing eachuser’s learning level and the real time transition in class.

4

3. Explanation of compilation and operation confirmation
method
4. Explanation of elements necessary for the task
5. Execute task
6. Repeat 4 and 5 for each task

EXPERIMENTS
4.2 Examples of the problem of programming
practiceThis chapter explains experiments conducted to estimate
the acquisition level using the method explained in Chapter
3. Section 4.1 describes the dataset. Section 4.2 presents
examples of the problem of programming practice forcollecting data. Section 4.3 presents the trend of the degreeof acquisition obtained using the collected data.

4.1

This section presents practice tasks prepared for datacollection. Figure 4 and Fig. 5 respectively present examplesof tasks involving tact switches and LEDs. Simple examples
illustrate the foundation of embedded programming, but
experienced persons and inexperienced people exhibit
differences in their times to perform tasks. For theexperiment, we prepared five tasks and gathered theoperation history of the experiment collaborators. Tasks 1–5
confirmed the following points.
Task 1: Using timer and external interrupt
Task 2: Port operation
Task 3: Basics of Interrupt Processing
Task 4: Application of interrupt Processing
Task 5: Application of Task 1-4
Moreover, in this experiment, hardware parts such as
LEDs and switches were prepared in advance. Data
collection and analysis including tasks other thanprogramming, such as hardware assembly, are the next stepof this research.

DatasetDatasets used for this experiment were collected using
data presented in Table 1. As described in this paper, theresults for two people are shown by experimentation.
Therefore, the data are few; the experiment results areunreliable. Currently, we are selecting procedures for asimilar curriculum and obtaining permission for datacollection; we are also collecting data individually bysoliciting cooperation with the experiment. This time wegathered data from experimental collaborators after
explaining the gist of data collection in advance and after
obtaining consent. Programming experience, embedded
programming experience, and embedded programming
experience using Arduino are presented in the table. We use
two people's data: both A and B have programming
experience, but A has embedded programming experience;
B does not. Neither A nor B has Arduino programming
experience. The number of hours A is 72 hr for C language
and 72 hr for embedded programming. B is 36 hr in C
language. The programming time using research and otherlanguages is not included. Furthermore, in general public
vocational training, the programming practice time is often
implemented in units of 36 hr. Therefore, we believe it will
be a reference for actual training measurements.

Figure 4: Example of task 1.
Table 1: Datasets
Experimental Programming
collaborators
experienceAB

3 years
4 years

Embeddedprogramming
experience

Embeddedprogramming
experience using
Arduino3 years
No

No
No

We gathered data by having each experimental
collaborator perform the same task. Because an Arduino isused for the first time in some cases, we use an explanation
for beginners to instruct everyone in the prior explanationand programming method. Therefore, the data collection
flow occurs according to the following procedure.

Figure 5: Example of task 2.

4.3

Experimental result

Table 2 and Fig. 6 present results from Task 1 of thestudent serving as the instructor of this experiment. The
student had learned about Arduino programming beforehandand had instructed the experiment examinees. The working
time was 4 min. Additionally, the coding file was generated
at 14:20 and was updated twice. Regarding the 14:22 update,it was impossible to generate an executable file because of aProcedure
1. Arduino description
2. Explanation of the programming method using a sample
program144

International Workshop on Informatics ( IWIN 2018 )compilation error, but compilation was successful at the
14:23 update. Therefore, one can confirm that theexecutable file was created at the same time. In addition, theexecution file has two events: 14:23 compiling success and
14:24 updating. The second event at 14:24 is an event by
writing to the microcomputer. Therefore, A finished task 1
in 4 min by two compiling works and one writing work.
Table 3 presents results of tasks from each of the twocollaborators shown in Table 1. A has experience withembedded programming. Therefore, the work time of Task 1
is shorter than that of B. Adaptation to Arduino
programming is apparent. In addition, Task 3, an intrinsic
programming specific task that uses interrupts and delay
functions, is applicable in a shorter time than B. Therefore,
embedded programming experience is apparently superior
for dealing with this Arduino programming. By contrast, B
tends to have fewer compile errors than A, and tends tocompile less frequently until completion of a Task.
Although B had no experience with embedded programming,
B’s programming proficiency is thought to have been higher
than that of A. In Task 4 and Task 5, which are application
tasks, the problem is a summary of the fundamental
programming of Arduino, which was undertaken this time.
Both A and B show almost no difference between these two
problems. Therefore, I think that basic skills can be learned
through the experiment Task. Achievement obtained using
the curriculum was obtained.
Although this is still in a hypothetical stage, we believethat grammatical errors and algorithmic errors can be
reduced by improvement of programming skills. Therefore,
at the time of programming, the skill is analyzed based on
the number of times used for compiling.

Table 3: Data of task 1 of experiment collaborator A
Experimental
collaborators

A

B

Task 1
Task 2
Task 3
Task 4
Task 5
Task 1
Task 2
Task 3
Task 4
Task 5

Trends of embedded programming experience andprogramming proficiency can be confirmed using thismethod. Therefore, if one were able to acquire a large
amount of data using this system, then analytical objectives
such as the proficiency level of detailed programming anddifficulty judgment of tasks could be considered using data
analysis. Furthermore, collaborators implemented the task
only once for this experiment, but it is expected that
repeating the programming practice will shorten the work
time and reduce the compiling errors for the task. Futurestudies must include experiments designed to collect andanalyze numerous histories of programming practices ofsingle subjects over a long time.

5

CONCLUSIONAs described in this paper, for vocational ability
development, we have constructed a system to collectoperation histories during programming practice as the firststep to visualizing skills and techniques with big data
analysis. The research, which started this fiscal year, is at
the stage of consultation with the ethics committee on datacollection related to personal information. Therefore, wehave not reached large-scale data collection. However,experimentally obtained results of collaborators during theseexperiments confirmed the differences between experienced
programming and mastery level programming. Results
demonstrate the possibility that accumulating large amountsof data using a data collection system in the future can
support analysis of practical programming skills andtechniques. Additionally, although targeting data inprogramming practice now, the author would like to conduct
research to clarify differences between inexperienced andskilled workers by sequentially expanding and collecting
data including human movement and biometric information.

Table 2: Data of task 1 of instructor

Times of Day
14:20
14:22
14:23
14:23
14:24

Task

Number of Number of
Number of Number ofWork time Coding file Execution file
Work time Coding file Execution
averagehistoryhistory[min]history file history[min]averageaverage[time][time][time][time]
11
7
7
4
2
2
5
5
4
4.6
3.6
3
1
2
1
2
2
1
14
4
4
4
2
2
11
4
4
6.4
2.4
2.4
2
1
1
1
1
1

History
Filename
Createkadai_01.ino
Update kadai_01.ino
Update kadai_01.ino
Create kadai_01.ino.hex
Update kadai_01.ino.hex

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant
Number 18K13254.

REFERENCES[1] Information and Communications Council, Regarding
how to utilize big data, Ministry of Internal Affairs andCommunications (2012).
[2] Japan Science Council Information Informatics
Committee, Recommendation Human Resources

Figure 6: Data of task of instructor.
145

International Workshop on Informatics ( IWIN 2018 )Supporting the Big Data Era, National Research andDevelopment Corporation Japan Science andTechnology Agency, (2014).
[3] Information and Systems Research Organization,
About developing specialized human resources toutilize big data, Inter-University Research InstituteCorporation Research Organization of Information and
Systems (2015).
[4] K. Furukawa, K. Ueno, T. Ozaki, S. Kamisato, R.
Kawamoto, K. Shibuya, N. Shiratori, M. Suwa, M.
Soga, H. Taki, T. Fujinami, S. Hori, Y. Motomura, and
S. Morita, Research Trend of Physical Skill Science –Towards Elucidation of Physical Skill –, Transactionsof the Japanese Society for Artificial Intelligence,
Vol.20, Issue2, pp.84-93, Japanese Society for
Artificial Intelligence (2005) (in Japanese).
[5] S. Yamagiwa, Y. Kawahara, N. Tabuchi, Y. Watanabe,
and T. Naruo, Skill Grouping Method: Mining andClustering Skill Differences from Body Movement
BigData, Proceeding of International Conference on
BigData 2015, IEEE (2015).
[6] Y. Miyadera, K. Kurasawa, S. Nakamura, N.
Yonezawa, and S. Yokoyama, A Real-time Monitoring
System for Programming Education using a Generatorof Program Animation System, Journal of Computers,
Vol.2, No.3, pp.12-20, Academy Publisher (2007).
[7] K. Tanigawa, D. D. Phuong, F. Harada, and H.
Shimakawa, Grasping Learned Items in C
Programming Exercise Using Function Call Logs,
IEICE Transactions on Information and Systems(Japanese edition), Vol.95, No.12, pp.2079-2089,
IEICE (2012).
[8] Study Group on Polytechnic Science, Introduction to
Polytechnic Science, JUSE Press Ltd. (2018) (inJapanese).
[9]
Foldermonitoring,tukaeru,
URL<http://www.saberlion.com/tukaeru/soft/folders.html>(2016).
[10] Ubuntu manuals, Ubuntu Manpage Repository, URL<http://manpages.ubuntu.com/manpages/bionic/man1/i
notifywait.1.html > (2018).

146

Session 6:
Systems and Applications( Chair: Ohki Tetsushi )International Workshop on Informatics ( IWIN 2018 )Interactive topic confirmation on decision support system
for information dissemination*
Kei Utsugi , Hajime Mori*, Kiyohiro Ohara*, Masahide Okamoto* and Yuuki Shimizu**
Hitachi Ltd., Japan{kei.utsugi.nz, hajime.mori.vh, kiyohiro.obara.pc, masahide.okamoto.ex, yuuki.shimizu.kf}@hitachi.com
Abstract - A decision-support system for quality-assurance

similarity of the document describing the target issue andother existing documents describing similar issues in eachdivision (Figure 1). The basic methodology for finding
appropriate departments is to search for similar documents,
namely, selecting similar technical documents to the one
with the current context. Many basic technologies forfinding similar documents have been proposed, but many
practical problems must be overcome before they can bepractically applied directly.

activities using existing business documents was
implemented as a prototype, and its novel features wereexperimentally evaluated. The prototype system helps theuser decide to which division within an organization
information of the current problem should be disseminatedby using the similarity between the problem document andexisting documents held by each department. The novel
point concerning the prototype system is its user-interaction
methodology used in recommendation process. The systempresents feature terms for explaining classifications given by
local interpretable model-agnostic explanations method andfinds other related documents from feature terms in asmooth interaction. The user-interaction ensures the user’s
perspective and attention from behavior of checking system
output, the perspective and behavior of the user are includedin the output of the prototype system and a user’s action logs
can be obtained and used to analyze how expert users readand check the target problem.

Keywords: Computer-supported collaborative work, Humaninteraction.

1

INTRODUCTION

Quality management of products and service is a highpriority for every responsible company. Such companies arealways devoted to learning from information about cases offailure through the whole product life cycle—from designand development to production, operation. They believe that
good decisions always rely on facts and experience, andanalysis of failure cases is the best teacher for those whobuild new products and processes. However, as the amountof information on case studies increases, it becomesincreasingly difficult for a single person to grasp a whole
picture of that information. To make full use of suchinformation, the help of a technology called “natural
language processing” (NLP) is required. In the present study,
to meet that requirement, a prototype of a decision-support
system for quality assurance (QA) activities utilizing
existing business documents was proposed and evaluated.

Figure 1: Use case: dissemination of information
concerning quality-assurance activities
Hereafter, this problem is mainly dealt with from apractical perspective, and examples of our prototype
construction are described. Especially, solving the problem
involves addressing two challenges. First, it is necessary toestablish a methodology for a mechanical decision-support
system based on document information accumulated inevery organization so far. The other challenge is to establish
a method for accumulating know-how in preparation for thestaff exchange accompanying generational change. To
address these two challenges, the following two approaches
are proposed:
1) Document similarity measurement and search system
based on grouping of technical terms.
2) Getting feedback from users’ behavior to confirm thereason for a judgment resulting from machinelearning and improve the next judgement.
Although these two challenges are different problems on
the surface, in terms of implementation, either process canbe represented as a technique that deals with elements thatdefine a topic of a document. The first one is a generating a
topic from a document corpus, and the second one isevaluating the topic.

Current QA activities have been basically based on
empirical knowledge by experts who are proficient inhandling information within an organization. However,given that changes in organization and replacement of
talented people are inevitable, the methods that rely uponsuch experience and empirical knowledge involve the risk ofoverlooking the target department in which the alert
information should be deployed.
The prototype system assists in finding divisions thatshould disseminate the target issue on the basis of the149International Workshop on Informatics ( IWIN 2018 )In the following chapters, the techniques used in our
prototype decision-support system for quality assurance
activities utilizing existing business documents and reports
is introduced. Section 2 describes related research. Section 3
introduces some ideas established for this development.
Section 4 describes the structure and procedure of the
prototype system and show actual system images. Section 5
presents the results of a basic experimental evaluation of the
prototype system. Note that the novelest point concerning
the system is its user-interaction methodology used in therecommendation process (see Sections 3-4) that
1. shows feature terms for explaining classifications byusing LIME [8];2. finds other related documents from feature terms used
in intuitive interactions.
Such interactions ensure the user’s explicit attention forchecking recommendation results and provide user’saction logs that are useful for analyzing the thinking
process of expert users.

2

WMD differs from Doc2Vec in that document distances are
entirely unsupervised and a mechanism for incorporatingsupervision is unavailable [6].
As a requirement of this application subject, the prototype
system requires a means to explain document classificationa posteriori, and a means to give feedback from humanbehavior as supervision.

2.2

The classification decisions made by machine-learningmodels are quite difficult to explain to non-data scientists.
Ribeiro’s Local Interpretable Model-Agnostic Explanations(LIME) [7] is a method for making these models at least
partly understandable. In the case of LIME, a model
function is approximated by locally fitting linear models topermutations of the original training set. At least in the local
approximation, the top-n-most-significant features for theclassification can be shown. The explaining processes weremainly used by data scientists to validate the model. Our
attempt in this study is to incorporate the explanation
method into a user interface for daily use and to utilize it toobtain feedback from users.

RELATED STUDIES
2.1

Explaining classification

Document-similarity metrics
3

In the research field of NLP, the most-classical ways torepresent a document are to use a bag-of-words (BOW) andterm-frequency inverse document frequency (TF-IDF) in
which each document is mapped into a vector space. These
representations, however, cannot capture relations between
different but similar words. There are several methods that
attempt to circumvent this problem by mapping them into
lower-dimensional representations. For example, latentsemantic indexing (LSI) [1] and latent dirichlet allocation(LDA) [2] make vector spaces in which similar words are
mapped into close points probabilistically, however, theeffects are limited.
In 2013, a novel word-embedding procedure, called
word2vec (W2V), was introduced by Mikolov et al. [3] This
procedure makes a model that learns a vector representationfor each word by using a shallow-neural-network language
model and generates a representation of word embedding (orword vector space) of unprecedented quality and scales
naturally to very large data sets. While W2V is widelyadopted in most NLP projects, methods for providing
document-similarity metrics are still controversial. For
example, Doc2vec (paragraph vector) [4] is an extension of
W2V and associates arbitrary documents with labels. As forthis method, each document is assigned to a specific point ina document vector space. Although it seems a natural
implementation, its results depend on the feature of theoriginal label in the learning process. Also, it is impossible
to know which element in the document defined the distance.
Word movers’ distance (WMD) [5] is another extension of
W2V. WMD represents a document as a set of embedding
words in W2V’s word vector space, and it makes adocument metric by formulating the distance between two
documents as an optimal transport problem between theembedded words. Theoretically, in WMD, it is easy toanalyze the influence of each word on the distance between
documents, but its processing cost is very high. Furthermore,

METHODThe goal of this study was to build a prototype system thatclassifies a given document that describes a target problem
to particular divisions in a company in accordance withinformation from existing technical documents. Taking
existing documents as data sources, the constructed
prototype system uses a document-similarity metric forsearching documents with various technical terms from agiven corpus and automatically recommends not only
documents but also divisions in which they should be
deployed.
Before the prototype system is described in detail, current
business activities and the target problem are described in
Section 3.1, and the proposed method is explained in
Section 3.2.

3.1

Target activity

A traditional and typical workflow of the target businessactivity, namely, a problem-checking process, is illustrated
in Figure 2. When a new problem concerning existingproducts is encountered, members of staff of the QA
department ask all other departments if they have a similarrisk potential and explain the problem in detail andrecommend countermeasures. In this department-searchingprocess, QA staff elicit empirical knowledge from experts
who are proficient in handling meta-level information within
the organization (Figure 2).
However, when the organization inevitably changes, andtalented people are replaced, the methods that rely uponsuch experience-based knowledge risk overlooking thetarget divisions in which the alert information should be
deployed. Especially when the structures of the organizationand human assets have been drastically changed owing to,150

International Workshop on Informatics ( IWIN 2018 )3.3

for example, the merger with another company, there is a
big risk of information development.
ProblemDefinition

Meta-expert
Problem

Measuring the similarity of documents is a “situationdependent” issue. Rather than using all words for judging
the similarity, it is better to extract the specific technicalterms related to the concerned context and to judge thedocument similarity on the basis of the similarity measure.
Such a measurement process would make a more reasonable
judgment of similarity. For example, under the assumptionthat groups of different corpus types are gathered together,such as design documents and operation manuals, accidentcase documents, the frequency of occurrence of terms varies
from corpus to corpus. To extract similar documents fromthese different corpora, it is effective to select thecharacteristic terms with more emphasis on the vocabularyelements commonly used across these corpora than to selectterms locally used in each corpus. The method for extracting
such a vocabulary set was studied and is described asfollows.

Department
In charge

HRknowledge

Problem・・・？

Figure 2: Current process for disseminating QA

3.2

Concept and outline of proposed system

Topic grouping
Term groups, named topics, are generated as follows. 1
Each topic consists of a main technical term (seeds) andseveral related technical terms tied by a particular featurethat can be understood by humans. An example of such
topics is shown in Figure 4.

The challenge in this study was to develop a system that
identifies divisions that deal with similar cases and possibly
have similar problems from document information on thebasis of NLP. The fundamental framework of the proposedsystem is shown in Figure 3. It has two functions: to searchfor documents with similar context to the current context
from a database of technical business documents (“1:Similar case finding” in the figure) and present the identified
documents to the user. A new and additional function forsearching for departments requesting confirmation ofsomething on the basis of the similar documents (“2:Dissemination support”) is introduced.

Design review
support system

Similar failure casesearch support

Figure 4: Example of topics (in Japanese)
In each document, the similarity of sentences is judged by
the presence or absence of these topic terms in concerned
sentence and the number of occurrences of that topic terms
in each document. Each topic group is managed in the
database and have a importance value that is updated withfeedback from users’ actions.

Dissemination ofproblemApplication
API
Function①Similar case finding-input：document-output：similar document

Technical term form docs

Database

Similar term grouping

Topics and document similarity②Dissemination support-input：document-output：division recommend

Generating a topic
A topic definition is composed of one representativetechnical term, a seed selected by a rule-based method, andgroups of words selected as related terms in word2vec space.

Technical feature
extraction

Figure 3: Total concept of the proposed system.

Selecting a seed of a topic
In regard to the handling of documents, management oftechnical terms is very important. Technical documents usedin manufacturing industries include a lot of specializedcompound words, abbreviations, code numbers, etc. that are
rarely used in general documents. These special terms arevery important factors in defining similarity of such kinds of
documents.

We describe the issues of the method along the following
three points. The three key challenges to overcome indeveloping the proposed system are summarized below.
1. Document topics and similarity: How to definefeature topics and similarity of documents (Section
3.3).
2. Modeling of classifier: How to extract therelationship between feature topics and specificdivisions (Section 3.4).
3. Interaction and feedback: How to design the userinterface of the system for enabling effective
feedback of expert’s knowledge (Section 3.5).

※The dimensionally compressed language model such as
LSA[1] or LDA[2] is also called TopicModel, but topic wedefine here is a grouping of semantic information of finer
granularity, and used in the meaning of topic tagging groupproposed in [9].
1151International Workshop on Informatics ( IWIN 2018 )The seed of each topic is a special technical term selectedby various rule-based methods as listed below.
 Technical terms extracted from existing glossaries usedin the business domain.
 English words expressed in Japanese katakana script.
 English abbreviations found in a Japanese corpus
 Various code numbers (e.g. product ID codes, project
ID codes, and so on) Composite technical terms by continuation of kanji(extracted from documents by a statistical method [7])The attribute tag is used in the characterization of theclassifier (described in Sections 3.4 and 3.5).
Measuring similarity of document on the basis of topics
Measuring similarity of documents on the basis of topics isdefined in Figure 6 as the following procedure.
1)
Transform the bag-of-words representation into a
topic matrix by linear transformation.
2)
When a user selects a specific topic,① add a weighting multiplier② select similar topics
3)
Take the dot product of the vectorized topics andperform threshold processing for each component.

Japanese technical documents used in business containmany specialized compound words, abbreviations, code
numbers, etc. that are rarely used in general documents.
They also include not only compound words with a series ofkanji characters but also technical terms based on
expressions in which English is written in katakana or aphonetic notation of foreign words. The characteristics ofthese Japanese documents can provide hints for retrieving
technical terms.

Document

3.4

Topic

Topic

Topic

Topic

Topic

Topic

Topic

Topic

Topic

Document

Classification

For judging which department should be announced to theconcerned case, a classifier is constructed so that its input is
an existing document vectorized by topic expression and its
output is a label (i.e.,, the name of the division that handled
the case in question).

Word2Vec Space 2
Word_1

Word_2

New problem：“Parts XXX contains problem
YYY. Checking ZZZ solution”

Word_2

Word_3

Word_4
Word_5

Decision support：「Check division YYY」
div. YYY：document 00X
sect.ZZZ： document AYY

Find robust similarity

Recommend
division info

Input documents

Word_7
Word_4
Word_5
Word_6

Topics(Word Group)

Topic

Topic

Evaluating topics on the basis of feedback
Each topic group is continuously maintained, and its
evaluation value is updated with human feedback
information.

Word_3

Word_8

Topic

Topic

Corpus A + Corpus C

Word2Vec Space 1
Word_1

Topic

Topic

Figure 6: Document similar based on topic method.
Measuring similarity of documents on the basis of
topicsSelecting related terms
The neighboring words of the seeds in each of the W2V
spaces with different learning circumstances in common are
selected as topic group words. Terms related to the selected
species are then generated as follows.
1) Generate multiple Word2Vec model spaces fromdifferent kinds of corpus. By changing the combinationof corpus used for learning and the order of learning, aW2V space of different learning models is made.
2) In each W2V model, the top-N vocabulary of the
distance close to the species word is selected. (In thisexperiment N (number of words) is about 30.)
3) Pick up terms that are commonly selected in all W2V
spaces as relatives of seeds.
4) Calculate the weight to each relative term. (The weightis a function of the distance between the seed and theterm and the frequency of occurrence of the term.)
Corpus A + Corpus B

Topic

Word_9Check each words’ IDF

Failure case

Designdocuments

Generaldocuments

Discerption
Situation：”….”
Problem：”….”
Action：”….”

Description
Background：”…”Check:”….”
Comment：”….”
Action：”….”

Description(Free format)

Word_7
Word_9Word_8

Word_6

Topic group

Topic Seeds(Important words )Account division

Division：○○Section：〇〇Figure 5: Extracting topic groups from multipledistributed-word vector space.

Account division

Division：○○Section：〇〇Topic extraction from each case
→ Topic Vector

Holding division

Division：○○Section：〇〇Topics [a,b,c..]Related division→LabelLearning

Related division [oo]Figure 7: Classification of related departments byusing existing document data.

Grouping attributes of topics
The prototype system uses about 30,000 kinds of topics.
To support human understanding, 72 kinds of attribute tagsfor grouping topics were made. Each topic is assigned someof the attribute tags. The assignment is derived from therule-based estimation. Each attribute tag represents a groupof technical topics, e.g., “heat,” “fluid,” and “monitoring.”

As for the proposed system, multiple classifiers, each called
an agent, are prepared, and each agent judges a divisionindependently. To support human understanding, each agent
specializes in a particular area, after which it is named. For
example, "agent of thermal attribute" gathers only
vocabulary related to elements such as heat, temperature,
152

International Workshop on Informatics ( IWIN 2018 )gas, control, friction, melting, and output, and judges theirclassification independently.
The system consolidates the judgments of the agents tomake an integrated judgment. The purpose of dividing the
judgment task into multiple agents is to make an opportunity
for human interaction. It enables to record information onwhat kind of agent’s opinion is confirmed by the user and toobtain information of the perspective in the judgmentprocess. As a result, information on what kinds of agent’s
opinion are confirmed by people can be recorded, and theagent’s judgement process can be put into perspective.
Input textTermextract

term

Topic
expansion

Classification

term

Explanation

term

Filter 1

term

term

term

term

term

searchAgent１term(b) Selection of topics
The system displays the topic terms that are importantreason elements in the judgment of the selected classifier.
The method for selecting the terms explaining this decision
is used method. LIME [7] was used to select terms that
explain the reason for this decision. From the explanatory
topics, the user selects some important topics that need to be
confirmed and searches for related documents.

Find SimilarCasesimilarterm

term

attribute that it is responsible for. The classifier filters thegiven data and selects only those words that are relevant tothe agent’s own attribute and topics, and it performs the
judgment processing of the related department on the basisof the screened data. The overall result is judged from theresult of the processing of each classifier. On the other hand,from the classifiers, the user selects a particular classifierthat is similar to their own perspective of interest, andchecks the topic that became the reason for judgment.

Similar
case

text

term

similar
term

term

term

term

term

(c) Selection of existing documents
The system searches for similar documents that contain theselected words and displays the documents in a group of
related departments. The user checks the contents of the
documents held by the recommended department and selects
the documents that may need to be specifically confirmed.
To the selected document, the user adds a comment to thedepartment’s staff related to the topic, and then the system
adds the document and comment to the checklist to ask for
confirmation.

Agnent2
term

Filter 2

Figure 8: Process for searching for divisions anddocuments by using classification agents.

3.5

User interaction and feedback

A distinctive point concerning the proposed system is theinteractive process that explains the reason for therecommendation of target departments The entire userinteraction process is designed to answer the following threequestions and store the answers as operation logs:

TargetDocument

Classifier based on
Heat related terms

General process(LIME)

Similar casesearchUser log

Feedback update

Parameter update(Feedback)

Human
ActionAutoprocess

Figure 10: Procedure for feedback from the user
By logging the user’s UI operation by the above three
processes, it is possible to estimate what kind of theperspectives (classification agents), topics, and documents
are used to determine the expert users’ checking points.

Context
DetailDecision Explanation

Topic
selection

Evaluation update

The flow of the process for the user to check the current
problem is illustrated in Figure 9.
Division
Recommendation

Topic
Suggestion

Topic setting(Manual)(a) Which perspectives are used in classification?
(b) Which topics (terms) are focused on by users to findsimilar problems?
(c) Which documents identified the target department?

extracting
topicsTopic
Extraction

Document 1

Term1
Current
Problem(sentence)

Classifier based on
Fluid related terms

Document 2

Document 3・・・Classifier based on

Term3

Failure mode

Term4

related terms(a) Selection of Perspective

4

Term2

Search(b) Selection of TopicAPPLICATION OF PROPOSED SYSTEM

Document 4

The prototype system operates according to the following
procedure in three stages.

(c) Selection of
Existing Documents

Preparation stage: Each document is labeled by eachaccording to its corresponding division and stored in the
database. Topic elements are then extracted from the
documents. The classification agents learn the feature termsof each department on the basis of the extracted informationin advance.

Feedback : decision making information

Figure 9: Process for user interaction. Among agents’
explanations, a topic selected by the user is used forchecking related documents.
(a) Selection of perspective
Several topics are extracted from the subject problem, andthe topic vector obtained is applied to the classifier of agents.
There are several agents, and each one is assigned an
153

International Workshop on Informatics ( IWIN 2018 )Sending stage: A concerned document to be disseminatedand additional topics given by a QA stuff (sender) arerepresented as a topic vector. The documents related to this
topic vector are used as a search key, and the relevantdepartment candidates to contact and the related documents
are confirmed.

Recommendation
From each agent

Receiving stage: In each department, the recipient checks
the incoming message. At that time, the system explicitly
presents the documents and topics referenced by the sender.
It supports the consideration of the problem on the recipient.

Division A, Division B Division C Division D Division E Division F Division G

Grouped by division

Figure 13: Display image of classification step.

4.1

Procedure and interaction

Step 3. Check explanation terms
Using LIME, each agent presents topics that have a great
influence as “explanation.” These topics are limited to the
vocabulary that may be related to the agents' attributes. As
shown in Figure 14, the user can check each agent's
judgement result and the reason for that judgement.

The process of the sending stage is shown in detail in
Figure 11. The proposed system works along five steps tohelp the sender find the target. The sending stage consists ofthe following five blocks.
Preprocess

Sending

Action & Report

Agent nameInteraction for explaining the reason for judgment
Problem definition

＊＊＊
＊＊＊

(a) Selection(b) Selection(c) Selection

Send requestof perspectiveof topicsof documents

message

Classification result＊＊＊

DRcorpus

Significant topics

＊＊＊
＊＊＊
＊＊＊

＊＊＊
＊＊＊
＊＊＊

Figure 11: Procedure of sending process

Figure 14: Display image of explanation step. The
user checks topic terms to be focused on.

Step 1: Input problem
An example of the input screen is shown in Figure 12. The
user inputs a sentence to define the context of the problem.
Topic keywords are extracted from the input sentences anddisplayed on the screen. From them, the user selects topics
to be focused on.

Step 4. Checking for related documents
By selecting the topic, After the topic is selected, thedocuments that include the topics are retrieved from the
document database. Examples of a UI for checking these
documents are shown in Figure 15. The chart shows the
documents in a timetable grouped according to department.
The system logs the referenced documents and gets
feedback for verification data on the importance of the
vocabulary.

Input text

Doc
title

Doc
title

Time

Extracted
Technical
topics

Doc
title
Doc
title

Figure 12: Display image of input step. Technical
topics are extracted from the input sentence.

Time-series displayof related documents

Doc
title
Doc
title

Doc
title
Doc
title
Doc
title

Figure 15: Display image of drill-down step. The user
checks related documents in this phase.

Step 2. Check classification result
By comparing the extracted keywords and the document of
the accumulated past cases, the system recommends
possible relevant departments presumed to cope with the
problem. Each agent recommends departments
independently, and an overall recommendation is madeaccording to the total judgments of all agents.

Step 5. Dissemination of documents to target division
The system converts the list of checked related documents
during the operation into a message document for theconfirmation request to the target division.

154

International Workshop on Informatics ( IWIN 2018 )

Doc
title

Doc
title

Table I lists conformity rates of the label that was assigned
by a user using the evaluation value given by this system.
The rate of conformity is less than 58% on average.
Compared with the existing method, the proposed method
shows an average improvement in precision of 30%.

The systemgenerate
Generateda template request
message Message
for local division

Time

Doc
title
Doc
title

Problem detail
Doc
title

lay
ents

Doc
title
Doc
title

selectDocuments
to be checked

Experiment 2
It was experimentally confirmed whether the subject andtarget department can be identified from the occurrence
frequency of the topic of the business document. The total
number of documents used in the test experiment was 1954,
separated into 1600 data for study and 354 test data. As forthe number of divisions that were classified, divisions wereclassified into 23 categories, and the sections (sub group ofdivisions) were divided into 136 categories.
The results of experiments on identifying related divisions(23 departmental units) from each sentence are listed inTable 2. "Top choice" is the rate at which the highest rating
was the actual responsible department. "Three choices" is acase in which three possible related departments arerecommended, and the actual responsible division wasamong the three options. Other results when each agent
classified cases into 136 sections are listed in Table 3.

Figure 16: Display image of dissemination. The
system makes a request template message for local
divisions.
Finally, the user sends the confirmation documents createdby the above process to the contact person in eachdepartment.

5

EXPERIMENT

As a basic experiment to evaluate the basic concept of the
proposed system, the following contents were evaluatedquantitatively: (i) the effect of using the proposed method(topic grouping) and (ii) whether the subject and the target
department can be identified from the frequency ofoccurrence of the topic of the business document.

As a goal set at the start of this study, a target of 70% was
set as the provisional recommendation accuracy. However,
the actual value is only about 60% when therecommendation value of top choice is considered, and it isinadequate for the process for estimating automatically(without human assist) especially in units of departments.
However, from perspective actual operation, since humaninteraction is carried out from people interact in selecting
three types of recommendation candidates to determine thedeployment destination, it is important to select the top-threeproblem areas.
At the present stage, in terms of the accuracy required for
QA activities, we could not obtain sufficient accuracy byautomatic recommendation alone, and human interaction(focusing on appropriate topics) is necessary for practicaluse. In future works, we must conduct an experiment toaccumulate information by actually introducing the
proposed method in the field and verifying its effect on
improvement of classification

Experiment 1
A case study of a similar-case search in the design-review
business is explained as follows. In this evaluation, theevaluation score obtained with the proposed method and thatobtained with an existing similar-document-search method
were compared. The existing search engine used as thebenchmark is based on associative-search technology [8],which selects feature words from a given document groupand selects the similar documents from that group. It ishighly regarded as a general-purpose similar-document
retrieval system. This approach is similar to that of the
proposed content, and the difference is that the words are
selected directly by using W2V without using a topic group.
The effect of the topic group that was adjusted as abenchmark for this existing system was verified.
In this evaluation, four kinds of design-review tasks weretargeted. First, 50 corpora considered to have a highprobability related to the target document were extracted.
For the primary-extraction data, the user decided whether
each corpus was related to the context of the design review
and labeled the success and failure.

Table 2: Classification accuracy (division: 23 labels)
ThreeClassifier unit nameTop choicechoices
Code
0.48
0.681
Term (general)
0.668
0.844
Topic (general)
0.603
0.857
Heat
0.467
0.772
Parts
0.584
0.792
Fluid
0.525
0.688
Failure and malfunctions 0.454
0.714
System design
0.467
0.746

Table 1: Comparison of results of similar-document searchby proposed (topic-based) and existing method.
Proposed
Existing
precision
precision
Improvement
Case#1
0.36
0.36
1.0
Case#2
0.5
0.44
1.13
Case#3
0.95
0.8
1.18
Case#4
0.52
0.26
2.0
average
0.583
0.465
1.328

155

International Workshop on Informatics ( IWIN 2018 )Table 3: Classification accuracy (division: 136 labels)
Top
Three
Classifier unit namecorrect
choices
Code
0.415
0.623
Term (general)
0.617
0.72
Topic (general)
0.558
0.766
Heat
0.467
0.668
Parts
0.526
0.727
Fluid
0.454
0.636
Failure and malfunctions 0.481
0.642
System design
0.396
0.59

6

[5] M. J. Kusner et al., From word embeddings to
document distances. ICML 2015, Vol. 37, pp. 957-966,(2015)[6] G. Huang et al., Supervised word mover's distance,
NIPS'16, pp. 4869-4877, (2016)[7] M. Tulio et al, Why Should I Trust You?: Explaining
the Predictions of Any Classifier, ACM SIGKDD 2016,
pp. 1135--1144, (2016)[8] H. Nakagawa and T. Mori, A simple but powerfulautomatic term extraction method, Computerm2,
COLING 2002 Workshop, pp. 29–35, (2002)

CONCLUSION

A prototype decision-support system for QA activity was
proposed and experimentally evaluated. The proposedsystem classifies documents that describe target problems
and assigns them to particular divisions in accordance withthe similarity of existing technical documents. It also utilizes
a description of the machine’s decision as a clue to userinteraction.
The goal of this method is to improve performance byhuman feedback. However, long-term accumulation of user
data in the field operation is indispensable in regard to thisgoal. At present, we are working on an experimental phasein the field with various additional developments andbrushing up the method for data accumulation andperformance improvement.
There is concern that AI supporting human judgment may
interfere with an expert user’s sense of responsibility. If thejudgment of AI reaches a reliable level, confirmation of
judgment by the user might become a ritual. As a means to
avoid this situation, we think that it is important to create aprocedure to obtain feedback that uses “checking reason for
a machine’s judgment” for “another searching process” andinvolve the expert users in order to accumulate informationof know-how. This prototype will be a typical case of such
UI procedure.

REFERENCES[1] T. K. Landauer and T. S. Dumais, A solution to Plato's
problem: The latent semantic analysis theory ofacquisition, induction, and representation ofknowledge. Psychological Review, Vol. 104 No. 2, pp.
211-240. (1997)[2] D. Blei et al., Latent Dirichlet Allocation, Journal of
Machine Learning Research, Vol. 3, pp. 993–1022.
(2003)[3] Y. Goldberg and O. Levy, word2vec Explained:
Deriving Mikolov et al.'s Negative-Sampling WordEmbedding Method, Cornell University Library (2014)[4] Q. Le and T. Mikolov, Distributed Representations ofSentences and Documents, PMLR, Vol. 32, pp. 11881196, (2014)156

International Workshop on Informatics ( IWIN 2018 )Initial Attempt of Bluetooth Low Energy Water-Level Estimator
Ryo Orihara† , Shigemi Ishida† , Masahiko Miyazaki† , Shigeaki Tagashira‡ , and Akira Fukuda†
ISEE, Kyushu University, Fukuoka 819-0395, Japan{orihara, ishida, miyazaki, fukuda}@f.ait.kyushu-u.ac.jp‡
Faculty of Informatics, Kansai University, Osaka, 569-1095, Japanshige@res.kutc.kansai-u.ac.jp†

Abstract In recent years, torrential downpours in a limited area occur frequently. To avoid flooding damages, it is important tocontinuously monitor the changes in river conditions, especially changes in the water-level. This paper proposes a lowcost water-level estimation method using off-the-shelf BLE(Bluetooth Low Energy) devices. BLE uses a 2.4-GHz band,
where the radio signals decay at a rapid rate in the water as
we increase the communication distance. We measure RSS(received signal strength) of BLE signals and estimate theriver water-level. The influence of RSS fluctuations are reduced by employing the RSS difference between two BLE
beacons installed in and by a river. We conducted an initial
experiment to confirm the feasibility of our water-level estimation method. The experimental evaluation demonstrated
that our water-level estimator successfully estimated waterlevel within a six classes of water-level with an absolute meanerror of 2.75 centimeters.

This paper proposes a water-level estimation method using
off-the-shelf BLE (Bluetooth Low Energy) devices. BLE uses
a 2.4-GHz band, where the strength of radio signals decays at
a rapid rate in the water as we increase the communication
distance. We measure RSS (received signal strength) of BLE
signals sent from a BLE beacon in a river water using a riverside BLE module and estimate the river water-level based on
the RSS. Temperature or environmental changes may affecttransmission and reception circuit operations, which resultsin changes of RSS. We therefore employ the RSS difference
between two BLE beacons installed in and by a river. We also
employ packet loss rate for water-level estimation because theestimation performance might degrade due to the decrease of
the number of received packets as the water-level increases.
To verify the feasibility of the water-level estimation system using BLE, we experimentally evaluated the water-level
estimation system. We changed water-level in six steps in abathtub and collected RSS samples. The 10-fold leave-oneout cross validation demonstrated that our water-level estimation system successfully estimated water-levels in six steps
with an accuracy of 85.3 %, which resulted in an absolutemean error of 2.75 centimeters.
Specifically, our main contributions are three-fold:

Keywords: BLE (Bluetooth Low Energy), RSS (received
signal strength), water-level

1

INTRODUCTION• We propose a new water-level estimation system thatuses low-cost off-the-shelf BLE beacons. The waterlevel estimation system estimates water-level by machine learning using RSS (received signal strength) ofadvertising packets sent from BLE beacons.
• We present an initial design of the water-level estimation system that relies on two BLE beacons. Using two
BLE beacons, we reduce the influence of RSS fluctuations.
• We show the feasibility of our BLE water-level estimator by initial evaluation conducted in a bathroom.

In recent years, torrential downpours in a limited area occur
frequently. The heavy rains sometimes cause river-bank collapse resulting in flooding disasters. Flooding damages not
only houses but also people who fail to evacuate from thedamaged area. To avoid such damage, it is important to continuously monitor the changes in river conditions, especiallychanges in the water-level.
Water-level is manually monitored at many rivers in Japan.
To monitor river water-level in realtime, water-level monitoring system using sensors such as floating and bubble waterlevel gauges have been deployed at some rivers. The automatic water-level monitoring systems, however, come withhigh installation and maintenance costs, resulting in limited
number of deployment. Monitoring of the river water-level
in Japan is mainly carried out on class A rivers designatedby the MLIT (Minister of Land, Infrastructure, Transport andTourism) and class B rivers designated prefectural governor.
For a total of 21,004 rivers managed by prefectures, 4,986
sensors have been installed, which indicates that less than
25 % of rivers are covered by automatic water-level monitoring systems.
In the heavy rain in northern part of Kyushu in July 2017,32 rivers managed by each prefecture were flooding and caused
serious damages. 30 rivers among the 32 lacked water-level
gauges. Although many local governments are aware of thenecessity of water-level gauge, high installation and maintenance costs of water-level gauges prevent the deployment.

The remainder of this paper is structured as follows. Section 2 shows an overview of water-level estimation systemusing BLE and issues in a practical environment. Section 3
describes a design of the water-level estimation system in apractical environment and Section 4 conducts an initial experimental evaluation. Related work is presented in Section 5.
Finally, Section 6 concludes the paper.

2

BLE WATER-LEVEL ESTIMATIONSYSTEM

Figure 1 depicts an overview of water-level estimation system using BLE. In the water-level estimation system, BLE
beacon and receiver are installed in a river and by the riverside, respectively. The BLE beacon periodically sends advertising packets. We receive the periodic advertising pack157

International Workshop on Informatics ( IWIN 2018 )Machine-Learning Classiﬁer
Reference BLE

Training Phase

RSS (dBm)

20

RSS Samples

PreProcess

SVM

ActualWater-Level

PreProcess

SVM

0

-20

0

0.5

1.0

1.5

2.0

Time (sec)

OutputEstimation Phase
BLE beacon

Target BLE
Machine-Learning
Classiﬁer

Figure 3: Overview of water-level estimation system

Receiverets.

Figure 1: Overview of water-level estimation system

3

EXTEND TO PRACTICALENVIRONMENT

3.1

Basic Idea

We address the two issues presented in the previous section
based on the following two basic ideas.
1) Utilize RSS difference:
We utilize RSS difference between two BLE beacons
installed in and by a river to estimate water-level. Using the RSS difference, we can remove the influence of
operational changes in a receiver circuit.
2) Utilize packet reception interval:
We use packet error rate as one of the features for waterlevel estimation to improve accuracy. Although packet
error rate is not always independent of RSS of advertising packets, packet error rate is also affected by factors other than the decrease of RSS. There is no API
defined in BLE standards to retrieve packet error information [1]. In addition, BLE beacons add a random delay on every transmission of advertising packets, whichmakes difficult to calculate packet reception rate fromthe number of received packets within a fixed duration.
We therefore use packet reception interval instead ofpacket error rate.

Figure 2: Variation of RSS
ets using the BLE receiver and measure RSS (received signal
strength) of the packets. BLE uses a 2.4-GHz band and thestrength of radio signals decays at a rapid rate in the water.
As the water-level increases, RSS of the advertising packet
decreases. We estimate the water-level from the RSS at thereceiver by a machine-learning classifier.
In a practical environment, we need to solve the following
two issues.

3.2

1) How do we reduce the effects of RSS fluctuations?:
Even when the water-level is unchanging, RSS is too
unstable for water-level estimation. Figure 2 shows anexample of RSS of advertising packets sent from a BLE
beacon installed in a bathtub as a function of time. Wechanged water-level from 0 to 50 centimeters with 10centimeter step and collected RSS samples. Intuitively,
RSS decreases as the water-level increases. Figure 2,
however, shows that the intuition is sometimes not true
because of the RSS fluctuations. The RSS fluctuations
are mainly caused by changes in radio propagation environment and in operations of transmitter/receiver circuits.
2) How do we reduce the influence of RSS error due todecrease in the number of received packets?:
RSS of packets sent from a BLE beacon in a river decreases as the water-level increases, which results in increase in packet error rate. In such a case, water-level
estimation error increases because of the RSS measurement error due to the limited number of received pack-

System Overview

Figure 3 shows an overview of the water-level estimation
system. As shown in Fig. 3, the water-level estimation system
uses target and reference BLE beacons, and a BLE receiver.
A target BLE beacon is installed at the bottom or the target
depth in a river, and a reference BLE beacon is installed by theriverside. A BLE receiver installed by the riverside receives
advertising packets from the BLE beacons and measure thepacket RSS. The RSS data is then sent to a machine-learningclassifier that consists of pre-processing and SVM blocks.
The following subsections describe the each process in details.

3.3

Pre-Processing Block

Figure 4 shows an overview of a pre-processing module.
The pre-processing block reformats RSS samples to keep a
specific sampling rate and calculates the RSS difference between target and reference BLE beacons. BLE beacons are
periodically sending advertising packets with a random delay
158

International Workshop on Informatics ( IWIN 2018 )Pre-Process
TargetBLE
Reference
BLE

RSS
Data

MovingAverage

Packet Reception
Interval
RSS Difference

MovingAverage

Figure 4: Overview of pre-processing blockon each transmission. The RSS samples of advertising packets are averaged over a specific period to derive the specificnumber of RSS data. The number of RSS data is dependent
on packet reception errors and the random delay. We derive
the inverse of the actual number of averaged RSS samples asa packet reception interval.

3.4

Figure 5: Frequency components of RSS samples

SVM Block

An SVM (support vector machine) block consists of training and estimation phases. The input of the SVM block is avector with two components: the RSS data and packet reception interval which are passed from the pre-processing block.
In a training phase, test data that consists of the RSS difference, packet reception interval, and actual water-level ispassed to an SVM classifier to construct an SVM machinelearning model. We used a multi-class SVM classifier with
the RBF kernel as we changed water-level from 0 to 50 centimeters with 10-centimeter step in our experiment. The RBF
kernel parameter and cost parameter are optimized by gridsearch.
In an estimation phase, the water-level is estimated by the
pre-trained SVM classifier. The feature values, i.e., the RSS
difference and packet reception interval, are passed from thepre-processing block in the same manner as the training phaseand water-level is estimated within the six steps.
It is natural to use SVM regression for continuous waterlevel estimation. We, however, use SVM classifier as an initial
evaluation in this paper because unstable RSS might fail toconstruct a valid regression model. In the near future, wewill collect much more training data and will utilize SVM
regression.

4

Figure 6: Empirical cumulative distribution of frequency
components
components, respectively. The figures indicate that 99.4 %of frequency components were below 0.5 Hz. A BLE receiver receives a packet at least every two seconds. We determined to use 2-second window for moving average in a
pre-processing block.

4.2

INITIAL EVALUATION

Figure 7 shows the equipments used in our implementation. We used a MacBook Pro laptop running macOS Sierra
10.12.6 as a BLE receiver and two MyBeacon Pro (MB004
Ac-DR) BLE beacons, as shown in the figure. We implemented node.js program running on a BLE receiver to receive
BLE advertising packets from the BLE beacons. Water-level
estimation, i.e., pre-processing and SVM classifier are implemented using Python 2.7.13 with scikit-learn 0.19.1.

To verify the effectiveness of the water-level estimation
system using BLE, we conducted experiments in a bathroom
as an initial evaluation because the bathroom is easy to control
the water-level.

4.1

Implementation

Preliminary Experiment

A preliminary experiment was conducted to determine thetime duration of the moving average in a pre-processing block.
Frequency component analysis was performed on the RSS
samples using FFT (Fast Fourier Transform). To apply FFT,
sampling frequency must be constant. We first derived constant sampled RSS by linear interpolation because BLE beacons are sending advertising packets with a random delay oneach transmission. We then analyzed frequency componentsof the interpolated RSS samples.
Figures 5 and 6 shows the result of frequency component
analysis and empirical cumulative distribution of the frequency

4.3

Evaluation Environment

An experimental evaluation was carried out in a condominium bathroom. The bathroom setup is shown in Fig. 8.
Figure 9 shows experiment setup. Two BLE beacons were
installed as shown in Fig. 9: a target BLE beacon at the bottom of the bathtub and a reference BLE beacon outside of
the bathtub. A BLE receiver, namely, a laptop, was installed
above the bathtub.
159

International Workshop on Informatics ( IWIN 2018 )Table 1: Confusion matrix of water-level estimation result
Estimated [cm]
0
10
20
30
40
50

Class
Precision
Recall

4.4

Bathroom

160cm

Figure 8: Size of bathroom
We changed the water-level from 0 to 50 centimeters with
10-centimeter step and measured RSS of received packets for
10 minutes for the each water-level. Using the RSS samples,
we performed 10-fold leave-one-out cross validation to estimate water-level in six water-level classes. For each waterlevel, we randomly sorted the data. We then divided the data
for the each water-level into ten chunks, nine of which wereused for training and the remaining was used for testing.
Comparing the estimated results with the actual water-level,
we calculated precision and recall for each water-level classusing a confusion matrix. Macro precision and recall, which
are the mean of precision and recall for each class, were also
calculated. We also calculated macro F1 score from the macro
precision and recall defined as
2 · Precision · Recall.
Precision + Recall

Target BLE beacon

Actual [cm]
20
30
0
1
19
0
159
4
2
284
57
0
0
0

40
0
0
93
4
211
1

50
0
0
0
0
13
299

0
0.997
1.00

10
0.936
0.843

20
0.532
0.671

30
0.953
0.983

40
0.706
0.683

50
0.997
0.909

Classification Performance

Table 1 shows a confusion matrix of the water-level estimation result. The diagonal of the confusion matrix, i.e., graycolored cells, represents the number of TPs (true positives),
which are the cases that water-level is correctly estimated, foreach water-level class. We can see that the water-level estimation system exhibited good estimation performance in waterlevel classes of 0 and 50 centimeters. This was mainly caused
because the RSS of BLE signals above or under specific levels were easily estimated as 0- or 50-centimeter class, which
resulted in the good performance.
Table 2 shows precision and recall calculated from the confusion matrix for each water-level class. As the confusion
matrix implied, precision and recall for 10- and 50-centimeterclasses were quite high. Precision and recall were the highest for a 0-centimeter class. 20- and 40-centimeter classes
were difficult to be distinguished because RSS data of theboth classes had small difference. It is considered that the estimation accuracy of 20- and 40-centimeter class happened to
be low by chance, because the number of data was insufficientin this time. In the future, we plan to utilize RSS distribution
to estimate water-level.
Macro precision, recall, and F1 score were 0.853, 0.848,
and 0.851, respectively. Although the results were insufficientfor practical use, the results demonstrated the feasibility ofwater-level estimation using BLE beacons as we derived theestimation performance above random selections.

110cm

120cm

55cm

F1 =

10
0
280
26
8
18
0

Table 2: Precision and recall for each water-level class

Figure 7: Equipments used in the implementation

Bathtub

0
297
0
0
0
0
0

(1)

Reference BLE beacon

Figure 9: Experiment setup

Figure 10: Distribution of water-level estimation errors
160

International Workshop on Informatics ( IWIN 2018 )4.5

Water-Level Estimation Error

Wireless sensing technologies using commodity WiFi devices are extended to gesture recognition nowadays. WiGestleverages changes in WiFi signal strength to sense in-air hand
gestures around the user’s mobile device [13]. Smokey utilize
CSI changes to detection smoking motion [14]. Both technologies utilize change of radio signal propagation environment. Utilizing the signal processing methods presented inthese works to extract features, we may improve water-level
estimation accuracy, which is one of our future works.

We evaluated water-level error because our system not forclassification but for water-level estimation. The water-level
error "i for trial i is defined as"i = dei

di ,

(2)

where dei and di is estimated and actual water-levels for trial i.
Note that we estimated water-level within six classes of waterlevel. The error is therefore in {0, ±10, ±20, ±30, ±40, ±50}.
Figure 10 shows distribution of water-level errors. From
the figure, we can see that our water-level estimation systemcorrectly estimated water-level for 85.3 % of trials. An abP
solute mean error |"| = i |"i | was 2.75 centimeters. This
result greatly exceeded the result that an absolute mean error
was 4.79 centimeters when using a single BLE beacon, which
demonstrated the effectiveness of the two-beacon approach.
We believe that the results successfully demonstrated the feasibility of our water-level estimation method using BLE beacons.

5

6This paper proposes BLE-based water-level estimator asa new low-cost water-level measurement method. We conducted an initial experiment and confirmed that the BLE waterlevel estimator estimated water-level with small errors in acontrolled environment. In our future work, we conduct experiments in a practical environment.

ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant
Numbers JP15H05708 and JP17H01741.

RELATED WORKREFERENCES

To the best of knowledge, there is no study working on estimation of river water-level using off-the-shelf radio modules.
This section reviews studies on water sensing technologies
and sensing technologies using radio signals.

5.1[1] Bluetooth Special Interest Group, “Bluetooth specification version 4.2,” Dec. 2014, http://www.bluetooth.
com/.
[2] W. Sun, H. Ishidaira, and S. Bastola, “Calibration of hydrological models in ungauged basins based on satellite
radar altimetry observations of river water level,” Hydrological Processes, vol. 26, no. 23, pp. 3524–3537,
Nov. 2011.
[3] Y. Ji, M. Zhang, Y. Wang et al., “Microwave-photonic
sensor for remote water-level monitoring based on
chaotic laser,” Int. J. Bifurcation and Chaos, vol. 24,
no. 3, pp. 1–7, Mar. 2014.
[4] J. Kim, Y. Han, and H. Hahn, “Embedded implementation of image-based water-level measurement system,”
IET Computer Vision, vol. 5, no. 2, pp. 125–133, Mar.
2011.
[5] Y. Kim, H. Park, C. Lee et al., “Development of acloud-based image water level gauge,” IT Converg.
Pract.(INPRA), vol. 2, no. 1, pp. 22–29, 2014.
[6] W. Yang, X. Wang, A. Song et al., “Wi-wheat: Contactfree wheat moisture detection with commodity wifi.”[7] W. Wang and Y. Dai, “A grain moisture detecting system
based on capacitive sensor,” International Journal of
Digital Content Technology and its Applications, vol. 5,
no. 3, pp. 203–209, 2011.
[8] Z. Liu, Z. Wu, Z. Zhang et al., “Research on online
moisture detector in grain drying process based on v/fconversion,” Mathematical Problems in Engineering,
vol. 2015, 2015.
[9] S. O. Nelson, A. W. Kraszewski, S. Trabelsi et al., “Using cereal grain permittivity for sensing moisture content,” IEEE transactions on instrumentation and measurement, vol. 49, no. 3, pp. 470–475, 2000.
[10] K. Kim, J. Kim, C. Lee et al., “Simple instrument formoisture measurement in grain by free-space microwave
transmission,” Transactions of the ASABE, vol. 49,
no. 4, pp. 1089–1093, 2006.

Water-Level Sensing

Water-level sensing technology is not much studied in literature because there are many off-the-shelf water-level sensors such as floating and bubble water-level gauges available.
There is another form of water-level sensors relying on ultrasound, radar [2], and laser [3] available today. These approaches require high cost equipments or high installation andmaintenance costs.
Image-based water-level sensing technologies have beenreported [4, 5]. River Eye [5] installs cameras by the riversideand other system components are deployed as a cloud system,
which minimizes installation and maintenance costs. Combined with the cloud-based approach with our BLE-based sensing technology, we believe that we can much more reduce theinstallation and maintenance costs.

5.2

CONCLUSION

Wireless Sensing

There are some studies reporting wireless sensing technologies in terms of water sensing. Wi-Wheat presents a nondestructive and economic wheat moisture sensing system with
commodity WiFi [6]. Wi-Wheat utilizes amplitude and phaseshift of WiFi wireless links in CSI (channel state information) data derived from a commodity WiFi module. Machinelearning using SVM with features extracted by PCA (principle component analysis) on CSI data estimates wheat moisture level. The experimental results demonstrated the WiWheat system can achieve higher classification accuracy forboth LOS (line-of-sight) and NLOS (non line-of-sight) scenarios. The use of commodity WiFi drastically reduces thecost of moisture sensing compared to the existing grain moisture sensing technologies relying on magnetic or electric properties [7–12].
161

International Workshop on Informatics ( IWIN 2018 )[11] Y. Yang, J. Wang, C. Wang et al., “Study on online measurement of grain moisture content by neutron
gauge.” Transactions of the Chinese Society of Agricultural Engineering, vol. 16, no. 5, pp. 99–101, 2000.
[12] D. Nath K and P. Ramanathan, “Non-destructive methods for the measurement of moisture contents–a review,”
Sensor Review, vol. 37, no. 1, pp. 71–77, 2017.
[13] H. Abdelnasser, M. Youssef, and K. A. Harras, “Wigest:
A ubiquitous wifi-based gesture recognition system,” in2015 IEEE Conference on Computer Communications(INFOCOM), April 2015, pp. 1472–1480.
[14] X. Zheng, J. Wang, L. Shangguan et al., “Smokey:
Ubiquitous smoking detection with commercial wifi infrastructures,” in Proc. IEEE Int. Conf. on Computer
Communications (INFOCOM), Jul. 2016, pp. 1–9.

162

International Workshop on Informatics ( IWIN 2018 )Development of a Self-guided Web-based Mental Healthcare Course for Day-To-Day
Practice
Takeshi Kamita*, Tatsuya Ito*, Atsuko Matsumoto**, Tsunetsugu Munakata***, and Tomoo Inoue*****Graduate School of Library, Information and Media Studies, University of Tsukuba, Japan{s1730527, s1721654}@s.tsukuba.ac.jp**
Graduate School of Comprehensive Human Science, University of Tsukuba, Japans1130368@u.tsukuba.ac.jp***
SDS Corporation, Japan
munakata21@yahoo.co.jp****
Faculty of Library, Information and Media Science, University of Tsukuba, Japaninoue@slis.tsukuba.ac.jp

Abstract - The need of mental health management inenterprises has become widely recognized. It is a big issue
because of the huge number of employees, which is beyond
the capacity of existing number of psychological experts
such as medical doctors and counselors. Thus, self-guided
method of mental healthcare is needed. A digital content of
a self-guided mental healthcare course based on thecounseling method SAT has been developed with the virtual
reality technology so far. However, the virtual reality course
was not so easy to use every day, although repetitive use of
the course is desirable for keeping good mental health. Thus,
we have developed the web-based self-guided mental
healthcare course that can be practiced by a smartphone, byapplying the same structure of the virtual reality course. The
14 days experimental use of the course in comparison withthe conventional breath relaxation method is reported aswell as the detailed design of the course.

In this research, we have developed a new web-based selfguided mental healthcare course (WEB course) that can bepracticed by a popular device, a smartphone, using the samestructure with the VR course, to improve the usability. It
enables a user to repeat the course easier and acquire a selfcare skill to cope with daily stresses so that the user can
gradually stabilize the emotion for oneself and improve
stress tolerance eventually. In order to examine the stress
reduction effect of the WEB course, we conducted anevaluation experiment to compare the effect by one-time useof the WEB course and continuous use during 14 days witha breathing relaxation method.

2

In recent years, researches have been conducted to apply
psychotherapy to digital contents and use them with mobile
devices as a complementary tool for treatment andcounseling, or a training tool. Researches on one of major
psychotherapy, Cognitive Behavioral Therapy (CBT) haveespecially progressed [8] [9] [10], and there are many
commercially available mobile applications [11]. In the
CBT sessions, homework such as keeping a diary orchanging a habit of behavior which is defined as an issue tobe fixed through a session is often imposed on the client. In
the next session, a counceller finds distortion of the clients
cognition from the record of the diary and makes correction.
Such the homework part of CBT is applied to digitalcontents and offered as mobile applications, however which
does not cover whole CBT processes.
"Mood Mint" [12] was developed as a mobile training
tool to reduce anxiety and depression with reference to the
Cognitive Bias Modification [13], which attracted attention
as a counseling technique.
Mindfulness-based Stress Reduction using meditation and
Mindfulness Cognitive Therapy are also active in researchand psychology clinic in Europe and the United States [15][16]. As smartphones become more prevalent, mindfulnessbased mobile applications [17][18] are attracting attention.
Among them, a meditation voice guide service “Headspace”[19] that can be used in smartphone applications iscommercially available. This application provides several
courses for each purpose, such as coping with anxiety anddepression, and assists the progress of meditation.

Keywords: Self-guided mental healthcare, Healthcare
course, SAT counseling method.

1

RELATED WORKINTRODUCTION

Research of online courses on mental healthcare has
become active, as the importance of keeping good mental
health has been widely recognized. The stress check system
to keep employees’ good mental health in companies haseven been legislated recently in Japan. This resulted insudden increase of the number of potential clients for
medical counseling most of whom are not in sick, whereas
the number of industrial physicians specialized in
psychology to carry out the counseling and to deal with
employees’ mental problems does not. Thus, at the sametime as expanding the support system by experts, means toincrease employees’ own ability to conduct mental
healthcare are more in demand.
A self-guided mental healthcare course[1] [2] based on acounseling and therapy method, the SAT (Structured
Association Technique) method [3], adopting virtual reality(VR) were suggested as one of the means. The course whichenables a user to carry out the therapy process for oneself
with wearing a VR Head Mounted Display (HMD) obtained
good stress relief evaluation.

163

International Workshop on Informatics ( IWIN 2018 )The above psychotherapy-based mobile applications arebasically aimed at providing a training program and notpracticing a therapy in one session. The purpose of ourresearch is to realize a self-guided mental healthcare course
through which a user can practice a therapy to reduce dailystresses and a training to acquire a skill to cope with such
stresses.

3
3.1

representation instead of nuanced linguistic expressions.
Also, it is well structured which can be practiced inrelatively short time in 5 to 10 minutes. But, in theconventional SAT Imagery Therapy, the expert evokes
client's association through hearing, counseling orpresenting thumbnails on paper media without images In
some cases, the image is not sufficiently evoked by merely
looking at the image on the paper medium, and thecounselor has a supplementary voice call or encourages eyes
to close to arouse the image while seeing the reactions suchas the words and expression. Therefore, counselors play animportant role in their progress. In this research, we createda course as a technique that can make self-progression even
without counselor guidance support by converting SAT
method to digital content and using a smartphone.

SAT METHOD
Overview of SAT method

SAT counseling method is a structured and interview form
counseling method proposed by Munakata. The SAT
method has a wide effective range such as a mental disorder(such as Depression, bipolar disorder, obsessive compulsive
disorder, personality disorder, schizophrenia, etc.) andvarious stress diseases. Unlike other conventionalcounseling methods focusing on the psychological aspects,
the SAT method puts an importance on physicality, andapproaches mental problems from the bodily symptoms.
Therefore, instead of working thought by linguistic stimulus,
use visual stimulus from the presented image. It is possible
to grasp unconscious true feelings and an essential desire ina short time because it can functionalize an association and aflash and intuition well.

3.3.1 Course Composition
Based on the SAT method, the composition andprocedures of WEB course are classified into three
categories (Table 1): (1) knowing their own mental
condition (Assessment Part), (2) stress reduction (SolutionPart), (3) knowledge and training to improve mental
resistance (Learning Part). In the learning part, based on theanalysis of the data obtained in the assessment part and thesolution part, learning contents suitable for individual stresscharacteristics are provided. In this research, the assessment
part and the solution part are developed prior to the learningpart and the stress relief effect of image therapy in thesolution part are investegated.

3.2 SAT Imagery Therapy using Light Imageand Surrogate Face Representation Image

Table 1. WEB course and SAT method content comparison

When a person who wants to have counselling recalls a
stress scene, it is perceived as physical discomfort (such as
Stomach shrinks, nervous, sweating hands, chest tightening).
The SAT Imagery Therapy using Light image is a techniqueto change the discomfort to a good feeling and reduce thestress by watching the light image selected and perceived asa pleasant stimulus[5].
The SAT Imagery Therapy using Surrogate Face
Representation image is a technique for transforming theimage for self to a good one by replacing the primitive landscape (for example scenery that many yell at aroundchildhood) in the interpersonal relationship of the consultant
with the image of surrogate face representation symbolizing
pleasure. In psychology research, it is generally known that
influence on self-esteem of a person is influenced by how
the child care attitude of a child career is positive or
negative. By allowing the person to select an image ofsurrogate face representation with a sense of good feelingand recalling the image of a scene that makes a sense of
security and providing a feeling of security, person
perceives a sense that is safely protected, enhances selfesteem and encourages stress reduction.

The course
category

Content of WEB course

Assessment
Part

SolutionPart
Learning
Part

3.3.2

To know mental
conditions andcharacteristics.
To carry out a therapy
process to reduce stress.

To learn the methodologyof the SAT method andunderstand the course.

Corresponded contents of
the SAT method
The SAT psychologicalscale used for healthcoaching by a SAT
therapist
The SAT imagery work
using Light Image andSurrogate Face
Representation Image
Psycho education
conducted by a SAT
therapist.

Assessment Part

In the assessment part, we conducted a mental
characteristic check test (Table 2) using the SAT four
psychological scale with the aim of measuring the mental
condition and characteristics of the user and clarifying thechanges before and after the use of the system.

3.3.3 Solution Part
The solution department presents the set questions inorder and is proceeded by the process that the user answers
to reduce the stress. In the first half (Table 3), first, the user
is asked to remember one of stress scenes in accordancewith the question and make aware of how much stress it is.
Next, by comparing the stress to color and form and making
imagination as if the stress image compared by color and3.3 Self-Guided Course based on SAT
MethodSAT counseling method does not need a client to tell
his/her traumatic episodes or secrets and uses visual
stimulation by images of light and positive face164

International Workshop on Informatics ( IWIN 2018 )form are approaching the user oneself, the user is
encouraged to perceive the physical discomfort.
Furthermore, by specifying body part and type of theperceived physical discomfort, user is prompted to focus
consciousness on the discomfort in the body. And then, byexpressing the stress level caused by the discomfort as anumerical value (%), the user recognizes more clearly thatthe discomfort is occurred while feeling the stress.
In the second half, the course steps based on the processof SAT Imagery therapy using light image and surrogate
face representation (Table 4) are proceeded to decrease thestress level%, in short, relieve the discomfort and reduce thestress.
Table 2. Mental characteristic check test

4

What kind of character are you going to be when you see
these people?

5

If such a personality, in the situation of stress, how can
you handle it? It's okay with what you came up withintuition.

6

What do you think is the result if you do that?

7

Who is the most interested of those who have chosen?

8What message will you give me?

9

How will you feel?

10

How did you feel about the stress that first came up when
you were watching all the faces of these people?

Scale

11

How has the degree of stress changed? (Choose from 3
stages "not so" to "very much")Content

State-Trait
AnxietyInventory(STAI)
Self-rating
Depression
Scale(SDS)
Self-repression
behavioral trait
Difficulty to feelemotionIt tends to fall into anxiety.
The degree of vague anxiety
that reflects the individual's
past experience rather thanstate anxiety that changes overtime [20]Evaluationofdepression
symptoms including mood,appetite, sleep [21]Behavior characteristics thatsuppresses one's feelings andthoughts
Even if feel a painful thing,
you tend not to be emotional,
and tend to endure yourself

Number ofquestions
20

Score
range
2080

20

2080

10

0-20

10

0-20

4 SELF-GUIDEDMENTALCARE WITH SMARTPHONE

The WEB course using smartphones was developed inaccordance with the composition of the digitized SAT
method. It is constructed as a web site that can be realizedwith multiple platforms so as to flexibly respond to users'
usage situations. Therefore, it can be accessed by using PC,
smartphone, etc. In this research, we will describe contents
assuming use on smartphones.
In the VR course, the immersive feeling image fits theSAT imagery therapy and can be as one of factors to bringeffect on the stress reduction. However, such effect is notexpected with the small flat display of the smartphone. On
the other hand, from the viewpoint of operability, swiping
and tapping operation on smartphone is easier and morefamiliar than moving the head to manipulate the cursor on
the VR screen. In the SAT therapy process, it is desirable tointuitively perform selection operation in a short time rather
than carefully selecting using long time. By using asmartphone, relatively complicated operations such as
button selection, cancellation, page advancement and returncan be performed more intuitively and quickly.
When the user logs in using the ID and password on thelogin page, a start page is displayed. The user select ether
the button of the Assessment part or the Solution part.
The mental characteristic check test is displayed in theassessment part.
When the user select the button of the solution part, thepage shown in Figure 4. In this screen, the user are

Table 3. First half of solution part
Order

Question

1

Please remind me again what you are concerned aboutnow

2What is it like? Please choose (Choose from 34 sourcesof stress such as your future, family health etc.)

3

How much is that degree? Please choose (Choose from 3
stages "not so" to "very much")

4

Does that stress comparable to color? (Choose from red,brown, black, gray, purple, navy blue, light blue)

5

If you compare the stress to the shape? (Square, rugosum,
muddy, fluffy, pointed, flat, selected from spheres)

6

Close your eyes, thinking about where this thing comes
and imagining this image, where do you feel strangenessin your body?

7

How is that strangeness? (Choose from throbbing, cold,heavy, dull, sore, tight, numb, stretch)

8What is the stress level of current discomfort? (answer
from 0%~100%)

Order

Table 4. Second half of solution part
Question

1

The part that feels that discomfort is healed by whichcolor light is being protected?

2

Please choose a comfortable face that came into youreyes. Do you have anyone who smiled easily?

3

Looking at that face, what percentage of stress is the same
as before? (Answer from 0%~100%)HEALTH

Table 3.
Question 1,2

Figure 4: The Stress source list screen
165International Workshop on Informatics ( IWIN 2018 )requested to recall the stress scene (Table 3, Question 1),
and select from the list of prepared stress sources (such as
things of own future, family health etc.) the one closer to the
problem of the stress scene (Table 3, Question 3). Thenchoose from 3 options for the degree of stress. In the scene
where stress is compared to color and shape (Table 3,
Question 4), make a selection from the image list. Returning
to the chat screen, while recognizing the color and shape ofthe selected color, perceiving physical discomfort andspecifying the part and type (Table 3, Questions 6, 7)(Figure 5). Finally, answer by entering% of stress received
by physical sense of discomfort.
Subsequently, questions are presented according to thelatter half of the solution part (Table 4). The user is asked to
select a light image (golden, green, peach, orange, blue,
white, cream, yellow color, provided based on the lightimage of SAT method) (Table 4, Question 1) from image
list. From the image list, after selecting the image that canmake an image that will heal physical discomfort, we moveto the screen and the selected image is displayed (Figure 6).
Multiple images can be selected. After that, select a
representative from the selected image and deepen the
feeling of being protected by imagining speaking. Finally, it
asked the user to answer how stress level against stresssource confirmed in the first half has changed, and it ends.

Education, Culture, Sports, Science and Technology inJapan [22] etc. as a relaxation method to consciously control
breathing. It has also been reported that it is a technique thatcan be instructed safely and effectively in clinical practice[23]. Even a busy worker can do with a little time hanging
on a chair and does not need special physical strength.
This method intends to increase the mobility of thediaphragm, and the respiratory movement that emphasizes
the process of expiration is performed at a speed of 3-4
times per minute with the eyes closed and sitting on thechair [24][25]. First, take about 4 seconds, breath in throughyour nose and inflate your abdomen. Next, after 1-2 seconds
between switching from inhalation to exhalation, pull out
the lower abdomen while drawing slowly and slowly forabout 8 seconds. In this experiment, according to thismethod, the subjects in the control group were asked toperform this method.
We created a website (BREATH course) to guide theexperiment subject's breathing practice. This has a screen(Figure 7) for presenting the implementation method and atimer screen (Figure 8) for displaying the execution time.

Tap the screen
to start the
timer
Figure 7: Breathing exercise
practice guide screen
Figure 5: Identifying
physical discomfort

5

5.2

Figure 6: Selecting lightimage and face images

Procedure

33 college students and 7 office workers were selected as
subjects, and randomly assigned to two groups, a WEB
course conducted experiment group and a BREATH course
running group (WEB course:N = 21, BREATH course:N =19). For each group, after explaining the SAT method orbreath relaxation method, the contents of the experiment andthe course, we started to use the course and measured thestate before the course by the mental characteristic checktest of the assessment part. Next, we conducted the course at
the solution part and asked the mental characteristics checktest of the assessment department again after the course.
This is the first day 's experiment.
After the second day, I instructed to carry out the course
once a day. In addition, in order to confirm theimplementation status of the course and the troubled points,
we notified by e-mail every day until the 7th day after thestart of the course and then notified two or three times.
The timing of the survey conducted mental characteristic
check test using four psychological scales at 4 points before
and after the first course, after the course on the seventh day
and after the course on the 14th day. We used a

EXPERIMENTIn this research, the evaluation experiment was carried out
with the approval of the ethics review committee in Facultyof Library Information and Media Science, University ofTsukuba (Notification No. 29-109)
In order to examine the stress reduction effect of one-time
use and continuous use of WEB course and the difference ofthe effect, an evaluation experiment to have the subject
continue to use this course for 14 days and to compare itwith the breath relaxation method.

5.1

Figure 8: 5 minutes timer
screenBreath Relaxation Method

We prepared a course to carry out the breath relaxation
method as a control group.
The breath relaxation method is a training method aiming
at improving the function of the mind and the body bybreathing. It is introduced in the data of the Ministry of166

International Workshop on Informatics ( IWIN 2018 )Table 5. Stress characteristic check scores (WEB course: N = 21, BREATH course: N = 19)
Scale
STAI
SDS
Self-repression
behavioral trait
Difficulty tofeel emotionWEB
BREATHWEB
BREATH

One-time Before
Average ± SD
42.95±9.233
37.68±6.57
32.48±13.04
31.16±7.37

WEB

10.00±2.67

9.85±3.30

8.88±1.50

8.75±2.08

BREATHWEB
BREATH

9.79±3.33
9.24±2.74
8.63±4.19

10.21±3.88
9.40±3.32
9.26±5.30

10.35±3.84
9.60±3.07
9.12±4.09

9.22±3.74
8.00±2.63
8.94±4.28

CourseOne-time After
Average ± SD
41.85±11.03
33.84±6.91
33.05±11.26
30.37±4.97

RESULTThe result of stress characteristic check test is shown on
Table 5. The average score of STAI scale and Selfrepression behavioral trait scale in the WEB course and SDS
scale in the BREATH course are gradually decreasing stepby step during the experiment period. The others are once
increased, but decreased at 14th day eventually. Thechanges in these average scores suggest that the stressrelieving effect was brought and accumulated in 14 days ineach course. This result shall be tested and analyzed in detailin the future.

7

14th dayAverage ± SD
36.88±9.52
33.67±8.51
29.50±7.67
27.83±6.44

Counseling Technique SAT Method, IPSJ SIG
Technical Report, vol.2018-DCC-18, no.37, 1-8(2018).
[3] Tsunetsugu Munakata.: SAT therapy,
KANEKOSHOBO, Japan, (2006).
[4]Tsunetsugu Munakata: The applicability of the simple
edition of SAT method in promoting universal
health, Journal of Health Counseling, 17, pp. 1-12
(2011).
[5]Tsunetsugu Munakata.: Does SAT Re-scripting
Expression Imagery Enable Us to Overcome Unendurable Hardships toward True Life Career ?,Journal of Health Counseling,15,pp.1-12 (2009).
[6] Yoshihumi Tsuji: Development of a self-image
focusing coping strategy aimed at improving
psychological competitive skills, PhD
Thesis.Universitiy of Tsukuba,Japan, (2011).
[7]Kelley, Christina, Bongshin Lee, and Lauren Wilcox:
Self-Tracking for Mental Well-ness: Understanding
Expert Perspectives and Student Experiences,
Proceedings of the SIGCHI conference on humanfactors in computing systems, CHI Conference 2017,
pp．629-641．
[8]P. J. Batterham, A.L. Calear, L. Farrer, S.M.
McCallum, and V.W.S. Cheng: FitMindKit:
Randomised controlled trial of an automaticallytailored online program for mood, anxiety, substanceuse and suicidality, Internet Interventions, Vol12,
pp 91-99, (2018).
[9]K.H.Ly, E. Jannib, R.Wrede, M.Sedem, T. Donker,
P.Carlbring, and G. Andersson: Experiences of aguided smartphone-based behavioral activation
therapy for depression: A qualitative study, Internet
Interventions, Vol.2, Issue1, pp.60-68, (2015).
[10]D. Bakker, N. Rickard: Engagement in mobile
phone app for self-monitoring of emotional
wellbeing predicts changes in mental health:
MoodPrism, Journal of Affective Disorders, Vol.227,
pp.432-442, (2018).
[11]J. Torous, M.E. Levin, D.K. Ahern, and M.L. Oser:
Cognitive Behavioral Mobile Applications: ClinicalStudies, Marketplace Overview, and Research
Agenda, Cognitive and Behavioral Practice, Vol.24,
Issue 2, pp.215-225, May (2017).
[12]“Mood Mint”, http://www.biasmodification.com/,(view 2017-11-16)．

questionnaire on the paper medium for the firstexperiment. On the seventh day and the 14th day, we used
the search function on a smartphone.The stress situationdifference was evaluated using the stress characteristic
check test (Table 2). Four psychological measures (StateTrait Anxiety Inventory, Self-rating Depression Scale, Selfrepression behavioral trait, Difficulty to feel emotion) usedin the usual SAT method were used.

6

7th day
Average ± SD
38.00±9.94
37.94±7.01
31.53±7.08
28.53±6.27

CONCLUSION

We developed a self-mental healthcare course based on
the SAT counseling method aiming at long-term andcontinuously available self-mental healthcare tool, using asimplicity of smartphone and intuitive operation of webbased self-mental care course. In this research, we
conducted an experiment to have the subjects continue to
use the course over 14 days and examined the stressrelieving effect of one-time use and continuous use. As aresult, it is suggested that the stress relieving effect by using
the course accumulated in 14 days. In the future, the result
shall be tested and analyzed in detail to verify the effect of
the course.

REFERENCES[1]Asuki Nakanishi, et al.: A VR Self Mental Healthcare
System by SAT-Based Meth-od,IPSJ SIG Technical
Report,vol,2017-DCC-15, no.35, pp.1-8 (2017).
[2] Tatsuya Ito, et al.: A Self-guided Mental Healthcare
Digital Content for Smartphone VR Based on the167

International Workshop on Informatics ( IWIN 2018 )[13]Wang Rui, et al,StudentLife: assessing mental health,academic performance and behavioral trends ofcollege students using smartphones, Proceedings ofthe 2014 ACM International Joint Conference on
Pervasive and Ubiquitous Computing, pp3-14,(2014).
[14]F.B. Dickerson, et al. : The token economy forschizophrenia: review of the literature andrecommendations for future research, Schizophrenia
Research, vol.75, pp.405-416, (2005).
[15] Jon Kabat-Z: An outpatient program in behavioral
medicine for chronic pain patients based on thepractice of mindfulness meditation: Theoreticalconsiderations and preliminary results, General
Hospital Psychiatry, Vol．4, Issue 1, pp.33-47,(1982).
[16] Rinske A. G. et al. : Standardised MindfulnessBased Interventions in Healthcare: An Overview of
Systematic Reviews and Meta-Analyses of RCTs,
PLOS ONE, (2015).
[17]Inmaculada Plaza, Marcelo Marcos Piva Demarzo,
Paola Herrera-Mercadal,and Javier GarcíaCampayo: Mindfulness-based mobile applications:
Literature review and analysis of current features.
JMIR mHealth and uHealth 1, 2, e24 (2013)．DOI:
http://dx．doi．org/10．2196/mhealth．2733[18]“Diana Wells: The Best Meditation Apps of2016.”, http://www．healthline．
com/health/mental- health/top- meditation- iphoneandroid- apps, (view 2016-07-18).
[19] “Headspace”, https://www.headspace.com/, (view2018-06-01)．
[20]Spielberger, C.D. : STAI manual, Palo Alto, Calif,
Consulting Psychologist Press, (1970)．
[21]Zung, W.K. K. : A self-rating depression scale,Archives of general psychiatry, 12, pp.63-70, (1965)．
[22]"Safety measure materials for overseas educational
institutions", Ministry of Education, Culture, Sports,
Science and Technology Home page,
http://www.mext.go.jp/a_menu/shotou/clarinet/002/
003/010/004.htm (view 2017-10-14).
[23]Kikuyo Kosakabashi: Aiming to incorporate therelaxation method into clinical nursing - Research,education and practice of relaxation method as
nursing intervention - , KMJ The Kitakanto Medical
Journal, 65, pp.1-10, (2015).
[24]Setsuo Arai: Examination of effective posture fordiaphragmatic respiration. Physical education
science, 41, pp.813-817, (1991).
[25]“Useful for nursing care【3 volumes】” ,https://www.igakueizou.co.jp/product/product_detail.php?product_code=NK (view 2018-01-08)

168

International Workshop on Informatics ( IWIN 2018 )Healthcare Promotion System by
Stimulating In-Group Contribution Mind and Inter-Group CompetitivenessYuki Aso† , Nobuyuki Ito‡ , Katsuhiro Naito*, Naoya Chujo*, Tadanori Mizuno*, Katsuhiko Kaji*†

Graduate School of Business Administration and Computer Science, Aichi Institute of Technology, Japan‡
Mitsubishi Electric Engineering Company Limited, Japan* Department of Information Science, Aichi Institute of Technology, Japanb18701bb@aitech.ac.jp

Abstract - Deskworker is a lack of exercise because there
are few opportunities for exercise. Lack of exercise is a primary cause of most chronic diseases. On the other hand there
have been many studies of health services combined with smart
devices, gathering a user’s health lifelog and managing his or
her health for the improve. The purpose of this study is to addgroup elements to health services and promote exercise. Because the group elements can stimulate competitiveness andcontribution mind, these will help to promote exercise. Other
studies have concluded that we changed the behavior of theteam based on life log technology and information sharing
and promoted exercise[1]. The model with ”competition”
technique resulted the most effective performance for competitive teams such as sport teams. However, in this study it
is inconvenient to require manual input by the user, and thepresentation method is inadequate. In this paper, in order toautomate acquisition of exercise amount and maintain continuous motivation, create a healthcare promotion system (Fig.
1) incorporating competitiveness and contribution mind. Asa method of automating gathering of user’s activity amount,use smartphone application and API provided by application.
To improve competitiveness, compare with other groups using group evaluation values. Group members are ordered indescending order of exercise amount, ranking advantage isused to present group contribution. We did create and operate a prototype to ﬁnd the best presentation method. Measure
the number of steps of the person belonging to the four laboratory as the activity amount. For grouping, it is between
laboratories, grade, between transportation, between favorite
sweets etc. Number of people belonging to the group, thegroup evaluation value changes. Therefore, it is necessary toconsider other calculation methods. It is insufﬁcient in terms
of privacy protection to present names, step counts and action
rankings within the group. Therefore, the user should indicatethat your position is in the top of the group. Information presentation using activity amount is inappropriate for maintaining motivation. Therefore, regular motivation is necessary.
We implemented an event function aimed at regular motivation. After the event period and period, there was an increaseor decrease of the activity amount of the user. From theseresults, we discussed the necessity of regular motivation.

1 INTRODUCTION
Deskworker is a lack of exercise because there are few opportunities for exercise [1]. Lack of exercise is a primarycause of most chronic diseases [2]. It is common for companies to grasp the health situation of employees and give
exercise instruction to improve [3]. However, because thatmethod is insufﬁcient as a precautionary measure, companies
need effective health management [4].
I believe that improving motivation has the effect of encouraging behavior leading to exercise. Ingress1 and Pokémon
GO2 , game portals are set for real world buildings and monuments. Therefore, in order to advance the game advantageously, there is a characteristic that it has to actually moveto that place. For example, Pokémon GO got about 5 billion
kilometers3 , about 10 billion km in 10 monthst4 . Even for
those who are not interested in healthcare, the number of steps
in everyday life is increasing naturally in order to favorably
perform these position information games. These location information games can be said to be examples that showed thepossibility of improving motivation and transforming humanbehavior if the location information service is properly designed. In this case, the user is acting to lead to exercise basedon the motivation to enjoy the game.
On the other hand there have been many studies of healtahservices combined with smart devices, gathering a user healthlifelog and managing his or her health for the improvementof the quality of his or her life, using various sensors. The
purpose of this study was to add group elements to healthservices and promote exercise. Because the group elements
can stimulate competitiveness and contribution mind, thesewill help to promote exercise.

2 RELATED RESEARCH
According to ”Draft Motion Standards for Health Promotion” announced by the Ministry of Health, Labor and Welfare in January 2006, walking of 8,000 steps to 10,000 steps
per day is considered good for prevention of lifestyle diseases[5]. In addition, excessive exercise, on the contrary, becomesstress, there is also a risk of lowering immune function [6].
1 Ingress:https://www.ingress.com/

Keywords: healthcare, group activity, contribution mind,
competitiveness2 Pokémon GO:http://www.pokemongo. For location information games
for smartphones such as jp/
3 https://pokemongo.nianticlabs.com/en/post/headsup
4 Pokémon
GO
Walk
and
go
for
adventure!
http://pokemongolive.com/en/post/adventureweek2017169

International Workshop on Informatics ( IWIN 2018 )Therefore, it can be said that about 8,000 steps per day is asuitable exercise amount. In this research, the objective is topromote health by exercise focusing on walking. Therefore,
it is necessary to introduce appropriate momentum based on
medical grounds as a motion index in the system.
It is difﬁcult to realize behavioral change by compulsion
from outside, and sustainability is not seen. Meanwhile, behavioral changes realized based on endogenous motivation
are sustained as long as their motivation is maintained. Therefore, promotion of healthcare of ofﬁce workers is not necessarily merely realizable momentarily, but sustainability is aproblem.
Human Based Computation’s research will be helpful inconsidering stimulatable motivation. Human Based Computation is an approach in which people are asked to performtasks that are difﬁcult to automate by computers, and there
are cases in which various motivations are stimulated. Hereare the main motivations. I want to get results, monetary compensation, stimulation different from everyday life, aesthetic
appetite, curiosity, volunteer spirit, contribution heart, competitive spirit, communication desire, desire to share knowledge, want to achieve what others can not do, game, fun isthere. As an example, re-CAPCHA creates an account of a
web system anew, or a mechanism for letting a person who istrying to download a certain ﬁle to input two words and judge
whether it is a person or a BOT [7]. At this time, one of thetwo is a word that failed OCR in the digitization project of
the old book. Behavior based on motivation to obtain results
such as accounts and ﬁles, and behind the scenes contributeto digitalization of old books by human power.
To improve the motivation of exercise, a method using reward can be considered. An example using a game is BitHunters[8]. BitHunters is a real world game for smartphones who canenjoy adventure walking in the real world. Through walking,the user can acquire tokens issued in the game. This token canbe exchanged for a virtual currency like a bit coin in the future. Therefore, the user can expect improvement in exercise
motivation for reward.
At the Osaka Minami Power Station, Kansai Electric Power
Co., Ltd. conducted group activities to improve lifestyle habits,
and achieved [9]. As for the health improvement measures
before group formation, the target person felt ”feeling being
carried away”, and the expected effect was not obtained. Asa breakthrough policy, we organized the group and practiced
health improvement small group activities to formulate andpractice health improvement measures by the members of thegroup themselves. Through this policy, we shared objective
consciousness within the group and expected improvement ofhealth with mutual sense of solidarity. In the ﬁrst year of
the group organization, it divided into three groups of ”GoodGroup”, ”Group that needs a little hard work”, ”Group that
got worse” from the trend of past examination ﬁgures. Basedon the advice of industrial physicians, the group discussed targets and all members worked on health examination to be carried out three months later. Of the 27 subjects, 7 people succeeded in health improvement, 11 people deteriorated morethan before. For this grouping, because the cause varies foreach subject, it is difﬁcult to share the purpose conscious-

ness. In order to facilitate the sharing of target consciousness
among subjects within the group, we organized the ”eradication high-fat team”, ”resurrected liver function team”, ”overthrowing hypertension team” for each cause item. We set
goals and take countermeasures in the same way as last time.
Thirteen out of 25 participants succeeded in improving health.
Five out of 13 people achieved all goals. From this result,grouping that is easy to give a common purpose consciousness is important.
Fitbit [10] which carries an acceleration sensor and an altitude sensor and measures the amount of activity is linked
with wireless application of smartphone, and data management and the amount of activity can be visualized. Within
the application, there is a function to share the momentumof friends and family members and make them compete. Byutilizing this function, it can be expected to stimulate user’s
motivation.
The paper by Nishiyama [11] have concluded that Theychanged the behavior of the team based on life log technology
and information sharing and promoted exercise. The modelwith ”competition” technique resulted the most effective performance for competitive teams such as sport teams. However, in this study it is inconvenient to require manual input
by the user, and the presentation method is inadequate In thispaper, in order to automate acquisition of exercise amount andmaintain continuous motivation, create a presentation system
incorporating competitiveness and contribution mind.

3 HEALTHCARE PROMOTION SYSTEMBY STIMULATING IN-GROUPCONTRIBUTION MIND AND
INTER-GROUP COMPETITIVENESS
In this research, we focus on contribution and competitiveness. These focuses are very good for the combination ofconcepts of groups. It stimulates the feelings that contributeto the group as a member and the feelings that you do notwant to lose to other groups. Improve motivation of exercise
by doing these appropriately. Therefore, we focused on thegroup because we thought that maintaining motivation for exercise was easier when exercising as a group as members ofarbitrary group than when doing voluntary exercise by individuals. When grouping ofﬁce workers, there are groups suchas departments and companies. It is also conceivable to form
groups for each user proﬁle such as gender, age, commutation method, favorite confectionery, and the like. To measure
the amount of activity, use a smartphone or a wearable device
that is considered to be carried around by an ofﬁce worker.
We use applications suitable for each device and collect andmanage data. Visually display the amount of activity, basedon the collected data. Users can switch, display among favorite groups.

3.1 Activity Amount Acquisition Method
In order to examine the presentation method effective for
improving motivation for exercise, we have constructed an
activity quantity presentation system among multiple groups.

170

International Workshop on Informatics ( IWIN 2018 )Fig. 1 is a diagram of the entire system. We utilized existingﬁeld service and automated acquisition of momentum.

Figure 1: Automatic activity record system
As a location information service to acquire activity amount,use ”Moves” [12] on iPhone or ”Google Fit” [13] on Android
device. Fitbit [10] which is one of wearable device installs
and uses a dedicated application. People using the systemshall always carry one of these.When the user uses this system for the ﬁrst time, proﬁle creation and registration of access tokens for exercise amount acquisition are performed.
We created a questionnaire page to have users use their device attributes and their own attributes. The answer items are
the device used, the major, the laboratory, the grade, the sex,
the height, the weight, the way of commuting, the presence orabsence of a circle of athletic meetings and cultures, Kinoko
no Yama or Takenoko no Sato. When the input is completed
and proceeding to the next step, the application authentication method suitable for the used Device is performed and theregistration is completed. At this time, the web server side
obtains the access token of the application and starts collecting the data of the exercise amount. In order to present information tailored to each individual, an authentication pagefor each person was created. For registration, ID and password authentication is common. To improve convenience, we
implemented a mechanism called open ID to log in using an
existing Google account. The user can save time of new registration and the system management side has advantages ofsimplifying user management and improving both sides.

Figure 2: User registration
there is a graphicization of activity amount of each favorite
sweets (Fig. 5).
Fig. 3, 4, 5 graph the average number of steps per group foreach day. When comparing the amount of activity, using the
cumulative value of the activity amount, the number of people
belonging to the group is different, and the small group is disadvantageous because it is disadvantageous. For this reason,
we are currently comparing using the group’s average activity
level.

3.2 Implementation of Activity Quantity
Visualization System
Although the main object of this research is an ofﬁce worker,
information university students are also common in that there
are many work sitting at the desk, We designed a system foruniversity students. We grouped them by laboratory and constructed a system to graph the momentum in each group (Fig.
3).
Furthermore, grouping by user proﬁle was done as a device
to attract users interest. As an example of more subdivided
grouping, there is a graphicization of activity amount of grade(Fig. 4). As an example of grouping according to hobby taste,

Figure 3: Amount of activity between laboratories

4 DISUCUSSION FOR OPERATION
We tested the proposed system in a short time. Also, we
discussed the display method and handling of data. In thischapter, discussion items for practical application will be discussed.

171

International Workshop on Informatics ( IWIN 2018 )Figure 5: Activity amount by favorite confectionery

Figure 4: Activity amount between grade

4.2 Reviewing grouping
4.1 Numericalization of Activity Amount
Although the average value of steps is used as the current
activity amount, the inﬂuence per capita changes according
to the number of belonging people (Fig. 6). We focus on thetransition of the user’s cumulative number of steps per day.
As a method, evaluate when the user continuously achieves
the goal or starts to work hard. The user who achieved thegoal after increasing more than the usual step number. Evaluate these users (Fig. 7). Person A has achieved the target
index for four days.Since this user is accomplished by daily
effort, it is necessary to evaluate the contribution to the group.
inﬂuence is weak in group B. For example, even if they all
have the same momentum, if there is a difference in the number of people, there will be a difference in inﬂuence per person. For that purpose we are planning to use the combinationof the variance of the activity amount of the group and theaverage value.
We focus on the transition of the user’s cumulative numberof steps per day. As a method, evaluate when the user continuously achieves the goal or starts to work hard. The user who
achieved the goal after increasing more than the usual step
number. Evaluate these user’s (Fig. 7). Person A has achieved
the target index for four days. Since this user is accomplishedby daily effort, it is necessary to evaluate the contribution to
the group. For Person C, the cumulative number of steps until
1st to 3rd day is low on average. However, on the fourth day,the ﬁgures showed an increase over the cumulative number ofdaily steps, achieving the target index. In this case, although
it is impossible to read the intention of the individual himself,
it can be inferred that we have started to achieve the target
index. In order to help maintain motivation, it is necessary toevaluate the beginning of efforts to exercise.

Prior to visualization of the amount of activity, it is necessary to ﬁrst classify groups without unfairness. As shownin the (Fig. 8), because there is a difference in the amountof activity due to differences in commuting methods, we areplanning to use the amount of activity during the working
hours for evaluation. However, since the administrator can acquire the activity amount of a speciﬁc user within an arbitrary
time, there is a possibility that problems related to privacy
may emerge from the viewpoint of behavior guessing and thelike. Also, after using the system for a certain period of time,
I plan to reorganize the group into a group more suitable forhealthcare. For example, it is conceivable that people withless activity from various groups are gathered, and the user
can be given a common conscience that ”they must exercise”.

4.3 Relationship between in-groupcontribution and privacy protection
To display the degree of contribution within the group, it isdisplayed using names and attributes, but it is considered to
be insufﬁcient from the viewpoint of privacy protection. For
example, when names and steps are used (Fig. 9) to indicate
the degree of contribution within a group, the recognition of
the degree of hard work between members increases, and improvement of motivation within the group can be expected.
However, for members with low group contribution, there isa possibility that exercise is forced from members with highcompetitive spirit or high contribution.
From the viewpoint of privacy, display that can specify thename and attribute of a member should be avoided. For example, comparing ﬁgures showing the amount of activity and degree of obesity (Fig. 10) and the ﬁgure of contribution within
the group, there is a danger that the degree of obesity andname will be inferred. In general, obesity degree is informa-

172

International Workshop on Informatics ( IWIN 2018 )Figure 7: Activity comparison for each user
Figure 6: Evaluation changes depending on the number ofmembers
tion that you do not want to know, so privacy protection isnecessary.
Reducing the sensing acquisition interval increases realtime performance. However, privacy protection is insufﬁcientfrom the viewpoint of user’s behavioral inference. It is necessary to set the minimum necessary interval for presentationand consider privacy protection.

4.4 Examination of Information Presentation
MethodIt is difﬁcult to obtain sense of accomplishment, superiority, and sense of crisis simply by presenting numerical values such as the number of steps. As a method to realize it,
we are studying the following information presentation. For
example, if you use Mt. Fuji as a motif, you can grasp the
position between the groups and visually obtain a sense ofaccomplishment and superiority (Fig. 11). Also, display theactivity level of each group in the same way, and measure thecontribution degree. Next, if you use a balloon diving gameas a motif, the amount of activity is proportional to the sizeof the balloon, and if the average value in the group exceeds
8000 steps, the balloon will break (Fig. 12). The upper groupshows the number of steps required to achieve the goal andgives another group a feeling of crisis. Also, to small balloon
group, present the number of steps to overcome and lead tobehavioral change. To maximize motivation, it is easier totackle if the way to overcome other groups is clear and simpler. We can build a strategy of winning with a little effort
and think that if the effect gets better in a short period of time,
it will lead to continuation of motivation. In order to havesustainability, it is necessary to devise not to get tired with adisplay method rich in variation. For example, giving eachgroup a character, metabolizing or slimming the style makes
it possible to obtain an evaluation of the image of each group.
Also, comparing rice cultivation to games, encourage contin-

ued hard work. Daily efforts are required to harvest rice, andthe achievement of rice growth and harvest can be obtained.
Because rice plants dying when you skip along the way, you
can expect participants to return to their original goals.

4.5 Ddeviceevice Difference of Acquisition
Activity Amount
Although this system can acquire the activity amount on
the smartphone or wearable device, there is a difference inthe amount of activity acquired by the device. A smartphoneis always carried by a user as compared with a wearable device worn on the body. Therefore, there is a difference in theamount of activity obtained by the device. Also, since the sensor accuracy built in iPhone, Android, Fitbit is different, there
is difference in activity amount You can think of possibility.
In the future, when the difference in activity amount due toaccuracy becomes an adverse effect of comparison between
groups, Or a method of comparing between groups within thesame device may be considered. As an acquisition method
other than smart phones, there is an employee ID card withembedded acceleration sensor [14]. It is possible to measure
the number of steps from built-in sensors. Since the employee
ID card is always carried by the ofﬁce worker inside the company, it is expected that the capture rate of the momentum
measurement can be improved more than the smartphone or
the wearable device.

5 IMPLEMENTATION FOR ACTUAL
OPERATION
Based on the discussion in Chapter 4, we propose individual and group evaluation method and presentation method.
Evaluate the ongoing efforts and measures when evaluating.
For this reason, composite evaluation values including step
count are converted to points and evaluated (Fig. 16). In addition, as a presentation method, an event period is set forthe purpose of evaluating ongoing efforts. In this chapter, we

173International Workshop on Informatics ( IWIN 2018 )Figure 10: Correlation diagram of activity and body mass index

Figure 8: Activity amount by transportation method

Figure 11: Visualization of activity amount with motif ofMount Fuji
Figure 9: Activity amount ranking
describe these proposals and implementations for evaluation
experiments.

5.1 Implementation of Person Points

Based on the discussion in Chapter 4, we propose individual and group evaluation method and presentation method. Asan evaluation method, in order to evaluate the daily effort andthe beginning of the initiative, a point which is a composite
evaluation value including the step count is deﬁned and used.
In order to evaluate the user ’s ongoing effort, group points(Pbonus ) are deﬁned below.

We propose an person points calculation method which isnecessary for group points calculation. For person points, include bonus points to evaluate ongoing efforts. Person points(Pp ) are deﬁned as follows.
Pp = Pbasic + Pbonus
Deﬁne the basic points (Pbasic ) for converting the numberof steps to a point as follows (Fig. 14). Deﬁne the minimum
target that is easy to achieve to 2000 steps and the maximum
target to 8000 steps. Acquired points vary from 2 points to
10 points between 2000 and 8000 steps. 0 point if the minimum target has not been reached. It is ﬁxed at 10 points with
the number of steps exceeding the maximum target. The upper limit of the acquisition points is to prevent the immune
function deterioration due to excessive exercise [6] and tohave a certain degree of convergence for comparison between
groups.

Pbonus = Pbasic (Cit − 1)scale
We set an intermediate target of 5000 steps between the maximum target (8000 steps) and the minimum target (2000 steps).
The achieved number of intermediate targets is Cit .Bonuspoints will increase in proportion to the number attained theintermediate target.
Ebonus2 × (Edays − 1) × Edays × max(Pbasic )scale = 1

Here, Edays is the number of days of the event. The scale isfor setting the cumulative upper limit (Ebonus ) of the bonus
points. By achieving the continuous mid-term goal, points
can be greatly acquired at the end of the event. By doing this,174

International Workshop on Informatics ( IWIN 2018 )Figure 12: Presentdeviceing information for winning othergroupsFigure 15: Person points when you continue to walk 8000
steps or more every dayof group members is small like (Fig. 16) and the averagevalue of individual points is high. This will better appreciate
the group that everyone is doing their best. Therefore, group
points (Gp ) are deﬁned as follows.

Gp =Figure 13: Point Calculation Schematic−σ)
μ + wμ (σmaxσμ + wμσmax(σ ≥ 1)(σ < 1)(1)Here, μ is the average of individual acquisition points ofgroup members. σ is the standard deviation of individual acquisition points of group members.σmax is the maximum possible value of standard deviation. w is a weight for how muchvariation is to be evaluated with respect to the average value.

it is possible to give appropriate points as compared to those
who have not achieved target.
Fig. 15 is a graph of points obtained when walking 8000
steps or more each day. Earn daily basic points 10 and earn
70 points. Bonus points earned 30 points total.

5.2 Implementation of Group Points
Based on the discussion in Section 4.1, we implemented
calculation of group points, including variations in individual
points calculated in the previous section. For group points,
we want to make the highest evaluation when the variation

Figure 16: Approximate drawing about points

5.3 Information Presentation Method

Figure 14: Basic Points

In the upper part of the page, based on the discussion inchapter 4.4, in order to maximize motivation, we made anindication that improves competition among the groups (Fig.
17, 18).
Display group information on the left and right and implement a comparable screen. For the group cohesion degree, thestandard deviation 0 is displayed as 100%, and the standard
deviation maximum value is calculated as 0%, and the result

175

International Workshop on Informatics ( IWIN 2018 )is displayed. Also, group points are displayed in a time series
in a line graph.
At the bottom of the page, we present information on individual acquisition points(Fig. 19, 20). Individual information
indicates the degree of contribution to an individual group andindividual acquisition points. To indicate contribution degree,
what is the top. For privacy protection mentioned in the discussion in section 4.3, do not display ranking.
Figure 19: User points view

Figure 17: Cumulative points and cohesion degrees of each
group

Figure 20: User points graphInstitute of Technology Kaji Lab. As an evaluation method,
increase / decrease of the average exercise amount for eachsubject is conﬁrmed at the event week and after the event. As
information presentation to the user, we conduct web page
opening, signage display (Fig. 21) in the laboratory whereparticipating members gather, and occasional distribution (Fig.
22) using communication tools. This is because it is important to conﬁrm your position in the group one after another.

Figure 18: Group points graph

6 EVALUATION EXPERIMENTIn order to verify whether the motivation can actually be
improved by the proposed method, we conducted an evaluation experiment in the real environment. The evaluation itemis the presence or absence of increase in momentum.

6.1 Experiment Setting
Week 1 from June 13 to June 19 was the event week, followed by a week outside the event for a total of two weeks
as the experiment period. The subject was 23 people, Aichi

Figure 21: Information presentation in the laboratory
176

International Workshop on Informatics ( IWIN 2018 )exercise by duty is more effective than exercise by motivation.

7 CONCLUSION
The purpose of this research is to build a visualization system of activity amount among multiple groups and to improve motivation to the movement of people belonging toeach group. As a preliminary step for actual operation, we
examined the method of improving motivation, building a visualization system. These systems are currently being implemented. Through actual operation for a long time, we areplanning to promote research to make appropriate motivation
stimulation and lead to continuous behavior change.

REFERENCES[1] Ministry of Health, Labor and Welfare, Exercise Guidelines for Health Promotion 2006,<http://www.mhlw.go.jp/bunya/kenkou/undou01/pdf/
data.pdf> (2006) (in Japanese).
[2] Ministry of Health, Labor and Welfare, Study Group Report on Revision of Exercise Standards and Guidelines,<https://www.mhlw.go.jp/stf/houdou/2r9852000002xpleatt/2r9852000002xpqt.pdf> (2013) (in Japanese).
[3] Ministry of Health, Labor and Welfare, eFigure 22: Distribution of interim progress at any time byhealthnet, Instruction Regarding Exercise andcommunication tool
Physical Activities for Speciﬁc Medical Check6.2 Experimental Results
ups and Health Guidance,<https://www.ehealthnet.mhlw.go.jp/information/exercise/s-01Experimental results are shown in Fig. 17 and Fig. 18.
003.html/> (accessed 2018-7-24) (in Japanese).
During the event period, Takenoko No Sato was victorious[4] Workshop for the Management of Health on Comwith 47 points of Kinoko No Yama and 50 points of Takenokopany and Employee, What is Management of Health,
No Sato. On May 13th of the event start date, Kinoko No<http://kenkokeiei.jp/whats/> (accessed 2018-7-24) (inYama was dominant, but on May 16th there was a reversal.
Japanese).
Also, the result of the exercise comparison between the event[5] Ministry of Health, Labor and Welfare, Exweek from June 13th to June 19th and the subsequent week is
ercise Standards for Health Promotion 2006,
shown in the Table 1. We recorded data even after the event<http://www.mhlw.go.jp/bunya/kenkou/undou02/pdf/
ended. The week when the open campus was included was adata.pdf> (2006) (in Japanese).
good record.
[6] S. Shinkai, Y. Aoyagi, T. Suzuki, Activity Life Expectancy and Walking Ability of Senior Citizens, Research journal of walking (4), pp. 15–21 (2000) (inTable 1: Average value per day during the event and after theJapanese).
event
Average Daily Steps ɹ [7] L. von Ahn, B. Maurer, C. McMillen, D. Abraham,
Periodand M. Blum, reCAPTCHA: Human-Based CharacBefor the event (6/20-6/27)
5559
ter Recognition via Web Security Measures, Science,
Event (6/13-6/19)
5643
Vol. 321, lssue 5895, pp. 1465-1468 (2008).
After the event (6/20-6/27)
5818[8] Real World Games, BitHunters, <http://bithunters.jp/>
Include Open Campus (7/18-7/24)
6477(accessed 2018-7-24) (in Japanese).
[9] Y. Shirai, Health improvement activities by mini group,
Safety and Health, Vol. 11, No. 3, pp. 294–296 (2010)(in Japanese).
6.3 Discussion[10] Fitbit Inc. : Fitbit, <http://www.ﬁtbit.com/us> (accessed
2018-7-24).
Experimental results show that the amount of changes be[11] Y. NishiyamaɼT. OkoshiɼT. Yonezawa, J. Nakazawa,
fore and after the event are small. There is a marked increasein open campus compared to events. This makes it clear that
K. Takashio, and H. Tokuda, Encouraging Team Behavtemporary increase in activity amount was not conﬁrmed. In
ior Modiﬁcation Using Life Log, The Special Interest
addition about the open campus, we should take into considGroup Technical Reports of IPSJ, Vol. 2014-MBL-70,
eration the fact that there is a duty to participate. Therefore,
No. 9, pp. 1–8 (2014) (in Japanese).
177International Workshop on Informatics ( IWIN 2018 )[12] Moves, <https://moves-app.com> (accessed 2018-722).
[13] Google Fit, <https://www.google.com/ﬁt/> (accessed
2018-7-24).
[14] T. Watanabe, M. Murase, K. Naito, N. Ito, K. Kaji,
N. Chujo, and T. Mizuno, Prototype Implementationof RFID Based Health Management System with Lowpower ARM Microcomputer, Consumer Communications & Networking Conference (CCNC), 2018 15th
IEEE Annual, pp. 1–2 (2018).

178

Keynote Speech 2:
Dr. Shigetoshi Sameshima(Deputy General Manager,
Center for Technology
Innovation / General Manager,
Yokohama Research Laboratory,
Hitachi,Ltd., Research &Development Group)International Workshop on Informatics ( IWIN 2018 )Digital transformation towards Society 5.0
Shigetoshi Sameshima
Hitachi,Ltd., Research & Development Group
Abstract:
Japan is sometimes referred to as "an advanced country with advanced issues" as we are thefirst among advanced nations to face the challenges of an increasingly aging population witha low-birthrate. Given this situation, both government and industry have a vested interested
in resolving the various challenges that arise. The government of Japan has launched theSociety 5.0 vision in "the 5th Science and Technology Basic Plan" on January 2016. Society
5.0 aims to realize a new social structure to match the needs of individual citizens by tacklingsocietal challenges and surfing the digitalization wave. We will head for the active society
promoting autonomous value creation to solve various problems and create value by wisdoms.
Also, from a technology perspective, Japan has been contributing to various projects towards
Smart Society. Hitachi solves challenges with IoT digital solutions. For example, to realize
optimized

transportation, wedeveloped

train

traffic

management systemwhich

automatically changes train schedule in accordance with the number of passengers. In thisway, Hitachi realizes the fine-grained service for consumers and improves their Quality ofLife with the IoT platform. Furthermore, Hitachi will co-create solutions with partners allover the world by creating wisdoms from high-quality IoT data, and by implementing thevalue in the society to build a new value chain.

Joined Hitachi, Ltd in 1993. He engaged in R&D of systemtechnologies for public infrastructure systems like railway operation
system, utility operation system, water supply system, especially indeveloping autonomous decentralized systems. Now he is directing
Center for Technology Innovation in Systems Engineering and
Production Engineering fields. Received Ph.D from University ofTokyo.

181

Session 7:
Trust and Risk
( Chair: Tomoki Yoshihisa )International Workshop on Informatics ( IWIN 2018 )A Study of Risk Management Method using Quantitative Process Management
Akihiro Hayashi† **, Nobuhiro Kataoka‡ , Yasunobu Kino*, Mikio Aoyama**†

Department of Information Design, Shizuoka Institute of Science and Technology, Japan‡
Interprise Laboratory, Japan* Graduate School of Business Science, The University of Tsukuba, Japan** Graduate School of Information Science and Engineering, Nanzan University, Japan
pixysbrain@gmail.com

Abstract - Risk management (RM) aims to lead system development projects to success by eliminating negative factors.
It is expected that the number of failed projects can be reduced
in organizations that have introduced RM successfully. However, we cannot conﬁrm this expected result yet. In this study,
we ﬁrst analyze the RM process for the so-called successful system development projects conducted recently and thenidentify the factors where RM did not meet the expected criteria. Next, to eliminate the factors, we propose a quantitative
RM method that could improve the RM process and project
management (PM) process by using Earned Value Management (EVM) and Logistic Regression Analysis (LRA). When
we apply the proposed method to a real system development
projects, we concluded that the proposed method is effective.
Keywords: Project Risk Management, Logistic Regression Analysis,Quantitative Project Management

1 INTRODUCTION”Software dependent Society” has arrived. Important functions such as organizational operation, home electronics, andautomobile control are controlled using software. Therefore,
many companies focus on system development.
However, according to reliable statistical information[1],only 27% of all projects succeed in all aspects of quality, cost,and delivery time (QCD) in domestic and foreign system development projects. Thus, three-fourths of the projects do notmeet all the QCD criteria, which leads to the cancellation of
24% of software development projects[2].
To solve this problem, interest in introducing RM processes
in system development has increased. It is believed that RM
has the ability to lead projects to success by eliminating negative factors that may cause the project to fail.
In general, to introduce RM processes, an international “bestpractice model” has adopted the Project Management Body ofKnowledge (PMBOK)[3], Program & Project Management
for Enterprise Innovation (P2M) [4], the 2nd version of Projects
in Controlled Environments (PRINCE2) [5] as reference models, and introduced speciﬁc practices for RM that all presented in these guides.
However, even the introduction of the “best practice process,” does not reduce the number of failed projects. The ﬁndings suggest that successful implementation of the RM pro-185

cess does not contribute toward the reduction of failed project
occurrence.
We believe that there are two major factors that contributeto the event described above. One factor is that the project
management standards and guides that have been proposedand developed overseas do not match practices that are inplace in the domestic system development projects. The otherfactor is that even though the standards and guides are correct,
they have not been successfully introduced to the system development site.
In both cases, it is necessary to establish an appropriate
methodology to introduce RM processes to the managementof standard-sized projects in Japanese industry.
To address this issue and decrease the incidence of failed
projects, we ﬁrst analyze four cases of a speciﬁc risk management process conducted recently and identify the factorsthat will create a bottleneck. Next, to solve the problem offailure, we proposed a method to introduce the RM process
appropriately. The proposed method included quantitativerisk management and the implementation of risk countermeasures. When we applied our proposed method to a real casefor the RM of system development projects, a measurable effect was observed in the form of a reduction in the number offailed projects and a reduction in the contingency budget.
In most of the earlier research on risk management of system development, Boehm[6] and Williams el al[7] initiallyintroduced the implementation methods of risk management
practices that are commonly used presently, such as risk identiﬁcation, evaluation, classiﬁcation, and prioritization. Whensuch a method is generally perceived, risk management process is also adopted widely for project management standards
and guides such as PMBOK, P2M, PRINCE2 and processevaluation models such as ISO9001[8] and CMMI (Capability Maturity Model Integration)[9], Since then, improvement
practice in each life cycle of risk management, risk identiﬁcation using risk breakdown structure[10] and risk assessment
method[11] have been reported. In addition, with the advancement of risk management technology, quantitative risk
analysis[12] and the Six Sigma approach[13] have been proposed and carried out. Although such prior research explains
the potential of success for the risk management process, itdoes not describe why such best practice does not successfully operate in the real world. To the best of our knowledge,
no particular prior research discusses the appropriate method

International Workshop on Informatics ( IWIN 2018 )roles and responsibilities were misunderstood, andthe risks were not reported correctly.

to introduce risk management process.
The remainder of this paper is organized as follows: In Section 2, we review the related works on PM, EVM and RM to
conﬁrm the originality of this study. In Section 3, we analyze
a case on the implemented RM practice where we consult andidentify the reason behind the failure of RM, and describe theproblem to be solved. In Section 4, we propose a quantitative
RM method by using a statistical tool called LRA. In Section
5, the effectiveness of the proposal is evaluated by applying
the method to a real case of system development. In Section
6, we discuss the results of the case study. Finally, Section 7
presents the conclusion.

The four problems listed above indicated that the RM practice was correct, but that risk countermeasure actions were not
implemented on time.

2.2 Factor Analysis to show that RM was not
Implemented on Time
Next, we analyze factors that contributed toward the delayed occurrence of RM, extracted from the documents andminutes of the meeting of the past RM assessment. As a result, the following four factors were clariﬁed:

2 FACTORS CONTRIBUTING TO THE
FAILURE OF RM
2.1 Case Analysis of RM Process
First, we aim to clarify the reasons due to which the numberof failed projects has not decreased, even after best practiceof the successful introduction of the RM process in organizations. We analyze the factors by taking up four organizations
for whom we have provided management consultation to date.
Below is a summary of the organizations to be analyzed:
Case 1 Electronic control of vehicle amenity
Case 2 Electronic notebook, which maintains a schedule,
dictionary, calculator, and custom program
Case 3 Air conditioner system controlled by an internetbased remote control
Case 4 Derivative development of value-added of acoustic measurement calibration equipment
By analyzing the 4 aforementioned development cases, wefound the following four problems in the RM process:Problem 1 Since the triggering of alarm for the notiﬁcationof risk was delayed by the project manager (PM),risk countermeasures could not be implemented
on time. A similar problem occurred multiple
times in a PM’s tenure; he thought that the problem could be solved every time. Therefore, he
did not report the emergence of risk to the higherlevel managers.
Problem 2 Since the development project was originally planned
the development period, it was biased toward keeping the delivery date. Therefore, the organizations were averse toward reworking due to RM
activities and hence hesitated to report risk occurrence.
Problem 3 Despite the original plan to activate risk response
measures in an event-driven manner, project members did not accurately understand the RM process and subsequently reported risks at weekly
progress meetings that caused notiﬁcation delays.
Problem 4 Despite a clear deﬁnition of the trigger and threshold for risk interpretation in the RM ledger, the186

Factor 1 There was no timely and correct presentation of arisk report:
Even when the risks expanded and severe delays
did not allow adequate management, organizations
sometimes reported less risk or kept critical risks
hidden since the project was at a stage wherein
it would be evaluated by higher-level managers.
Therefore, a correct risk report was not deliveredon time. This is the cause of Problem 1.
Factor 2 Insufﬁcient risk judgment skill of the PM:
Due to the PM’s insufﬁcient risk judgment skill,such as insufﬁcient identiﬁcation of risk and undistinguished critical risk, the organization failed to
manage the risk properly. This led to Problems 1,
3, and 4.
Factor 3 Divided RM method:
Due to insufﬁcient communication between the
PM and higher-level managers, unclear terms andmethods were used, and insufﬁcient information
was presented, which contributed to Problems 2
and 3.
Factor 4 Inadequate risk visualization:
The RM ledger included the PM’s subjective evaluations. Higher-level managers were unable tomonitor risk situations of the projects, which contributed to the Problem 4.

3 RM METHOD USING QUANTITATIVEPROCESS MANAGEMENT
3.1 Basic policy
A risk is a potentiality thing, and it does not necessarily
become explicit. Therefore, the activation of risk countermeasures ahead of schedule will lead to the employment of
unnecessary labor and costs.
For establishing a good RM process, it is important to deﬁne the risk of each project and judge them objectively byusing quantitative data.
(1) Set a clear risk criterion
Objective judgment criteria are set for practicing eachInternational Workshop on Informatics ( IWIN 2018 )Figure 1: Procedure of this proposal
RM process, such as registering in the RM ledger, identifying risk explicitly, and using the quantiﬁcation method.
(2) Process Performance Baseline (PPB)
With an emphasis on historical project data, not focusing
on each project database, but the PPB focus on all historical projects’ data accumulated.
(3) Evaluating the entire project status using a statistical method
The individual risk threshold is not evaluated, but wholeproject threshold is evaluated using a statistical method.
(4) Introduction of subject matter expert (SME) and quality
assurance (QA): To solve the problem of skill shortage
related to RM, an SME, a specialist of the development
process who belongs to the engineering ﬁeld, and a QA
are introduced to discuss risks activities.
(5) Alignment Aligning the basic policies of (1) to (4) with
the RM process.

response strategy, the threshold of action taken, risk countermeasures, and priority. Priority is that the magnitude of therisk inﬂuence is sorted by order.
At the project planning stage, we also make WBS. WBS
is a key project deliverable that organizes the team’s work
into manageable sections by hierarchical decomposition of
the work to be executed by the project. At the project planning stage, we review all the tasks and set start and end datesof each task and efforts that are to be spent on the task.
When we identify all the tasks and make the WBS of a
project, we automatically know the EVM value of the current
status because the planned value (PV) and budget at completion (BAC) are calculated entirely.
In the RM process, the inﬂuence of risk is converted to“time” or “money.” In this study, we convert the inﬂuence
to “time” (minutes). The project stakeholders can understandthe amount of inﬂuence quantitatively.

3.2 Procedure
3.2.2 Quantitative Progress Management using EVM

In order to solve the issue concerning a delay in the RM,
which was identiﬁed in section 2.2, we will introduce the following RM procedures: making RM ledger and WBS, quantitative progress management using EVM, risk analysis using
LRA, and risk counter strategy. These RM Procedures areshown in Figure 1. Hereafter, we will explain the RM procedures in detail.

In Stage 2, the project is managed on a weekly basis. Usually,a progress management meeting is conducted on a weekly basis. The project manager asks project members in charge to
update WBS for the concerned week at the meeting. Subsequently, the actual value compared to the estimated value
achieved in the project is known for a particular week. At
the meeting, project members report the manager’s current
progress status by calculating these EVM values and problems that occurred, if any. This enables the managers to estimate the project’s total cost at project completion, which isalso referred to as estimate at completion (EAC)
In system development, by using a waterfall model, it has
been empirically found that the risk is often explicit at process
breaks. Therefore, at the progress meeting that is held at theend of the process, project members and SME conduct a risk
review meeting. They reevaluate the risks according to thechange of the environment at that time and update the RM
ledger.

3.2.1 Making RM Ledger and WBS
In Stage 1, we ﬁrst make RM ledger and WBS. In the systemdevelopment project, risks of the entire project are identiﬁed
at the project planning stage. Mainly, risks are obtained asa result of awareness created during the process of analyzing customer need, creating customer requirement deﬁnition,
creating project planning, and reviewing the QA Document.
The obtained risks are listed in the RM ledger of the project,
and the properties of each risk are set. These properties include risk description, probability, impact, rework time, risk

187

International Workshop on Informatics ( IWIN 2018 )3.2.3 Risk analysis using LRA
At stage 3, we can predict whether the project will fail in
the future, by using LRA. LRA is a statistical method thatpredicts the occurrence probability of an event from the sizeof accumulated data.
When we use a risk value as an explanatory variable, valuethat can take only a binary response (Yes / No), like the occurrence of project failure as a dependent variable, the probability of the inﬂuence on the occurrence of the failed project
can be determined.
According to the basic policy (3), we decided that the wholeproject risk, instead of individual risk, should be set as the
progress management threshold. Therefore, we decided that
EAC/BAC should be set as the criteria for evaluating a project’s
success or failure. When the EAC/BAC exceed the speciﬁc
trigger, the project manager should take appropriate action
under the RM.
Organizations that conduct system development projects
often have similar degrees of difﬁculty and similar scales. We
have created repositories of PPB by accumulating project dataover the past several years. We can also quantify the recoverable period for each construction period if the project is delayed.
For example, it is empirically known, “If you are projecting
for 18 months, you will recover and meet the delivery date if
the progress delay is less than 5% of the entire project period.”
We could calculate the recoverable period for each project by
employing the LRA. It was found that if the period exceeds
the value of (EAC/BAC- 100%), then it will lead to a delay inthe delivery date. Subsequently, it will be necessary to takecountermeasures that will not make the risk manifest in theschedule of recoverable limits.

Figure 2: The Causes of Failure and its Proportion

Figure 3: LRA and Inverse Estimation

3.2.4 Risk Counter Strategy

4.1 Case Study for Embedded System
Development

In stage 4, the project manager monitors the risk status at a
progress meeting and takes appropriate risk actions if necessary. If the result of LRA exceeds the threshold, then the
project manager can compensate for the project delay by immediately activating risk counter measures according to thepriority measures set in the RM ledger until the EAC/BAC
comes below the value EAC/BAC-100%. Then let the next
PDCA cycle start.

Company A introduced project management using PMBOK
for about 10 years. Development projects comprising the basic operation of the project management using the PDCA cycle, is well established.
However, in reality, even though it was called the RM process, its focus was on creating a risk matrix and completion
record for the preparation of assessment evidence. This is asituation that does not lead to effective RM.
Company A set the development process standard, basedon the waterfall model that considered the entire organization.
Additionally, at the time of the introduction of the PMBOK,
the RM plan, the creation of the RM ledger, and the risk assessment checklist were managed within the company. The
company mainly undertook derived development. The company focuses on delivering in a timely manner in a short cycle. In the cases where the delivery is delayed among QCDs,

4 EVALUATION OF THE APPLICATION
IN ACTUAL DEVELOPMENTORGANIZATIONIn Section 4, a case study wherein the proposed RM method
is applied at Company A and its effectiveness is evaluated.
R
For the application of LRA, JMP14(SAS Institute Inc.,
Cary, NC, USA), which operates on Windows PC, was used.

188

International Workshop on Informatics ( IWIN 2018 )the project is labeled as a “failed project.” The policy of theorganization is to prevent delayed delivery.
Company A’s customers do not present their requirements
clearly. Therefore, the project manager in Company A must
analyze customer needs and make a feasible completion plan,and formulate the budget and set the construction durationaccordingly.
Subsequently, to implement the requirement deﬁnition, thecompany undertakes project planning and conducts a QA review for implementing the requirement deﬁnition and projectplan.
After a discussion with the PM, SME, QA, and project
members, the company rules out the related risks and prepares
the RM ledger based on the risks.
The project manager holds a progress meeting on a weekly
basis. The project manager asks all the project members incharge to update the WBS. Subsequently, the project manager calculate the projected EAC and EAC/BAC. Then they
update the RM ledger with latest data and entire EVM value
is converted into “time.”
The development period at Company A is set at around 618 months, depending on the scale of development. We calculate the probability of project failure by using the LRA.
For example, when a project construction term is 18 months,
the predicted probability of failure would be as shown in Figure 2. The cross-hair tool indicates that the prediction probability is 0.421 at the time of 6% delay. In other words, if
the project is delayed by 6%, then the delivery date will bedelayed with a probability of 42%.
A 42% probability is difﬁcult to employ as a psychologicalmilestone to activate risk countermeasures. A value at which
the prediction probability becomes 50% was calculated using
an inverse estimation of LRA, and it was 6.41%. Conversely,
it means that “If the project is delayed by 6.41%, then there
can be a 50% chance of a recovery.” This state is shown in
Figure 3.
Actually, if the recoverable range exceeded the threshold(like 6.41%), they activated risk countermeasures to reduce
the project delay in descending order of inﬂuence until thevalue fell below the threshold to ensure that the value ﬁt within
the recoverable range.

5 DISCUSSION

4.2 Evaluation of EffectivenessIn this section, we discuss whether the factors presented in
section 2.2 have been resolved.

Figure 4: No. of Delivery Delayed Projects

Figure 5: Transition of Contingency Budget
on to prevent project settlement deﬁcit. It is preferable not touse contingency funds because it is recorded as a proﬁt if notused. In the ﬁrst year, after the introduction of our proposed
method, we consumed nearly 30% of the contingency. It was
suppressed to 20% or less in the third year. Since countermeasures are given priority in the order of the risk of damage
due to anticipated risk, we believe that it contributed to theprevention of major deﬁcit in projects. We conﬁrmed that the
proposed method also improves cost.

Figure 4 shows the three-year trend of delayed project delivery in Company A. Although the duration for which the
project delays were observed is small, the number of projects
subjected to delivery time delay has deﬁnitely been reduced.
Meanwhile, Company A did not introduce any other measuresduring this time, but the proposed method has been introduced. Company A considered the transition shown in Figure 4 which this can be regarded as an improvement that isachieved through the proposed method.
Figure 5 shows the transition of contingency for 3 years,
after the introduction of the proposed method in Company A.
Contingency is a reserve expenditure fund that can be drawn

5.1 Timely and Correct Risk Report
In this study, project risks at each process break are identiﬁed at the project planning stage and reviewed by PM, SME,
and QA during the RM meeting. The RM ledger and WBS
are updated. Subsequently, we obtain the EVM value and objectively calculate appropriate parameters for the project. At
these meetings, the overall project risk and progress status arereported in a timely manner. The project manager can judge
the status promptly.
Thus, nobody can reduce the number of risks or hide critical risks while reporting them; the risks are reported in an

189

International Workshop on Informatics ( IWIN 2018 )REFERENCES

accurate and timely manner, thereby resolving the issue ofdelayed and erroneous risk reports.

5.2 PM’s Risk Judgment Skill
In this study, to identify and judge risks, an SME and a
QA reviewer are assigned to the project. Instead of a project
manager, they support and perform RM processes, including
risk identiﬁcation and the activation of risk countermeasure.
In addition, risk is evaluated objectively by using parametric data to avoid the biases of project managers that might result from their assumed expertise over all the technical ﬁelds.
It implies the resolution of the issues concerning skill shortage in project managers and their lack of risk-judgement skill.

5.3 RM Method in an Organization
In this study, the organization’s historical project data are
accumulated through the PPB. Recoverable range of each project
period calculated by LRA method are accumulated.
Project managers or higher-level managers can judge risks
objectively and activate risk countermeasures. The proposed
RM method facilitated the uniﬁcation of the organization’s
RM method. Thus, this method contributed toward the effective implementation of the RM method in an organization.

5.4 Full Visualization of Risk
In this study, after identifying the risk through an upstream
process, we monitored the risks through weekly meetings andvisualized the magnitude of an allowable recovery range byusing statistical methods. Furthermore, the magnitude of therisk effect was converted into “time.” We visualized the possibility of leadtime to delivery delay. Thus, the issue of fullvisualization of risk was resolved.

6 CONCLUSIONSIn this research study, we analyzed four actual RM processes carried out at the system development site. Additionally, the study identiﬁed the factors that contributed to delayed
RM, despite the introduction of the correct RM process at theorganization. Subsequently, we proposed the RM method using a quantitative process management approach that included
PM, EVM, and LRA.
When we applied the method to the actual embedded development projects, we could verify and conﬁrm the improvement effect on the reduction of the number of projects withdelayed delivery times and a decrease in contingency. Thus,
the proposed method is considered potentially effective.
This proposed method can be introduced easily in any organization implementing process improvement. In the future,
it is necessary to increase case examples, evaluate effectiveness, and make improvements to the existing RM process.

190[1] Nikkei Computer, Survey on Information Actual Condition (2003)[2] Standish Group International, Inc,“CHAOS Summary
2009,” http://www.standishgroup.com, (2009)[3] K.H. Rose, “A guide to the project management body ofknowledge (pmbok guide) ﬁfth edition,” Project management journalɼvol.44ɼno.3ɼ(2013)[4] Project Management Association Japan, P2M Program,Project Management Standard guidebook (2014)[5] Martin Tomanek and Jan Juricek , “Project RM model
based on prince2 and scrum frameworks,” International
Journal of Software Engineering Applications (IJSEA),
Vol.6, No.1 (2015)[6] B.W. Boehm, “Software risk management: principles
and practices,” IEEE software, vol.8, no.1, pp.3241(1991)[7] R.C.Williams, J.A.Walker, and A.J.Dorofee, “Putting
risk management into practice,” IEEE software, vol.14,
no.3, pp.75-82,(1997)[8] ISO9001(2015).Quality Management System Requirement. Japanese Standards Association.
[9] Chrissis,M.B. Konrad,M. and Shrum,S. CMMI for development: guidelines for process integration and product improvement, Pearson Education.(2011)[10] Rasool,M, Franck,T. Denys,B. and Halidou,N. Methodology and tools for risk evaluation in construction
projects using risk breakdown structure, European J.
of Environmental and Civil Engineering, vol.16, pp.7898.(2012).
[11] Menezes,J, Gusm o,Jr,C. and Moura,H De ning indicators for risk assessment in software development projects, Clei electronic J., vol.16, no.1, pp.1721.(2013)[12] Galway,L. Quantitative risk analysis for project management, a critical review, (2004).
[13] Zafar. K Software project risk management by using
Six Sigma approach, Int.J. of Engineering Research andGeneral Science, vol.3, no.4, pp.17-21 (2015)

International Workshop on Informatics ( IWIN 2018 )Trust Representation under Confusion and Ignorance
Kazuhiko Ohkubo1 , Tetsuhisa Oda2 , Yuki Koizumi3 , Tetsushi Ohki4 ,Masakatsu Nishigaki4 , Toru Hasegawa3 and Yoshinobu Kawabe5
1

NTT Secure Platform Laboratories, NTT Corporation
3-9-11 Midori-cho, Musashino-shi, Tokyo 180-0012, Japan
E-mail : ohkubo.kazuhiko@lab.ntt.co.jp
23

Department of Business Administration, Aichi Institute of Technology
2-49-2 Jiyugaoka, Chikusa-ku, Nagoya, Aichi 464-0044, Japan
E-mail : oda@aitech.ac.jp

Graduate School of Information Science and Technology, Osaka University
1-5 Yamadaoka, Suita, Osaka 565-0871, Japan
E-mail : { ykoizumi, t-hasegawa }@ist.osaka-u.ac.jp
4

Graduate School of Science and Technology, Shizuoka University
3-5-1 Johoku, Naka-ku, Hamamatsu, Shizuoka 432-8011, Japan
E-mail : { ohki, nishigaki }@inf.shizuoka.ac.jp

5Department of Information Science, Aichi Institute of Technology
Yachigusa 1247, Yakusa-cho, Toyota, Aichi 470-0392, Japan
E-mail : kawabe@aitech.ac.jp

Abstract - Handling trust among agents/users is an important issue in the Internet, and many researchers have tackled
on describing and analyzing trust values. Marsh and Dibben
used a real value in the range of [−1, 1) to express a trust
value, where value 1 corresponds to a state for an agent fully
trusted and value −1 corresponds to a state of a complete distrust. However, there are cases where such a one-dimensional
representation of trust is not sufficient; for example, we cannot give a proper trust value on an agent if the agent is both
trusted and distrusted — this kind of confusion may happen
since trust is a property closely related to human’s impressions. Also, ignorance on an agent may affect giving a trust
value. To deal with trust values for cases where some confusion or ignorance is allowed, this study employs a multidimensional representation of Fuzzy-set Concurrent Rating
method, FCR for short. The FCR method is originally developed for the study of mathematical psychology, and it canhandle an answer like ”basically good, but somewhat bad simultaneously”. With the FCR method, we can describe trust
values in a two-dimensional way, and this enables a proper
description of trust states. In this study we introduce how torepresent a trust value, and we discuss how the trust value corresponds to a one-dimensional trust value defined by Marshand Dibben.

in the major disaster of the Great East Japan Earthquake of2011, many messages on human life and relief are exchanged
through social media. For such situations, it is an important requirement to deal with the trustworthiness on participants/messages for communications.
Marsh and Dibben [1] have formalized and classified sometrust notions as follows:• Trust: a measure of how much an agent believes a trustee,• Distrust: a measure of how much an agent believes
that the trustee will actively work against the agent in agiven situation,• Untrust: a measure of how little the trustee is actuallytrusted, and• Mistrust: a misplaced trust.
In their study the degree of trust is modeled with a score in[−1, 1), and the notions of trust, distrust and untrust are formally defined as follows:• Trust: a state where the trustee’s trust value is morethan a threshold value and enough to cooperate,• Distrust: a state where the trustee’s trust value is negative, andKeywords: On-line Trust, FCR Method, Fuzzy Logic, I/Oautomaton• Untrust: a state where the trustee’s trust value is positive but not enough to cooperate.
For the notions of trust and distrust, Lewicki et.al [2] suggestthat trust and distrust are entirely separate dimensions, and as
described in [1][2], low distrust is not same as high trust, andhigh distrust is not same as low trust. From this discussion, we
can see that there are cases where one-dimensional expression
is not sufficient for trust values.

1 Introduction
Nowadays a huge amount of sensors and devices are connected to the Internet and various messages which may contain private information are massively exchanged. Also, people often exchange important messages actively; for example,
191International Workshop on Informatics ( IWIN 2018 )

We describe an example situation where one trusts and distrusts a message simultaneously. Suppose that in a large scaledisaster you have received a message that describes the current situation of your friend. The content of the message wasenough consistent, and you understood that the message wasprobably true. However, someone later told you that the message was not sent from a correct sender. In this case, you
may trust the message from the viewpoint of message’s consistency, but simultaneously you may not trust the message
from the viewpoint of sender’s integrity.
Trust is a property closely related to human’s impressions.
Hence, we consider that a mathematical psychology’s method
for impression formation is well applicable to describing trust
values. Oda [3][4][5] developed such a method in the filed offuzzy set theory. The Fuzzy-set Concurrent Rating method,
FCR for short, enables us to measure and analyze human’s
impressions. By applying the FCR method to trust representation, we can describe trust and distrust notions properly.
Furthermore, we can introduce some “finer” untrust notions.
That is, it is possible to define two sorts of untrust notions:
The following word is for describing your evaluation on
the government of your country. Please circle the number that best matches your feelings.


Figure 1: Conventional questionnaireanswers how much s/he thinks so. The range of each confidence value is from 0 to 1. Then, we calculate an integration value from a set of the confidence values by applying the
fuzzy inference. From the theoretical point of view we haveno restriction with regard to the dimension (i.e. the numberof items), but from the application standpoint dimensions ofabout 2 or 3 are considered practical; for example, two dimensions by “true” and “false”, three dimensions by “agree”,“disagree” and “I do not know”, and so on. In the rest of thispaper, we have discussions for the case of two dimensions.

• Untrust under confusion: a state where the trustee isboth trusted and distrusted, and• Untrust under ignorance: a state where the trustee isignored; that is, the trustee is both little trusted and little
distrusted.
The original untrust notion by Marsh and Dibben corresponds
the untrust under ignorance. In this paper, we give a FCRbased semantics for the trust notions.
This paper is organized as follows. Section 2.1 describes
an overview of the FCR method. In Section 3, we describe
the trust notions shown in [1], and we apply the FCR method
to represent a trust value. Finally, we discuss the validityof our trust representation in Section 4; also, an analysis fortime-related trust notions such as mistrust or swift trust is discussed.

2.1.1 HLS Model
The FCR method employs the Hyper Logic Space model,
HLS for short, as a logic space of the multiple-dimensional
multiple-valued logic. Figure 3 shows a two-dimensional space
based on two metrics of true (T ) and f alse (F ). An observation (t, f ), or an observed point, is a point in T ×F , where wehave t, f ∈ [0, 1]. Note that t and f are independent; that is,
we do not introduce restrictions such as t+f = 1 or t+f ≤ 1.
A domain {(t, f ) | t, f ∈ [0, 1] ∧ t + f > 1} is called the region of contradiction. Also, a domain {(t, f ) | t, f ∈ [0, 1] ∧t + f < 1} is called the region of ignorance, or the region ofirrelevance. A domain {(t, f ) | t, f ∈ [0, 1] ∧ t + f = 1}
is called the numerical truth value space, or the consistentregion. If we regard observations A = (tA , fA ) and B =(tB , fB ) as truth values, logical operations over HLS are defined as:

2 Preliminaries
2.1 Fuzzy-Set Concurrent Rating Method
The rating scale method (Fig. 1) is often used for questionnaire, where there are items such as “poor”, “fair”, “average”,“good”, and “excellent” and a respondent chooses one fromthem. This is a simple way, but this method has a problem
that the respondent tends to choose a “middle” item of a scale,
which makes an analysis difficult. There are two cases where
this problem occurs. The first case is that the respondent has
multiple candidates for an answer. Especially, if the candidates are at both extremities, then the respondent may choose
one of them (forcibly) or a middle item. The chosen middle item is not a true answer of the respondent; moreover, themiddle item usually has a label such as “average”, “neutral”
or “I do not know”, and this makes the analysis more difficult. Another case is that the respondent does not have enoughknowledge/interest for judgment. In any case, we cannot determine if the chosen middle item is the true answer or not.
In the FCR method, a respondent is requested to answer
the confidence for each item (Fig. 2); that is, the respondent

A∨B
A∧B
¬A

= (max(tA , tB ), min(fA , fB )),= (min(tA , tB ), max(fA , fB )), and= (1 − tA , 1 − fA ).

For details on the logical operations, see [6].
2.1.2 Integration Value and Degree of Contradiction
For observation (t, f ), truth value t and falsity value f are
given independently. From these values, we need to calculate
an “actual” truth value. This special truth value is called an
192

International Workshop on Informatics ( IWIN 2018 )The following word is for describing your evaluation on
the government of your country. Please check a markfor each of the seven scales below. Each scale represents the ratio how much you think so. The left edgeof a scale represents that the scale never matches yourfeelings, while the right edge corresponds to a complete
match to your feelings. Note that the sum of the values
need not be 1.

Figure 4: Graphical calculation for integration values
Figure 2: Rating with the FCR method

These methods are sometimes combined; for example, to calculate an integration value we employ I1 (t, f ) and I3 (t, f ) for
the region of irrelevance and for the region of contradiction,
respectively. Note that we have I1 (t, f ) = I3 (t, f ) for theconsistent region. In the FCR method, we can calculate theintegration values I1 (t, d), I2 (t, d) and I3 (t, d) in a graphicalmanner (see Fig. 4). For I2 (t, d), we draw a perpendicularline from the observation (t, d) to Fig. 4’s diagonal line, andwe read the value of “Degree 1”1 .
Another important special value, which is called the degreeof contradiction, has been introduced in the FCR method. In
the field of personality psychology, we allow situations like“I like it but I don’t like it” or “I don’t like it but I don’t dislike it.” This kind of confusion is formulated and measured
with a value called the degree of contradiction [3]; we employ
this value to handle situations like “This message is trustable
but simultaneously not trustable.” The degree of contradiction C, which is more precisely the degree of contradictionirrelevance, has the value of C = 1 for the case of total contradiction, C = 0 for the case of t + f = 1, and C = −1 forthe case of complete ignorance, respectively. We often use
the definition for the degree of contradiction given by:

Figure 3: Two-dimensional HLS model
integration value. There are several ways to calculate integration values, and it is chosen properly according to the analysis
target. Mainly, the following methods are used:• Simple scoring method: In this case, integration value
I1 is defined as: 0.5
if t + f = 0,t
I1 (t, f ) =otherwise;
t+fC(t, f ) = t + f − 1• Reverse-item averaging method: Integration value I2 isgiven by calculating the average of t and 1 − f , where t
is a positive measure and f is a negative measure. Thatis, we have:
I2 (t, f ) =• Inverse scoring method: For measures t and f , we employ their negations f ′ = 1 − t and t′ = 1 − f . Then,
we apply the simple scoring method for t′ and f ′ . Thatis, integration value I3 is given as: 0.5
if t + f = 2,
1−f
I3 (t, f ) =otherwise.

2−t−fwhich represents the distance between the observation (t, f )and the consistent region.
1 For I (t, d), we draw a line from (0, 0) to (t, d); I (t, d) is the value of1
1
Degree 1 at the crossing point with the diagonal line. For I3 (t, d), we draw
a line from (1, 1) and we read the value of Degree 1 at the intersection withthe diagonal line.

t + (1 − f );2
193

International Workshop on Informatics ( IWIN 2018 )2.2 Four Trust Notions

In this formula, CTx (y, α) is a cooperation threshold which
is given by:

Marsh and Dibben[1] introduced four trust notions: trust, distrust, untrust, and mistrust, and they formulated the condition for each of the trust notions. To formalize the trust notions, Marsh and Dibben have used the following functions
with range [−1, 1):

CTx (y, α) =Riskx (α)\
Compx (y, α) + Tx (y)× Ix (α)

where Riskx (α) is a value on x’s perceived risk in situation
α, and Compx (y, α) is a x’s evaluation value on how muchy has a competence for α. From this definition, we can seethat the cooperation threshold for trust is set high if the valuesof risk Riskx (α) and importance Ix (α) are high. Also, thecooperation value becomes low if the value of Compx (y, α)
is high, that is, if the truster x believes that the trustee y candeal with things in situation α. The cooperation value also
becomes low if the value of T\
x (α) is high.
Finally, the notion of mistrust is introduced in [1]. This isa misplaced trust, and it is discussed in Section 4.2.

• General trust Tx (y): how much x trusts y; and• Situational trust Tx (y, α): how much x trusts y in situation α.
The situational trust Tx (y, α) is specifically defined as:\
Tx (y, α) = Ux (α) × Ix (α) × T
x (y)where Ux (α) is the utility function to agent x for situation α,\
Ix (α) is the importance of α to x, and T
x (y) is x’s estimationon the trust value for trustee y [7]. Note that Ux (α), Ix (α) and\Tx (y) range over [0, 1], [0, 1] and [−1, 1), respectively. For
example, let us consider the “two agents and two pieces of furniture” example shown in [7]. There are two agents (A and B)
in a room, each with the task of carrying out one piece of furniture. Each agent cannot move his piece of furniture alone,and each piece of furniture needs two agents to move it. Eachagent must consider to cooperate with the other in order to get
the job done. However, an agent may choose not to cooperate, and in this case there are no sanctions. This situation is labeled with α. Suppose agent B is relatively trustable for agent
A, and this is estimated with T\
A (B) = 0.63. Agent A considers the importance on the situation is fair; i.e. IA (α) = 0.5.
The furniture of agent A is his own, and he believes that theutility to move it from the room is very high. This is estimatedwith UA (α) = 0.8. Summarizing, the estimated situational
trust is TA (B, α) = 0.8 × 0.5 × 0.63 = 0.252. If Tx (y) is
greater than a value called a cooperation threshold, then y istrusted by x; similarly, we say y is trusted by x in situation α
if Tx (y, α) is greater than the threshold value.
Distrust is a measure of how much an agent believes thatthe trustee will actively work against the agent in a given situation. That is, if agent x distrusts y in situation α, the agent x
expects that y will work to make sure the worst (or at least not
the best) will happen in situation α. In other words, distrust
is an active judgment in the negative intentions of the other.
This is defined as:

3 FCR-based Trust Representation
In the previous section we have shown the functions such as
Tx (y) and Tx (y, α) to explain a formalization of three trust
notions (except the notion of mistrust). A trust value is an element in [−1, 1), and the notions of trust, distrust and untrust
are defined one-dimensionally. However, as described in [1],low distrust is not same as high trust, and high distrust is not
same as low trust. In this section we assume that trust anddistrust are entirely separate dimensions, and we use the FCR
method in Section 2.1 for trust representations.
In this paper we use the set of trust values T rust and theset of distrust values DisT rust, which are defined as:
T rust = DisT rust = { v | 0 ≤ v ≤ 1}.
An observation is an element of T rust × DisT rust. For any
observation o ∈ T rust × DisT rust, we can see that
1. If o is around (1, 0) ∈ T rust × DisT rust, then theobservation has a high trust value and a low distrust
value;
2. If o is around (0, 1), then the observation has a low trust
value and a high distrust value;
3. If o is around (0, 0), then both of the trust and distrust
values are low;
4. If o is around (1, 1), then both of the trust and distrust
values are high; andTx (y, α) < 0 =⇒ Distrust(x, y, α).
We can see that the notion of distrust is defined as a casewhere Tx (y, α) is negative.
Untrust is a measure of how little a trustee is actually trusted.
If we say a trustee is untrusted, then the truster has little confidence in the trustee acting in their best interests in a situation.
In other words, the truster cannot determine if the trustee istrustable. In such a situation, it is not enough for a truster tocooperate with a trustee. Formally, a state of untrust is definedwith:

5. If o is around (0.5, 0.5), the both kind of values aremoderate.
Here we have five cases, and it is natural to understand thatthe first case corresponds to Section 2.2’s trust state while thesecond case corresponds to a distrust state. For cases 3, 4 and
5, we cannot determine whether a trustee is actually trustable;
hence, these cases basically correspond to a state of untrust.
However, for cases 3, 4 and 5, if the distrust value is greaterthan the trust value, it is considered appropriate that the observation belongs to a state of distrust.

(Tx (y, α) > 0 ∧ Tx (y, α) < CTx (y, α))=⇒ U ntrust(x, y, α).
194

International Workshop on Informatics ( IWIN 2018 )diagonal line is given by |t+d−1|, which is the absolute valueof the second element of the resulting point (i, c). This value
coincides (the absolute value of) the degree of contradictionirrelevance in the FCR method.

4 Discussions
4.1 Validity of Two-Dimensional Trust
Representation
Let (t, d) ∈ T rust × DisT rust be an observation and (i, c)be its corresponding resulting point by the transformation of
the previous section. We can see that the “actual” trust value
i = t − d is calculated by subtracting the degree of distrustfrom the degree of trust; we believe that this is consistent with
the intuition. Furthermore, we can explain the trust notions in
Section 2.2 properly by defining:

Figure 5: From 2D representation to 1D representation• A state of trust is a state with i ≥ CT , where CT is acooperation threshold; andPoints (1, 0) and (0, 1) are both ends of the diagonal line
shown in Fig. 3. Also, we can see that every point in the
diagonal line is ideal in the sense that the degree of trust t
and the degree of distrust d follow the consonance law; thatis, t + d = 1. In this study we transform the diagonal line
to the measure by Marsh and Dibben. Specifically, we movepoints (1, 0), (0, 1) and (0.5, 0.5) to (1, 0), (−1, 0) and (0, 0),
respectively. Note that (1, 0), (−1, 0) and (0, 0) correspondto trust values 1, −1 and 0, respectively. Observation (t, d) ∈T rust × DisT rust is mapped to (t − d, t + d − 1) by• A state of distrust is a state with i < 0.
The second element c = t + d − 1 of (i, c) is the degree ofcontradiction-irrelevance in the FCR method, and the value
enables us to introduce a new sort of untrust notion. In thefield of fuzzy logic, it is considered that a state of contradiction, where the value of c is high, is caused by an information overload. Also, a state of irrelevance, where the valueof c is negative, is considered due to a lack of information.
This study considers that an information overload is closely
related to an excessive interest in a trustee. Actually, if you
are excessively interested in a trustee, you may have many evidences for judgment. Some of the evidences might increase
the trust value on the trustee, but simultaneously there may
exist another evidence that increases the distrust value. Also,
this study considers that a lack of information is caused byan ignorance. If you are not interested in the trustee and you
do not know anything, you cannot discuss whether the trustee
is trustable or not. At least, in such a case, both of the trust
value and the distrust value remain low, since there exist no
evidences that increase the trust/distrust value.
With the degree of contradiction-irrelevance, we can further classify the notion of untrust. This study introduces thefollowing two types of untrust notions:[() { ( ) ( ) } ( √ )]2
1
cos π4 − sin π4
t
1
−
+ 2
× √
π
π
2
sin 4
cos 4
d
0
0
2
(
)
t−d
=
t+d−1
and we call the resulting point (i, c); see Fig. 5. Note that
we have c = 0 if the original point (t, d) is in the consistentregion (i.e. t + d − 1 = 0).
For the resulting point (i, c), the first element i = t − d represents the “actual” trust value given by Marsh and Dibben,
and this is calculated with the reverse-item averaging method
in Section 2.1.2; actually, the range of integration value I2 (t, d)
is [0, 1] and the value of i is calculated by normalizing I2 (t, d)
to be a value in region [−1, 1]. The second element c =t + d − 1 coincides the degree of contradiction-irrelevancein the FCR method.
As described in Section 2.1.2, we can calculate the integration values in a graphical manner. Observing the graphical calculation, points in the same perpendicular line have thesame integration value with regard to I2 . For example, observation A = (t, d) and its nearest point on the diagonal line
A′ = (• Untrust under confusion: This is a state where a trustee
is both trusted and distrusted. Formally, this is a statewith 0 < i < CT and c ≥ 0; and• Untrust under ignorance: This is a state where the trustee
is ignored; in other words, the trustee is both little trusted
and little distrusted. Formally, this is a state with 0 <i < CT and c < 0.

t + (1 − d)t + (1 − d)
, 1−
)
2
2

The original untrust notion in [1] is considered as the notionof untrust under ignorance. On the other hand, this paper introduced a new kind of untrust notion from the viewpoint of“contradiction”, and this result is obtained based on the FCR
method.

have the same integration value, which means that i = t − d
t + (1 − d)t + (1 − d)and i′ =− (1 −) are equivalent.
22′
However, for A and A , the distance from the diagonal line
is different. The distance between the observation and the195International Workshop on Informatics ( IWIN 2018 )

automaton mistrust(id, alpha)signature
output move(i: Nat,
pt: VL, pd: VL,
ev:Event,
dt: VL, dd: VL)
statest: Array[Nat, VL],
% Sort VL is the set ofd: Array[Nat, VL],
% real values [-1, 1]
step: Nat := 0,stateOfId: State := InitState(id, alpha)so that (\A i:Nat (t[i] = 0 /\ d[i] = 0))

5 Conclusion
In this paper, we introduced a trust representation based on atheory for impression formation — the FCR method. A trust
value was given as an observation in the two-dimensional
HLS, where we employed two metrics of “trust” and “distrust”. We discussed the validity of this representation by
showing a mapping to the one-dimensional representation by
Marsh and Dibben. For the untrust notion, we introduced a
new classification; that is, there are two sorts of untrust notions called the untrust under confusion and the untrust under
ignorance.
In this study we discussed how to represent a single truststate, but we are actually interested in trust transitions. For
future work, we are planning to formalize and analyze timerelated trust notions such as swift trust. We believe that results from theory of distributed algorithms, such as analysis
methods for safety properties or liveness properties, are applicable to analyze trust notions. In this paper we described
an automaton (in Fig. 6) as a first step to define and analyze
the trust properties. It gives one idea for the analysis of timerelated trust, but we do not say that it is a complete definition;
it should be refined and sophisticated.
It is also crucial to conduct experiments to apply our representation of trust to real situations. We consider that dealing
with a disaster situation is especially important, since adversaries and malicious actors often seek to exploit the vulnerable situation. For such situations it is important to evaluate
the trust of messages and participants with our trust representation.

transitions
output move(i, pt, pd, ev, dt, dd)
prei = step/\ pt = t[i]/\ pd = d[i]/\ (0 <= pt + dt /\ pt + dt <= 1)/\ (0 <= pd + dd /\ pd + dd <= 1)/\ condition(id, alpha,i, ev, pt, pd, dt, dd,
stateOfId)
eff t[i+1] := t[i] + dt;
d[i+1] := d[i] + dd;
step := step+1;
stateOfId := change(id, alpha,stateOfId, i, ev)

Figure 6: IOA specification

4.2 Mistrust
Mistrust is a state in which initial trust has been betrayed;
more precisely, the notion of mistrust can be considered as“either a former trust destroyed, or former distrust healed”
since the trustee may not have had bad intentions and it is
not always “betrayed”. This is considered as a property withregard to a change of trust value over time.
We consider that it is possible to model a change of trust
value with IOA [8], which is a formal specification language
based on I/O-automaton theory [9][10]. For example, the automaton in Fig. 6 models the change of actor id’s trust value
in situation alpha. In this specification, step is a clock,and t[i] and d[i] are the trust value and the distrust value
at clock i, respectively. The range of t[i] and d[i] is[0, 1]. The variable stateOfId is actor id’s state and its
initial value is given by InitState(id, alpha). The
only action is move(i, pt, pd, ev, dt, dd), andan occurrence of this action means:
Actor id’s trust value is (pt, pd) at clock i, andthere is an occurrence of event ev which changes thetrust value to (pt + dt, pd + dd) at clock i+1.

Acknowledgment
This work has been supported by the National Institute of Information and Communications Technology (NICT) in Japan(Contract No. 193).

REFERENCES[1] S. Marsh and M. R. Dibben, “Trust, untrust, distrust andmistrust – an exploration of the dark(er) side,” in Proceedings of the Third International Conference on TrustManagement, iTrust’05, (Berlin, Heidelberg), pp. 17–
33, Springer-Verlag, 2005.
[2] R. J. Lewicki, D. J. B. McAllister, and R. J. Bies, “Trustand distrust: New relationships and realities,” Academyof Management Review, vol. 23, pp. 438–458, 1998.
[3] T. Oda, “Fundamental characterestics of fuzzy-set concurrent rating method,” Journal of Japan Association forManagement Systems, vol. 12, no. 1, pp. 23–32, 1995. In
Japanese.
[4] T. Oda, “Fuzzy set theoretical approach for improving
the rating scale method : Proposing and introducing the
FCR-method and the IR-method as novel rating methods,” Japanese Psychological Review, vol. 56, no. 1,
pp. 67–83, 2013. In Japanese.
[5] T. Oda, “Measurement technique for ergonomics, section 3: Psychological measurements and analyses(3) measurements and analyses by kansei evaluation,”

To analyze time-related trust properties such as mistrust or
swift trust [11][12], a verifier should appropriately describe
the predicate condition in the precondition part of action
move. For example, in verifying swift trust, the verifier should
define condition to satisfy that t[step] − d[step]
becomes greater than a cooperation threshold within a given
short period. We consider that it can be modeled as a livenessproperty defined with a trace set. Swift trust is attracting attention for disaster situations [13][14][15], and its importanceis getting larger recently. It is a future work to formalize thevarious trust notions for disaster situations.
196

International Workshop on Informatics ( IWIN 2018 )The Japanese Journal of Ergonomics, vol. 51, no. 5,
pp. 293–303, 2015. In Japanese.
[6] T. Oda, “A proposal for multidimensional multivalued
logic space: Hyper logic space — Extended fuzzylogic for the fuzzy-set concurrent rating method —,”Japanese Journal of Industrial Management Association, vol. 49, no. 3, pp. 135–145, 1998. In Japanese.
[7] S. P. Marsh, “Formalising trust as a computational concept,” tech. rep., University of Stirling, 1994.
[8] A. Bogdanov, “Formal verification of simulations between I/O-automata,” Master’s thesis, Massachusetts Institute of Technology, 2000.
[9] N. A. Lynch, Distributed Algorithms. Morgan Kaufmann Publishers, 1996.
[10] N. Lynch and F. Vaandrager, “Forward and backward
simulations — part I: Untimed systems,” Information
and Computation, vol. 121, pp. 214–233, Sept. 1995.
[11] D. Meyerson, K. E. Weick, and R. M. Kramer, Swift
Trust and Temporary Groups in Trust in Organizations:
Frontiers of Theory and Research. SAGE, 1995.
[12] J. Wildman, M. Shuffler, E. Lazzara, S. Fiore, and
S. Burke, “Trust development in swift starting action
teams: A multilevel framework,” Group & organization
management, vol. 37, no. 2, pp. 137–170, 2012.
[13] Y. Murayama, “Issues in disaster communications,”
Journal of Information Processing, vol. 22, no. 4,
pp. 558–565, 2014.
[14] M. G. Busa, M. T. Musacchio, S. Finan, and C. Fennell,“Trust-building through social media communications
in disaster management,” in Proceedings of the 24th
International Conference on World Wide Web, WWW’15 Companion, (New York, NY, USA), pp. 1179–1184,
ACM, 2015.
[15] F. Lemieux, “The impact of a natural disaster on altruistic behaviour and crime,” Disasters, vol. 38, pp. 483–
499, July 2014.

197

International Workshop on Informatics ( IWIN 2018 )Elliptic Curves Suitable for Elliptic Curve Signature
Masaaki Shirase††

School of Systems Information Science, Future University Hakodate, Japanshirase@fun.ac.jp

Abstract - In an elliptic curve signature using 256-bit primep, thousands of modular multiplications X · Y mod p performed according to a signature algorithm are dominant. Therefore, how to speed up multiplication and mod p computations is one of the objectives of researches on elliptic curve
signature implementations. One of speeding up method of reduction mod p is to use a special form of prime called pseudo
Mersenne prime such that p = 2n − k, where k is a smallvalue. However, in an elliptic curve signature, computationof mod l with another integer l, which is the order of a basepoint, also requires although the number is a few.
In this paper, the authors give a program to construct elliptic curves such that reduction mod l can be computed as
mod a pseudo Mersenne prime and give an example of such
curves.
Keywords: Elliptic Curve, Elliptic Curve Signature, Modular Multiplication, Pseudo Mersenne Prime

1 INTRODUCTION
Recently, elliptic curve signatures as ECDSA are often usedin the TLS communication and block chains. In Europe, it
was decided to use an elliptic curve signature in the V2X communication [5]. However, signature verification in the V2X
communication requires further speeding up.
Elliptic curve is a cubic curve given by Weierstrass form(y 2 = x3 +ax+b) or Montgomery curve (By 2 = x3 +Ax2 +x) [9]. A remarkable feature of elliptic curve is that an operation + is defined [14]. Dominant processes of the operation+, which is explained in Sec.2.2, are modular multiplications
X · Y mod p for X, Y ∈ Fp = {0, 1, 2, · · · , p − 1}, where
p is typically a 256-bit prime. The calculation of it is divide
into
Z mod p .
Y and W ← |{z}Z ← |{z}X · |{z}|{z}
256 bit 256 bit

512 bit

In public key encryptions as ECElGamal and key agreements as ECDH using elliptic curves, required reduction ismod p only. On the contrary, in elliptic curve signatures as
ECDSA, required reductions are not only mod p but also modl, where l is the order of a base point. As far as the authors
know, the efficiency of mod p is taken into account in theconstruction of an elliptic curve for ECCs, but the efficiency
of mod l is not considered.
The purpose of this paper is to make a program to findMontgomery curves such that mod l can be computed by same
way of mod p for pseudo Mersenne prime and A = 6, 10,14, and 18, and to give examples of such curves.
Sec.2 explains the definition of elliptic curve, operation +,scalar multiplication, coordinate system, secure elliptic curve,
and Curve25519. Sec.3 introduces ECDSA that is the mostpopular elliptic curve signature. Sec.4 introduces efficient reduction methods. Sec.5 is the contribution of this paper. Sec.5
proposes a requirement for elliptic curves to be suitable for
ECDSA, makes a program to find elliptic curves that meetthe requirement, and gives examples of such curves. Sec.6
concludes this paper and gives future works.

2

ELLIPTIC CURVE

2.1

Definition of Elliptic Curve

Elliptic curve is a cubic curve given by
E : y 2 = x3 + ax + b (Weierstrass form)(2)

E : By 2 = x3 + Ax2 + x (Montgomery curve)

(3)

or

with variables x, y. When used in cryptosystems, Montgomery
curve (3) is often selected because it can reduce the cost of
cryptographic processes.
For a prime p, the set Fp is defined as Fp = {0, 1, 2, · · · , p−1}. The set E(Fp ) is of points on E be considered on Fp ,where E(Fp ) includes a special point O called the point at
infinity.
The order of E(Fp ), #E(Fp ), is defined as the number ofpoints in E(Fp ). The trace of E(Fp ) is defined as an integer
such that#E(Fp ) = p + 1 − t.

(1)

256 bit

Dominant processes of elliptic curve cryptosystems (ECCs)
including elliptic curve signatures are thousands of modular
multiplications. Therefore, it is important to speed up (1) tospeed up processes of elliptic curve signatures. As explainedin Sec.2.3, using Montgomery curve rather than Weierstrass
form reduces the number of modular multiplications required
for signature generation and verification. Moreover, when acoefficient A of Montgomery curve is 6, 10, 14, and 18, thenumber of modular multiplications is further reduced.
Montgomery reduction [8] that can be applied to arbitrary
odd number p is a famous method for reduction mod p. Also,
when p is a pseudo Mersenne prime written as p = 2n −k, k < 2n/2 , reduction mod p can be very efficient.

When a Montgomery curve E : By 2 = x3 + Ax2 + x on
Fp has the trace t, another Montgomery curve E 0 : B 0 y 2 =x3 + Ax2 + x has the trace t or −t. In other words, we have
#E 0 (Fp ) = #E(Fp ) or 2p+2−#E(Fp ). When #E 0 (Fp ) =2p + 2 − #E(Fp ), E 0 is called the twist of E.
199

International Workshop on Informatics ( IWIN 2018 )2.2 Operation +Table 1: Cost of scalar multiplication of nP , where n is k-bit

Let E be an elliptic curve given by Weierstrass form (2) ora Montgomery curve (3). Then, the operation + in E(Fp ) isdefined as follows.

Used curve
Cost
Weierstrass
18kM + kMa + 18k add
Montgomery 10kM + kMA0 + 10k add

1. For any P ∈ E(Fp ), P + O = O + P.
2. In the case of P = (x1 , y1 ), Q = (−x1 , y1 ) ∈ E(Fp ),P + Q = O.

Algorithm 1 (Binary method)
Input: P ∈ E(Fp ), n = (nk−1 nk−2 · · · n0 )2 ∈ N
Output: nP ∈ E(Fp )1. Q ← P
2. for i = k − 2 down to 03.
Q ← 2Q4.
if ni = 1 then Q ← Q + P
5. end for6. return Q

3. In the case of P = (x1 , y1 ), Q = (x2 , y2 ) ∈ E(Fp ), x1 6=
x2 , and P 6= Q, P + Q = (x3 , y3 ) is computed as
λ=y1 − y2,x1 − x2

x3 = λ2 − x1 − x2 , y3 = λ(x1 − x3 ) − y1 .

(4)

Algorithm 2 (Montgomery ladder)
Input: P ∈ E(Fp ), n = (nk−1 nk−2 · · · n0 )2 ∈ N
Output: nP ∈ E(Fp )1. Q0 ← O, Q1 ← P
2. for i = k − 1 down to 0
3.
if ni = 0 then Q1 ← Q0 + Q1 , Q0 ← 2Q0
4.
if ni = 1 then Q0 ← Q0 + Q1 , Q1 ← 2Q1
5. end for6. return Q0

4. In the case of P = Q = (x1 , y1 ) ∈ E(Fp ), P + Q =(x3 , y3 ) is computed as
 2

3x1 + a
E: Weierstrass 



2y1


λ=

2


3x1 +Ax1 +1

E: Montgomery,
2By1




2

x3 = λ − 2x1 ,
y3 = λ(x1 − x3 ) − y1 .

(5)

be the binary representation of n. Then, Algorithm 1 takes
k doubling formulas and k/2 addition formulas on average,
and Algorithm 2 takes k doubling formulas and k addition
formulas.
We will estimate the cost of theses algorithms. Let M ,Ma , MA0 , and add denote a modular multiplication in Fp ,a modular multiplication in Fp with a constant a, a modular
multiplication in Fp with a constant A0 , and a modular addition/subtraction in Fp , respectively, where a is of (2), and
A0 = (A + 2)/4 for A of (3). The cost of a scalar multiplication is given by Table 1, where add-1998-cmo-2 anddbl-2007-b1 for Weierstrass, and dadd-1987-m-3 anddbl-1987-m-3 for Montgomery are used as algorithms for
addition and doubling [2].

Eqs.(4) and (5) are called addition formula and doubling formula, respectively.
For P − Q = (x4 , y4 ) on Montgomery curve, x3 can becomputed as(x1 x2 − 1)2(6)x3 =x4 (x1 − x2 )2when x1 6= x2 . Eq. (6) is an important property of Montgomery curve.

2.3 Scalar Multiplication
For a base point P ∈ E(Fp ) and a natural number n, additions of n terms of P ,2.4

nP = P + P + · · · + P

Secure Elliptic Curve

The security of ECCs including digital signature depends
on the maximum prime factor l of the order L = #E(Fp ),not the size of p√[12]. Attack time against ECCs is roughly
proportional to l and then the larger l is, the more secure
ECCs are. Therefore, we have to select elliptic curve E suchthat
L = #E(Fp ) has a big prime factor(7)

is called scalar multiplication. For P ∈ E(Fp ), the order of P
is defined as the smallest positive integer l such that lP = O.
For the order L of E(Fp ) and the order l of P ∈ E(Fp ),the followings are held (Lagrange’s theorem).
1. l is a divisor of L.
2. LP = O.

for ECCs. Also, it is desirable that

Algorithm 1 (Binary method) and Algorithm 2 (Montgomery
ladder) are algorithms for computing a scalar multiplication.
Let n be a k-bit integer, andn = (nk−1 , nk−2 , · · · , n0 )2

2p + 2 − L that is the order of the twist of E
has a big prime factor
according to [3]. Moreover, we have to select E such that
L 6= p, p ± 1

200

(8)

(9)

International Workshop on Informatics ( IWIN 2018 )Algorithm 3 (Montgomery Reduction)
Input: Odd number N of n bits, R = 2n ,N 0 = (−N −1 ) mod R, natural number u < RN
Output: u mod N in Montgomery representation
1. t ← u
2. k ← T N 0 mod 2n
3. t ← t + N R
4. t ← t/R
5. if t ≥ N then t ← t − N
6. return t

according to [6], [13].
Curve25519 [1] is a Montgomery curve
E25519 : y 2 = x3 + 486662x2 + x
with p = 2255 − 19. The order L = #E25519 (Fp ) is
L = 22 · l,l = 2252 + 27742317777372353535851937790883648493,
where l is a 253-bit prime. Curve25519 meets the security
requirement in Sec,2.4.
Curve 25519 has been applied in many cryptographic libraries such as NaCl [10], and Curve 25519 was added toSpecial Publication 800-186, which specifies the approved elliptic curve used by the US federal government by NIST in2017.

2. Computing d = v −1 mod l.
3. Computing U 0 =(ud)Y| {z }.

scalar mul. on E(Fp )4. Computing u0 = (the x coordinate of U 0 ) mod l.

ECDSA is a digital signature using an elliptic curve. ECDSA
consists of system parameter held by all users, key generation
for generating each user’s (private key, public key), signature
generation for generating using a user A’s secret key, and signature verification for verifying the signature using A’s public
key.
System parameter
A sufficiently large (e.g. 256-bit) prime p, an elliptic curve E
such that E(Fp ) meets the security requirements (7), (8), and(9), and a base point G ∈ E(Fp ) of which the order is l are
selected. Also a hash function H : {0, 1}∗ → {0, 1, 2, ..., l −1} is selected. (p, l, E, G, H) is the system parameter.

5. If u = u0 then the signature is accepted, and if u 6= u0
then it is rejected.
Thus, the dominant processes of ECDSA is scalar multiplications in E(Fp ). Signature generation of ECDSA takes one
scalar multiplication and signature verification of ECDSA takes
two scalar multiplications. Therefore, we see thatin order to speed up processes of ECDSA,
it is important to speed up scalar multiplication.
As seen Table 1, using not Weierstrass form but Montgomery
curve reduces the number of modular multiplications required
for a scalar multiplication. Moreover, using a Montgomery
curve with appropriate coefficient A such as A = 6, 10, 14, 18
further reduces the number of modular multiplications.

Key generation
User A choose s ∈ [1, l − 1] at random, and computes Y =sG (scalar multiplication) in E(Fp ). Then, s and Y are A’s
private key and public key, respectively.

4

MODULAR REDUCTION

In order to speed up ECCs including ECDSA, it is important not only to reduce the number of modular multiplications
but also to reduce the cost of one modular multiplication. Thissection introduces efficient reduction methods.
The Montgomery reduction (Algorithm 3) [8] is a method
for efficiently calculating X mod N for general odd number
N and X given in Montgomery representation1 .
When a prime p is written as

Signature generation
User A generates a signature of a message m ∈ {0, 1}∗ asfollows.
1. Computing m0 = H(m).
2. Choosing r ∈ [1, l − 1] at random, and compute
rG
|{z}

+

scalar mul. on E(Fp )3 ECDSAU=(m0 d)G| {z }

p = 2n − k, k < 2n/2 ,= (ux , uy ),it is called pseudo Mersenne prime. For pseudo Mersenne
prime p, reduction mod p can be computed at high speed using by Algorithm 4 [7]. Notice that u/2n in step 1 and v/2n
in step 3 are integer divisions and then they are performed byshift operations.

scalar mul. on E(Fp )u = ux mod l.
3. Using the secret key s to computev = r−1 (m0 + su) mod l.

5

4. The pair (u, v) is the signature of m.

CONTRIBUTIONS

5.1

Signature verification
A recipient of the message m with signature (u, v) verifies
the signature as follows.

Program to Search Elliptic Curve Suitable
for ECDSA

The purpose of this paper is to make a program to searchfor elliptic curves that is secure and suitable for high-speed

1. Computing m0 = H(m).

1 For Montgomery representation, refer to [8] or [4].

201International Workshop on Informatics ( IWIN 2018 )Thus, Num1 is the order of E(Fp ), and Num2 is the order of
the twist of E(Fp ). isprime is a prime decision function.
write is an output function to text.
By this program, the following elliptic curves are found.

Algorithm 4 (Reduction mod pseudo Mersenne prime)
Input: prime p = 2n − k (k < 2n/2 ),integer 0 ≤ u ≤ (p − 1)2
Output: u mod p
1. u0 ← u mod 2n , u1 ← u/2n
2. v ← u1 k + u0
3. v0 ← v mod 2n , v1 ← v/2n
4. w ← v1 k + v0
5. if w ≥ p then w ← w − N
6. return w\\Checking pseudo Mersenne prime
check_mer(p)={local(n,c,k);
n=0;
c=0;
while(c==0,n++;if(2ˆ(n-1)<=p && p<2ˆn,c=1););if(2ˆn-p < sqrt(2ˆn),k=floor(log(2ˆn-p)/log(2)+1);
return([n,k]),return(0));}

implementation of ECDSA (especially by hardware implementation), and to give examples of such an elliptic curves.
Specifically, we will search curves that meet the following requirements.
Elliptic Curve Requirements to Search
1. Montgomery curve is selected because a scalar multiplication takes fewer modular multiplications (Table 1).

\\Main program{
count=0;
for(a=0,3,
A=4*a+6;
e=ellinit([0,A,0,1,0]);for(k=1,2ˆ20,
print([a,k,count]);p=2ˆ256-k;
if(isprime(p)==1,
t=ellap(e,p);num1=p+1-t;num2=p+1+t;
Num1=num1;
Num2=num2;
check1=0;
check2=0;
if(num1%2==0,num1=num1/2;check1=1);
if(num1%2==0,num1=num1/2;check1=2);
if(num1%2==0,num1=num1/2;check1=3);
if(num1%2==0,num1=num1/2;check1=4);
if(num2%2==0,num2=num2/2;check2=1);
if(num2%2==0,num2=num2/2;check2=2);
if(num2%2==0,num2=num2/2;check2=3);
if(num2%2==0,num2=num2/2;check2=4);
isprime_num1=isprime(num1);
isprime_num2=isprime(num2);
if(t!=0 && isprime_num1==1&& isprime_num2==1&& (check_mer(Num1)!=0|| check_mer(Num2)!=0),
if(check_mer(Num1)!=0,
count++;
write("iwin.txt",k","A","Num1","num1););
if(check_mer(Num2)!=0,
count++;
print("A="A);write("iwin.txt",k","A","Num2","num2);
);
);
);
);
);
}

2. Moreover, Montgomery curve with A0 = 1, 2, 3, 4, that
is, A = 6, 10, 14, 18 is selected according to Table 1.
3. Prime p is is a pseudo Mersenne prime p = 2n − kof 256-bit. For convenience of execution time, set therange of k to k ≤ 220 .
4. To meet the security requirement (7), the order L =#E(Fp ), which is always a multiple of 4, is as L =4l, 8l, 16l, where l is a prime.
5. To meet the security requirement (8), L0 = 2p + 2 − L
is as L0 = 4l0 , 8l0 , 16l0 , where l0 is a prime.
6. To meet the security requirement (9), curves such as
L = p, p ± 1 is removed.
7. L is written as L = 2n − k 0 , k 0 < 2n/2 . Then, a reduction mod l is computed by an algorithm, which is as
efficient as reduction mod a pseudo Mersenne prime.
Note Curve25519 also meets the requirements 1, 4, 5, and7, and Curve25519 adopts not 256-bit prime but 254-bit for aprime field. Curve25519 does not consider the requirements
2 and 7.
The authors made a program as Fig.1 in PARI/GP [11] tosearch elliptic curves meeting the requirements. The programis straightforward and then it may be easy for some readers
to make a similar program. But, giving the program makes
all readers (especially PARI/GP users) generate good curves.
Notice that the program output only a prime p, a coefficient
A of Montgomery curve, the order L of E(Fp ), and the orderl of a base point. At the moment, it is necessary to manuallyfind another coefficient B of Montgomery curve, generate abase point whose order is l, and check L 6= p, p − 1.
This program is briefly explained. The line
e=ellinit([0,A,0,1,0]);Figure 1: Proposed program to find elliptic curves suitable for
ECDSA

sets (Montgomery) elliptic curve E : y 2 = x3 + Ax2 + x toe. The function ellap(e,p) outputs the trace of E(Fp ).
202

International Workshop on Informatics ( IWIN 2018 )1. p = 2256 − 58097,
ES1 : 638y 2 = x3 + 10x2 + x,
base point P = (11, 2),
L = 2256 − k 0 , where k 0 is 125-bit integer
k 0 = 25181363380428710453079967399017869328,
l = L/16, which is 252-bit prime.

Proposed Algorithm 5
Input : integer l such that 2m l = 2n − k (k < 2n/2 ),integer 0 ≤ u ≤ (l − 1)2
Output : u mod l
1. v ← 2m u
2. v0 ← v mod 2n , v1 ← v/2n
3. w ← v1 k + v0
4. w0 ← w mod 2n , w1 ← w/2n
5. x ← w1 k + w0
6. if x ≥ 2m l then x ← x − 2m l
7. y ← x/2m
8. return y

2. p = 2256 − 507225,
ES2 : 82y 2 = x3 + 18x2 + x,
base point P = (2, 1),
L = 2256 − k 0 , where k 0 is 127-bit integer
k 0 = 134184981501621384111934924743103436264,
l = L/8, which is 253-bit prime.
3. p = 2256 − 979077,
ES3 : 3805y 2 = x3 + 18x2 + x,
base point P = (20, 2),
L = 2256 − k 0 , where k 0 is 126-bit integer
k 0 = 67240641251824776802983670794157366424,
l = L/8, which is 253-bit prime.

REFERENCES

For the convenience of time, the authors set the search range
to k < 220 , however, if the search range is expanded, more
appropriate elliptic curve may be found.

5.2 Proposed Modular Reduction
This section proposes an algorithm (Algorithm 5) 2 similar
to Algorithm 8 for computing a reduction mod l for a prime
l such that L = 2m l is written as L = 2n − k, k < 2n/2 .
Notice v, v0 and k are multiples of 2m . Thus, w is also amultiple of 2m . As well,
x is a multiple of 2m .

(10)

Steps from 2 to 7 are same as Algorithm 8 and then we see
x = 2m u mod 2m l.

(11)

By (10) and (11), we see x/2m = u mod l.
Note that v/2n in step 2, w/2n in step 4, and x/2n in step
7 are integer divisions and then they are performed by shiftoperations, and 2m u in step 1 is also performed by a shiftoperation.

6 Conclusion
This paper searched three elliptic curves suitable for ECDSA.
In these curves, not only the reduction mod p but also the reduction mod l can be computed at high speed, where p is of
Fp and l is the order of a base point, and doubling is fasterbecause of a coefficient of curves A = 10 or 18. ECDSA
adopting the searched curves has the same security as ECDSA
adopting Curve25519, and it can process faster.

Acknowledgment
This work was supported by JSPS KAKENHI Grant Number 16K00188.
2 Although the authors do not know whether Algorithm 5 is alreadyknown, it may be already known because Algorithm 5 is almost same as
Algorithm 8.

203[1] Daniel J. Bernstein. Curve25519: New Diffie-Hellman
Speed Records. Public Key Cryptography - PKC 2006,
LNCS 3958, pp. 207–228. Springer (2006).
[2] Daniel J. Bernstein and Tanja Lange.
ExplicitFormulas Database. https://hyperelliptic.
org/EFD/.
[3] Daniel J. Bernstein and Tanja Lange. Safecurves:
Choosing Safe Curves for Elliptic-Curve Cryptography.
https://safecurves.cr.yp.to.
[4] Henri Cohen and Gerhard Frey. Handbook of Elliptic
and Hyperelliptic Curve Cryptography. Chapman andHall/CRC (2005).
[5] ETSI. ETSI TS 103 097 v1.1.1 Intelligent Transport
Systems (ITS); Security; Security Header and Certificate Formats (2013).
[6] Gerhard Frey and Hans-Georg Rück. A Remark Concerning M -Divisibility and the Discrete Logarithm inthe Divisor Class Group of Curves. Mathematics ofComputation, Vol. 62, No. 206, pp. 865–874 (1994).
[7] Alfred J. Menezes, Paul C. van Oorschot, and Scott A.
Vanstone. Handbook of Applied Cryptography. CRC
Press (1997).
[8] Peter L. Montgomery. Modular Multiplication Without
Trial Division. Mathematics of Computation, Vol. 44,
No. 170, pp. 519–521 (1985).
[9] Peter L. Montgomery. Speeding the Pollard and Elliptic
Curve Methods of Factorization. Mathematics of Computation, Vol. 48, No. 177, pp. 243–264 (1987).
[10] NaCl: Networking and Cryptography Library. https://pari.math.u-bordeaux.fr.
[11] PARI/GP. https://pari.math.u-bordeaux.
fr.
[12] S. Pohlig and M. Hellman. An Improved Algorithm
for Computing Logarithms Over GF (p) and Its Cryptographic Significance. IEEE Transactions on Information
Theory, Vol. 24, No. 1, pp. 106–110 (1978).
[13] Hans-Georg Rück. On the Discrete Logarithm in the
Divisor Class Group of Curves. Mathematics of Computation, Vol. 68, No. 226, pp. 805–806 (1999).
[14] Joseph. H. Silverman. The Arithmetic of Elliptic Curves.
Springer-Verlag New York (1985).